Add streaming tests to Storage Write API Python wrapper

[ahmedabu98]

No description provided.

[ahmedabu98]

R: @chamikaramj

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[codecov]

Codecov Report

Merging #26491 (82d9ec3) into master (a13749f) will decrease coverage by 9.20%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #26491 +/- ##
==========================================
- Coverage 81.11% 71.92% -9.20% 
==========================================
Files 469 752 +283 Lines 67438 101698 +34260 ==========================================
+ Hits 54705 73142 +18437 - Misses 12733 27065 +14332 - Partials 0 1491 +1491 

Flag	Coverage Δ
python	81.10% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 290 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Abacn]

Some tips to make tests running faster

• use wheels for sdk_location (like Use wheels for Dataflow postcommit tests #25970use wheel sdk location for PostCommit_Py_Examples #26473), this would save 3.5 min per test
• run multiple tests in parallel by xdist option of pytest, like what is already doing for python postcommit

[Abacn]

Adding the flag is not yet effective as the sdk_location is set here for this test:

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 2489 in 4a09ea9

"--sdk_location=${pythonDir}/build/apache-beam.tar.gz"

instead of needsSdkLocation, probably provide the wheel location here and config that the test suite depends on initializeForDataflowJob.

beam/sdks/python/test-suites/dataflow/common.gradle

Line 457 in f5f7a47

needsSdkLocation: true,

[Abacn]

This actually won't work. initializeForDataflowJob exists in project like :sdks:python:test-suites:dataflow:py310 not :sdks:python:test-suites:dataflow.

We can add a parameter(s) setupTasks to createCrossLanguageUsingJavaExpansionTask , and then declare here

pythonTask.configure {dependsOn setupTask}

and call

createCrossLanguageUsingJavaExpansionTask(setupTask: 'initializeForDataflowJob'
...

[chamikaramj]

Thanks.

[chamikaramj]

LGTM.

[ahmedabu98]

Run Python_Xlang_Gcp_Dataflow PostCommit

[ahmedabu98]

Run Python_Xlang_Gcp_Direct PostCommit

[Abacn]

LGTM

possible followup: move CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIONS and the logic of pick up two py versions from jenkins script to gradle property of test-suites,

beam/sdks/python/test-suites/gradle.properties

Line 36 in dc30afb

dataflow_validates_runner_batch_tests_V2=3.7,3.11

Then claim a gradle task in dataflow/build.gradle:

beam/sdks/python/test-suites/dataflow/build.gradle

Line 64 in dc30afb

task validatesRunnerBatchTestsV2 {

like what we did for validate runner tests:

The running time of the dataflow test suite would decrease by half because now py37 and py311 tests are running concurrently. Same thing can be done for the direct runner test.