workflows: add ubuntu-latest to beam_PreCommit_Python_ML, beam_PostCommit_Python, and beam_PreCommit_Python_Coverage

[moawnallah]

Description

In this PR, we add ubuntu-latest to beam_PreCommit_Python_ML, beam_PostCommit_Python, and beam_PreCommit_Python_Coverage workflows. Currently some tests (e.g Milvus tests) have issues running on self-hosted env due to docker_in_docker env limitation.

Dependency for #35577.

Motivation and Context

This change was removed in #35734 due to overlapping issues with pytest markers so it wasn't clear what was the issue actually. With this experimental change we will have more clarity what the issues are of adding ubuntu-latest to those workflows (if any).

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

[moawnallah]

cc @damccorm

[damccorm]

Thanks! I copied your changes and am running a couple of test runs to try them out:

• https://github.com/apache/beam/actions/runs/16814631250
• https://github.com/apache/beam/actions/runs/16814620128

[github-actions]

Assigning reviewers:

R: @damccorm for label build.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

Thanks! I copied your changes and am running a couple of test runs to try them out:

• https://github.com/apache/beam/actions/runs/16814631250
• https://github.com/apache/beam/actions/runs/16814620128

Some things failed, but it may just be flakes. The Python Postcommit workflow seems to be broken anyways

[moawnallah]

The precommit ml tests failing are currently the ones running on ubuntu-latest. I will try to investigate this by using the same ubuntu-latest variant locally and see if any errors I could reproduce there 👍

[damccorm]

There is good news and bad news. The bad news is that the new docker_in_docker jobs failed.

The good news is that they failed in an expected/positive way - they returned exit code 5 which means no tests were run. Since no tests actually have the require_docker_in_docker tag yet, this is not surprising. We should just hold off on this PR until we have tests which require the marker.

[moawnallah]

The good news is that they failed in an expected/positive way - they returned exit code 5 which means no tests were run. Since no tests actually have the require_docker_in_docker tag yet, this is not surprising.

Nice!

We should just hold off on this PR until we have tests which require the marker.

I merged the change of upstream beam master branch in this PR #35577 to be reviewable state. With this PR we would have require_docker_in_docker marked as well +1 for making milvus as extra dependency. Will invite for a look after the relevant CI tests pass

[damccorm]

Testing this against latest master:

• https://github.com/apache/beam/actions/runs/17000463519
• https://github.com/apache/beam/actions/runs/17000460527

[moawnallah]

Testing this against latest master:

• https://github.com/apache/beam/actions/runs/17000463519
• https://github.com/apache/beam/actions/runs/17000460527

This seems promising. There are no docker_in_docker issues on Python_ML PreCommit 3.11 and milvus tests pass there:
https://github.com/apache/beam/actions/runs/17000460527/job/48201093064

Postcommit failures and other precommits ml python variant versions seems not related

[moawnallah]

I found some issues in the CI due to milvus extra dependency not added properly. Submitted commit to fix this

[moawnallah]

We can trigger those workflows again after properly making milvus as extra dependency

EDIT:
I've triggered postcommit python just to make sure the code changes in this PR don't cause any regressions. For workflow changes they had to be triggered manually to reflect those changes

[moawnallah]

Hey @damccorm, Currently most of postcommits are failing (mostly due to google cloud access) and a few precommits failing. I am curious is this change regards adding milvus extra dependency properly (475289f) included in the trigger?

Sorry missed this. Would you mind pulling in the latest changes to master + resolving conflicts? I'm guessing at that point the issues should be resolved

Hey @damccorm, I've merged the latest changes from master branch and added ubuntu-latest to beam_PreCommit_Python_Coverage workflow. Perhaps we can trigger relevant workflows with those changes to see if any other issues?

[damccorm]

Yes, running here:

• Postcommit python - https://github.com/apache/beam/actions/runs/17324563704
• Precommit coverage - https://github.com/apache/beam/actions/runs/17324579789
• Precommit ML - https://github.com/apache/beam/actions/runs/17324590596

[damccorm]

Looks like the precommit ML workflow failed and so did the postcommit python workflow.

The precommit ML workflow at least seems to be an error related to tfx - https://stackoverflow.com/questions/77972546/pip-vendor-resolvelib-resolvers-resolutiontoodeep-200000

Could you try updating pip as part of the workflow?

I'll also note that you should be able to at least run the precommit ML workflow on your fork to test for correctness, would you mind trying that?

[moawnallah]

Looks like the precommit ML workflow failed and so did the postcommit python workflow.

The precommit ML workflow at least seems to be an error related to tfx - https://stackoverflow.com/questions/77972546/pip-vendor-resolvelib-resolvers-resolutiontoodeep-200000

Could you try updating pip as part of the workflow?

I'll also note that you should be able to at least run the precommit ML workflow on your fork to test for correctness, would you mind trying that?

Added that commit c8d88b0 that would upgrade pip in precommit ML workflow and this is the PR on my fork:
moawnallah#7

[damccorm]

Sorry, I lost track of this PR. It looks like all of the runs on your PR got cancelled, not sure why, but I'm running the relevant workflows now:

• Postcommit python - https://github.com/apache/beam/actions/runs/17493410318
• Precommit ML - https://github.com/apache/beam/actions/runs/17493427302

[moawnallah]

Sorry, I lost track of this PR. It looks like all of the runs on your PR got cancelled, not sure why, but I'm running the relevant workflows now:

• Postcommit python - https://github.com/apache/beam/actions/runs/17493410318
• Precommit ML - https://github.com/apache/beam/actions/runs/17493427302

@damccorm – have just removed ubuntu-latest from postcommit and forced all milvus tests to run on precommit ml by making is_integration_test to be false. Can we trigger them with those updates to make sure that postcommits pass and all milvus tests run in the precommit?

[damccorm]

Sure, kicking off now:

• Postcommit python - https://github.com/apache/beam/actions/runs/17501563166
• Precommit coverage - https://github.com/apache/beam/actions/runs/17501571103
• Precommit ML - https://github.com/apache/beam/actions/runs/17501579036

[moawnallah]

• Postcommit python - https://github.com/apache/beam/actions/runs/17501563166
• Precommit coverage - https://github.com/apache/beam/actions/runs/17501571103
• Precommit ML - https://github.com/apache/beam/actions/runs/17501579036

Precommit ML currently failing on python versions < 3.12 with Dependency resolution exceeded maximum depth error. This seems related to this discussion (#35816 (comment)). I have updated pip version in those workflows in this commit c8d88b0) and is currently up to date with version 25.2. It looks like not resolving this issue. Since ubuntu-latest passes on python version 3.12 perhaps we can exclude the other versions from precommit ml python workflow as a temprorary solution

[moawnallah]

Reverted this commit c8d88b0. Also excluded Python versions <3.12 in the ubuntu-latest environment. These tests are expected to run and pass on Python 3.12 with ubuntu-latest on Precommit ML. @damccorm, Can you please take another look?

[damccorm]

This sounds good to me. Kicking off another set of runs:

Postcommit python - https://github.com/apache/beam/actions/runs/17553439850
Precommit coverage - https://github.com/apache/beam/actions/runs/17553445547
Precommit ML - https://github.com/apache/beam/actions/runs/17553449564

[moawnallah]

The relevant tests are passing with the changeset in this PR 👍

PS:
The postcommits failing with the following and seems unrelated:

1. CloudSQL (PostgreSQL, MySQL, SQLServer) testcontainers seems having troubles while starting on self-hosted environment (attempting to resolve by [2/2] sdks/python: enrich data with CloudSQL [PostgreSQL, MySQL, SQLServer] #35473 (comment)):

│2025-09-08T15:41:01.0433159Z _ ERROR at setup of TestExternalSQLServerEnrichment.test_sql_enrichment_with_redis _ │
│2025-09-08T15:41:01.0433695Z [gw2] linux -- Python 3.9.22 /runner/_work/beam/beam/build/gradleenv/1398941893/bin/python3.9 │
│2025-09-08T15:41:01.0433965Z │
│2025-09-08T15:41:01.0434131Z self = <docker.api.client.APIClient object at 0x7a1418302580> │
│2025-09-08T15:41:01.0434445Z response = <Response [500]> ┤
│2025-09-08T15:41:01.0434597Z │
│2025-09-08T15:41:01.0434920Z def _raise_for_status(self, response): ┤
│2025-09-08T15:41:01.0435505Z """Raises stored :class:`APIError`, if one occurred.""" │
│2025-09-08T15:41:01.0436017Z try: │
│2025-09-08T15:41:01.0436358Z > response.raise_for_status() ┤
│2025-09-08T15:41:01.0436551Z ┤
│2025-09-08T15:41:01.0436878Z ../../build/gradleenv/1398941893/lib/python3.9/site-packages/docker/api/client.py:275: │
│2025-09-08T15:41:01.0437304Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ │
│2025-09-08T15:41:01.0437537Z ┤
│2025-09-08T15:41:01.0437633Z self = <Response [500]> │
│2025-09-08T15:41:01.0437775Z │
│2025-09-08T15:41:01.0438111Z def raise_for_status(self): │
│2025-09-08T15:41:01.0438653Z """Raises :class:`HTTPError`, if one occurred.""" │
│2025-09-08T15:41:01.0439026Z │
│2025-09-08T15:41:01.0439366Z http_error_msg = "" │
│2025-09-08T15:41:01.0439892Z if isinstance(self.reason, bytes): │
│2025-09-08T15:41:01.0440455Z # We attempt to decode utf-8 first because some servers │
│2025-09-08T15:41:01.0440999Z # choose to localize their reason strings. If the string │
│2025-09-08T15:41:01.0441519Z # isn't utf-8, we fall back to iso-8859-1 for all other │
│2025-09-08T15:41:01.0441997Z # encodings. (See PR #3538) ┤
│2025-09-08T15:41:01.0442355Z try: │
│2025-09-08T15:41:01.0442904Z reason = self.reason.decode("utf-8") │
│2025-09-08T15:41:01.0443457Z except UnicodeDecodeError: │
│2025-09-08T15:41:01.0444027Z reason = self.reason.decode("iso-8859-1") │
│2025-09-08T15:41:01.0444479Z else: │
│2025-09-08T15:41:01.0444829Z reason = self.reason │
│2025-09-08T15:41:01.0445137Z │
│2025-09-08T15:41:01.0445556Z if 400 <= self.status_code < 500: │
│2025-09-08T15:41:01.0446100Z http_error_msg = ( │
│2025-09-08T15:41:01.0446949Z f"{self.status_code} Client Error: {reason} for url: {self.url}" │
│2025-09-08T15:41:01.0447601Z ) │
│2025-09-08T15:41:01.0447861Z │
│2025-09-08T15:41:01.0448292Z elif 500 <= self.status_code < 600: │
│2025-09-08T15:41:01.0448721Z http_error_msg = ( │
│2025-09-08T15:41:01.0449572Z f"{self.status_code} Server Error: {reason} for url: {self.url}" ┤
│2025-09-08T15:41:01.0450207Z ) ┤
│2025-09-08T15:41:01.0450461Z │
│2025-09-08T15:41:01.0450760Z if http_error_msg: ┤
│2025-09-08T15:41:01.0451239Z > raise HTTPError(http_error_msg, response=self) ┤
│2025-09-08T15:41:01.0452377Z E requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.43/containers/09aedca9488b11927943a0650f83b6103d9e5eb8ff4326bc3b5fd03b9e30512e/start │
│2025-09-08T15:41:01.0453062Z ┤
│2025-09-08T15:41:01.0453431Z ../../build/gradleenv/1398941893/lib/python3.9/site-packages/requests/models.py:1026: HTTPError

2. Milvus tests currently running, likely the require_docker_in_docker marker not included in the postcommit trigger (seems unrelated):

2025-09-08T15:36:52.5113336Z [gw2] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_filtered_search_with_cosine_similarity_an│
│2025-09-08T15:36:52.5115616Z [gw5] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_keyword_search_with_inner_product_sparse_│
│2025-09-08T15:36:52.5118066Z [gw7] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_vector_search_with_euclidean_distance │
│2025-09-08T15:36:52.5120255Z [gw6] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_invalid_query_on_non_existent_field │
│2025-09-08T15:36:52.5122382Z [gw4] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_invalid_query_on_non_existent_collection │
│2025-09-08T15:36:52.5124576Z [gw1] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_filtered_search_with_bm25_full_text_and_b┤
│2025-09-08T15:36:52.5126773Z [gw0] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_empty_input_chunks │
│2025-09-08T15:36:52.5128484Z apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_vector_search_with_inner_product_similarity │
│2025-09-08T15:36:52.5130438Z [gw3] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_hybrid_search │
│2025-09-08T15:36:54.3109653Z [gw0] ERROR apache_beam/ml/rag/enrichment/milvus_search_it_test.py::TestMilvusSearchEnrichment::test_vector_search_with_inner_product_similari│
│2025-09-08T15:36:54.3110403Z

3. Bigtable failures (unrelated):

│2025-09-08T15:39:32.4169351Z raise ValueError( ┤
│2025-09-08T15:39:32.4170057Z ValueError: no matching row found for row_key: 11 with row_filter=<google.cloud.bigtable.row_filters.CellsColumnLimitFilter object at 0x7e8df0┤
│2025-09-08T15:39:32.4170730Z

[moawnallah]

@damccorm - this PR seems ready for merge as well given the changes pass in the relevant workflows?

[damccorm]

I agree that the bigtable failures seem unrelated/unconcerning.

The other ones seem more like real problems that we should deal with (probably in other prs) before merging this one. I don't think we should merge here until we're confident that workflows will be green (otherwise this will create work for others)

[moawnallah]

The other ones seem more like real problems that we should deal with (probably in other prs) before merging this one. I don't think we should merge here until we're confident that workflows will be green (otherwise this will create work for others)

Sounds good and thoughtful. Perhaps waiting after this PR #35473 is merged hopefully no longer be issues on postcommit regards cloudsql tests 👍

Regards milvus tests they aren't supposed to running on postcommit python workflow given they are already marked with require_docker_in_docker and excluded to run in this workflow since we have removed ubuntu-latest:

beam/sdks/python/apache_beam/ml/rag/enrichment/milvus_search_it_test.py

Lines 472 to 480 in 10f0f07

	@pytest.mark.require_docker_in_docker
	@unittest.skipUnless(
	platform.system() =="Linux",
	"Test runs only on Linux due to lack of support, as yet, for nested "
	"virtualization in CI environments on Windows/macOS. Many CI providers run "
	"tests in virtualized environments, and nested virtualization "
	"(Docker inside a VM) is either unavailable or has several issues on "
	"non-Linux platforms.")
	classTestMilvusSearchEnrichment(unittest.TestCase):

beam/.github/workflows/beam_PostCommit_Python.yml

Lines 109 to 113 in 10f0f07

	-Pposargs="${{
	contains(matrix.os, 'self-hosted') &&
	'-m (not require_docker_in_docker)' ||
	'-m require_docker_in_docker'
	}}" \

[moawnallah]

Regards milvus tests they aren't supposed to running on postcommit python workflow given they are already marked with require_docker_in_docker and excluded to run in this workflow since we have removed ubuntu-latest:

beam/sdks/python/apache_beam/ml/rag/enrichment/milvus_search_it_test.py

Lines 472 to 480 in 10f0f07

	@pytest.mark.require_docker_in_docker
	@unittest.skipUnless(
	platform.system() =="Linux",
	"Test runs only on Linux due to lack of support, as yet, for nested "
	"virtualization in CI environments on Windows/macOS. Many CI providers run "
	"tests in virtualized environments, and nested virtualization "
	"(Docker inside a VM) is either unavailable or has several issues on "
	"non-Linux platforms.")
	classTestMilvusSearchEnrichment(unittest.TestCase):

beam/.github/workflows/beam_PostCommit_Python.yml

Lines 109 to 113 in 10f0f07

	-Pposargs="${{
	contains(matrix.os, 'self-hosted') &&
	'-m (not require_docker_in_docker)' ||
	'-m require_docker_in_docker'
	}}" \

@damccorm you are right. Milvus tests failure turned actually to be real problem. It turns out the issue forgetting removing dockerInDockerIntegrationTest from the test-suites gradle file (so they were triggered on postcommit given no ubuntu-latest though). With this commit cc9f74c this issue would be addressed

Regards CloudSQL tests given that PR #35473 is merged (also merged the latest changes here). They are expected not to run in the postcommits so no issues supposed to happen regards them in PostCommit python workflow

Can we trigger the workflows again (similar to #35816 (comment)) with those changes to make sure no regressions?

[damccorm]

Triggered!

Postcommit python - https://github.com/apache/beam/actions/runs/17676270810
Precommit coverage - https://github.com/apache/beam/actions/runs/17676311003
Precommit ML - https://github.com/apache/beam/actions/runs/17676320186

[damccorm]

Hm, the postcommit failures look like they may be related to the last pr - 1da37bc#diff-ebd8674386d314aac76d74aaec08bc2fc23427577e3021ce5014c080c4939ca9

I see :sdks:python:test-suites:direct:py39:testcontainersTest FAILED caused by bigtable enrichment issues. @mohamedawnallah could you take a look please?

I think this PR is fine, so we can merge here for now

[moawnallah]

I see :sdks:python:test-suites:direct:py39:testcontainersTest FAILED caused by bigtable enrichment issues. @mohamedawnallah could you take a look please?

@damccorm – In this PR #35473 and current one we haven't touched the bigtable enrichment code just the CloudSQL enrichment code. I have also checked the logs in the PostCommit workflows run (https://github.com/apache/beam/actions/runs/17676270810/job/50239024594) cloudSQL tests wasn't running and that is expected

$ cat "job-logs-python-3.9.txt"| grep "cloudsql"
2025-09-12T13:48:40.6195331Z #9 153.4 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/lib.linux-x86_64-cpython-39/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:40.6197152Z #9 153.4 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-39/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:40.6201888Z #9 153.4 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/lib.linux-x86_64-cpython-39/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:40.6995855Z #9 153.5 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql.py -> build/lib.linux-x86_64-cpython-39/apache_beam/ml/rag/ingestion
2025-09-12T13:48:40.6996190Z #9 153.5 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-39/apache_beam/ml/rag/ingestion
2025-09-12T13:51:13.3258607Z #9 306.1 INFO:root:copying build/lib.linux-x86_64-cpython-39/apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:13.3260727Z #9 306.1 INFO:root:copying build/lib.linux-x86_64-cpython-39/apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:13.3266127Z #9 306.1 INFO:root:copying build/lib.linux-x86_64-cpython-39/apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:13.4147035Z #9 306.2 INFO:root:copying build/lib.linux-x86_64-cpython-39/apache_beam/ml/rag/ingestion/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:51:13.4147442Z #9 306.2 INFO:root:copying build/lib.linux-x86_64-cpython-39/apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:51:14.6732152Z #9 307.4 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql.py'
2025-09-12T13:51:14.6732532Z #9 307.4 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql_it_test.py'
2025-09-12T13:51:15.7710474Z #9 308.5 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql.py'
2025-09-12T13:51:15.7710925Z #9 308.5 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py'
2025-09-12T13:51:15.7711400Z #9 308.5 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_test.py'

$ cat "job-logs-python-3.10.txt"| grep "cloudsql"
2025-09-12T13:48:58.9028997Z #9 122.9 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/lib.linux-x86_64-cpython-310/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:58.9030871Z #9 122.9 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-310/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:58.9035431Z #9 122.9 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/lib.linux-x86_64-cpython-310/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:58.9554468Z #9 123.3 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql.py -> build/lib.linux-x86_64-cpython-310/apache_beam/ml/rag/ingestion
2025-09-12T13:48:58.9555197Z #9 123.3 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-310/apache_beam/ml/rag/ingestion
2025-09-12T13:51:40.6986079Z #9 304.7 INFO:root:copying build/lib.linux-x86_64-cpython-310/apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:40.6989720Z #9 304.7 INFO:root:copying build/lib.linux-x86_64-cpython-310/apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:40.6997091Z #9 304.7 INFO:root:copying build/lib.linux-x86_64-cpython-310/apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:40.7944543Z #9 304.8 INFO:root:copying build/lib.linux-x86_64-cpython-310/apache_beam/ml/rag/ingestion/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:51:40.7945462Z #9 304.8 INFO:root:copying build/lib.linux-x86_64-cpython-310/apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:51:41.9564684Z #9 305.9 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql.py'
2025-09-12T13:51:41.9565338Z #9 305.9 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql_it_test.py'
2025-09-12T13:51:43.1498356Z #9 307.1 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql.py'
2025-09-12T13:51:43.1498837Z #9 307.1 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py'
2025-09-12T13:51:43.1499331Z #9 307.1 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_test.py'

$ cat "job-logs-python-3.11.txt"| grep "cloudsql"
2025-09-12T13:48:55.9911738Z #9 140.9 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/lib.linux-x86_64-cpython-311/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:55.9913728Z #9 140.9 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-311/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:55.9918358Z #9 140.9 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/lib.linux-x86_64-cpython-311/apache_beam/transforms/enrichment_handlers
2025-09-12T13:48:56.1628991Z #9 140.9 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql.py -> build/lib.linux-x86_64-cpython-311/apache_beam/ml/rag/ingestion
2025-09-12T13:48:56.1629697Z #9 140.9 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-311/apache_beam/ml/rag/ingestion
2025-09-12T13:51:56.8935772Z #9 298.1 INFO:root:copying build/lib.linux-x86_64-cpython-311/apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:56.8937888Z #9 298.1 INFO:root:copying build/lib.linux-x86_64-cpython-311/apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:56.8943917Z #9 298.1 INFO:root:copying build/lib.linux-x86_64-cpython-311/apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:51:56.9648710Z #9 298.2 INFO:root:copying build/lib.linux-x86_64-cpython-311/apache_beam/ml/rag/ingestion/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:51:56.9649138Z #9 298.2 INFO:root:copying build/lib.linux-x86_64-cpython-311/apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:51:56.9959580Z #9 299.2 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql.py'
2025-09-12T13:51:56.9959790Z #9 299.2 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql_it_test.py'
2025-09-12T13:51:57.0100949Z #9 300.3 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql.py'
2025-09-12T13:51:57.0101317Z #9 300.3 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py'
2025-09-12T13:51:57.0101664Z #9 300.3 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_test.py'

$ cat "job-logs-python-3.12.txt" | grep "cloudsql"
2025-09-12T13:50:19.3965118Z #9 143.1 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/lib.linux-x86_64-cpython-312/apache_beam/transforms/enrichment_handlers
2025-09-12T13:50:19.3967125Z #9 143.1 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-312/apache_beam/transforms/enrichment_handlers
2025-09-12T13:50:19.3971592Z #9 143.1 INFO:root:copying apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/lib.linux-x86_64-cpython-312/apache_beam/transforms/enrichment_handlers
2025-09-12T13:50:19.5547731Z #9 143.2 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql.py -> build/lib.linux-x86_64-cpython-312/apache_beam/ml/rag/ingestion
2025-09-12T13:50:19.5548308Z #9 143.2 INFO:root:copying apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/lib.linux-x86_64-cpython-312/apache_beam/ml/rag/ingestion
2025-09-12T13:53:14.7865100Z #9 297.0 INFO:root:copying build/lib.linux-x86_64-cpython-312/apache_beam/transforms/enrichment_handlers/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:53:14.7867497Z #9 297.0 INFO:root:copying build/lib.linux-x86_64-cpython-312/apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:53:14.7872644Z #9 297.0 INFO:root:copying build/lib.linux-x86_64-cpython-312/apache_beam/transforms/enrichment_handlers/cloudsql_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/transforms/enrichment_handlers
2025-09-12T13:53:14.8536146Z #9 297.2 INFO:root:copying build/lib.linux-x86_64-cpython-312/apache_beam/ml/rag/ingestion/cloudsql.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:53:14.8536719Z #9 297.2 INFO:root:copying build/lib.linux-x86_64-cpython-312/apache_beam/ml/rag/ingestion/cloudsql_it_test.py -> build/bdist.linux-x86_64/wheel/./apache_beam/ml/rag/ingestion
2025-09-12T13:53:14.8837149Z #9 298.2 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql.py'
2025-09-12T13:53:14.8837320Z #9 298.2 INFO:wheel:adding 'apache_beam/ml/rag/ingestion/cloudsql_it_test.py'
2025-09-12T13:53:14.8974400Z #9 299.2 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql.py'
2025-09-12T13:53:14.8974621Z #9 299.2 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_it_test.py'
2025-09-12T13:53:14.8974826Z #9 299.2 INFO:wheel:adding 'apache_beam/transforms/enrichment_handlers/cloudsql_test.py'