Adds helper functions for computing and using code object identifiers to facilitate more efficient pickling of lambdas. (#35656) by kristynsmith · Pull Request #35656 · apache/beam

kristynsmith · 2025-07-22T17:57:19Z

Creates consistent pickling for lambdas by creating a unique name for lambdas that can be used for stable referencing.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…y into code_object_pickler_test.py, rename files in test_data, rename tests in code_object_pickler_test.py

tvalentyn

Thanks!

tvalentyn · 2025-08-14T16:47:36Z

@@ -60,15 +65,13 @@ def get_normalized_path(path):


 def get_code_path(callable: types.FunctionType):


(totally optional) wdyt about naming this method:
get_code_object_identifier

…to facilitate more efficient pickling of lambdas. (apache#35656) * Update code_object_pickler.py to include consistent pickling for lambdas * Create code_object_pickler_test.py * Update code_object_pickler_test.py * Create after_module_add_function.py * Create after_module_add_lambda_variable.py * Create after_module_add_variable.py * Create after_module_remove_lambda_variable.py * Create after_module_remove_variable.py * Create after_module_with_classes.py * Create after_module_with_global_variable.py * Create after_module_with_nested_function.py * Create after_module_with_nested_function_2.py * Create after_module_with_single_class.py * Create before_module_with_classes.py * Create before_module_with_functions.py * Create before_module_with_lambdas.py * Create module_with_default_argument.py * Create pickle_code_path_test.py * Update code_object_pickler.py * Update pickle_code_path_test.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * Update after_module_add_function.py * Update after_module_add_lambda_variable.py * Update after_module_add_variable.py * Update after_module_remove_lambda_variable.py * Update after_module_remove_variable.py * Update after_module_with_classes.py * Update after_module_with_global_variable.py * Update after_module_with_nested_function.py * Update after_module_with_nested_function_2.py * Update after_module_with_single_class.py * Update before_module_with_classes.py * Update before_module_with_functions.py * Update before_module_with_lambdas.py * Update module_with_default_argument.py * Update after_module_add_lambda_variable.py * Update after_module_with_nested_function.py * Update code_object_pickler.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler.py * Update pickle_code_path_test.py * Update after_module_with_nested_function.py * Update after_module_with_classes.py * Update after_module_with_nested_function_2.py * Update code_object_pickler_test.py * Update code_object_pickler.py * Update pickle_code_path_test.py * Update pickle_code_path_test.py * Update code_object_pickler.py * Update after_module_with_classes.py * Update after_module_with_nested_function.py * Update after_module_with_nested_function_2.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * Update pickle_code_path_test.py * Create __init__.py * Update before_module_with_classes.py * Update after_module_with_single_class.py * Update after_module_with_classes.py * Update pickle_code_path_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * Update after_module_with_nested_function.py * Update after_module_with_classes.py * Update after_module_with_single_class.py * Update before_module_with_functions.py * Update after_module_add_lambda_variable.py * Update after_module_with_global_variable.py * Update after_module_with_nested_function_2.py * Update after_module_add_function.py * Update before_module_with_lambdas.py * Update before_module_with_classes.py * Update after_module_add_variable.py * Update code_object_pickler.py * Update pickle_code_path_test.py * Update after_module_add_function.py * Update after_module_add_lambda_variable.py * Update after_module_add_function.py * Update after_module_add_function.py * Update after_module_add_variable.py * Update after_module_remove_lambda_variable.py * Update after_module_remove_variable.py * Update after_module_with_classes.py * Update after_module_with_global_variable.py * Update after_module_with_nested_function.py * Update after_module_with_nested_function_2.py * Update after_module_with_single_class.py * Update before_module_with_classes.py * Update before_module_with_functions.py * Update before_module_with_lambdas.py * Update module_with_default_argument.py * Update after_module_with_classes.py * Update before_module_with_classes.py * Update code_object_pickler_test.py * Update after_module_add_function.py * Update pickle_code_path_test.py * Update code_object_pickler_test.py * Update code_object_pickler.py * Update code_object_pickler_test.py * Update pickle_code_path_test.py * fix formatting and lint * formatting and lint changes * Update code_object_pickler.py * change type hints * fix typo that caused error * fix formatting and lint errors * fix typehints * fix formatting and lint * Update code_object_pickler.py * Update code_object_pickler.py * fix union error * import union * update code_object_pickler.py docstring, move pickle_code_path_test.py into code_object_pickler_test.py, rename files in test_data, rename tests in code_object_pickler_test.py * fix error * rename functions in code_object_pickler.py * fix errors * fix formatting and lint errors * Update sdks/python/apache_beam/internal/code_object_pickler.py --------- Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

* implement lambda name pickling in cloudpickle * add enable_lambda_name to __init__ * fix formatting and lint * fix typo * fix code paths in test * fix tests * fix lint * fix formatting and failing test * fix formatting again * remove cloudpickle implementation to leave only typo fixes and fixing test structure. * fix _make_function typo * revert regex * fix failing tests * fix formatting * update prefix to not hardcode

* sdks/python: properly make milvus as extra dependency * sdks/python: update image requirements * .github: trigger postcommit python * sdks/python: fix linting issues * sdks/python: fix formatting issues * .github: trigger beam postcommit python * sdks/python: revert milvus version in itests * sdks/python: update image requirements * trigger_files: trigger postcommit python * Bump github.com/docker/go-connections from 0.5.0 to 0.6.0 in /sdks (#35906) Bumps [github.com/docker/go-connections](https://github.com/docker/go-connections) from 0.5.0 to 0.6.0. - [Commits](docker/go-connections@v0.5.0...v0.6.0) --- updated-dependencies: - dependency-name: github.com/docker/go-connections dependency-version: 0.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add the readme link to new YAML examples (#35941) * Bump google.golang.org/api from 0.247.0 to 0.248.0 in /sdks (#35969) * Remove mysql-connector-python dependency (#35932) * Fix typos and update test implementation from #35656 (#35958) * implement lambda name pickling in cloudpickle * add enable_lambda_name to __init__ * fix formatting and lint * fix typo * fix code paths in test * fix tests * fix lint * fix formatting and failing test * fix formatting again * remove cloudpickle implementation to leave only typo fixes and fixing test structure. * fix _make_function typo * revert regex * fix failing tests * fix formatting * update prefix to not hardcode * feat(mongodb): upgrade MongoDB Java driver to version 5.5.0 (#35946) * feat(mongodb): upgrade MongoDB Java driver to version 5.5.0 Update MongoDB Java driver from 3.12.11 to 5.5.0 and refactor code to use new API Add mongo-bson dependency required by new driver version Replace deprecated MongoClient with MongoClients and update GridFS implementation * refactor(mongodb): update MongoDB client usage to modern API Replace deprecated MongoClient with MongoClients.create() and update database drop method * build(dependencies): add mongodb driver core dependency Add mongodb-driver-core to support MongoDB Java driver functionality. Also mark mongo_java_driver as permitUnusedDeclared and add testImplementation. * fix(mongodb): update embedded mongo version and fix split key filtering Update embedded MongoDB test dependency to version 3.5.4 and simplify split key filtering logic by using BsonObjectId for range queries. This ensures proper type handling when filtering MongoDB documents by _id field. * build: add mongodb-driver-core dependency Add mongodb-driver-core version 5.5.0 to support MongoDB Java driver functionality * use version * refactor: simplify mongo client creation logic Remove redundant null check and consolidate uri handling in MongoDbGridFSIO * Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks (#35974) Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.18.6 to 1.18.7. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.18.7/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@config/v1.18.6...config/v1.18.7) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.18.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google.golang.org/grpc from 1.74.2 to 1.75.0 in /sdks (#35971) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.74.2 to 1.75.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.74.2...v1.75.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.75.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Override localhost endpoint when a worker is running in docker on mac (#35964) * fix(parquetio): handle missing nullable fields in row conversion (#35948) * fix(parquetio): handle missing nullable fields in row conversion Add null value handling when converting rows to Arrow tables for nullable fields that are missing from input data. This fixes KeyError when writing to Parquet with missing nullable fields, addressing issue #35791. * fix lint * Bump cloud.google.com/go/storage from 1.56.0 to 1.56.1 in /sdks (#35980) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.56.0 to 1.56.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.56.0...storage/v1.56.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.56.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [Prism] Fix segv when docker container self-terminated. (#35977) * Fix segv when docker container is self-terminated * Add some debug logging for docker and process env. * add a jinja % include/import pipeline example to docs (#35931) * add a jinja include pipeline example * update yaml doc with import example * address gemini and other comments * fix table of contents for readme * add link to jinja pipeline examples * Bump github.com/aws/aws-sdk-go-v2/config from 1.31.2 to 1.31.3 in /sdks (#35983) * Add a security GCP log analyzer (#35922) * Add the base log_analyzer * Add github action for security logging * Enhance LogAnalyzer to filter logs by time range and include file names in event summary * Add dry-run option for weekly email report generation in LogAnalyzer * Better error handling for timezones and missing details * Refactor LogAnalyzer to use SinkCls for type consistency and enhance bucket permission management for log sinks * update py containers (#35982) * [YAML]: add import jinja pipeline example (#35945) * add import jinja pipeline example * revert name change * update overall examples readme * fix lint issue * fix gemini small issue * Update sdks/python/apache_beam/yaml/examples/transforms/jinja/import/README.md --------- Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com> * workflows: capture DinD tests in PreCommit Py Coverage workflow * workflows: temporarily removing `ubuntu-latest` till resolving deps * workflows: add `matrix.os` label to `beam_PreCommit_Python_Coverage` --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Mohamed Awnallah <mohamedmohey2352@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chamikara Jayalath <chamikaramj@gmail.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: kristynsmith <kristynsmith@google.com> Co-authored-by: liferoad <huxiangqian@gmail.com> Co-authored-by: Shunping Huang <shunping@google.com> Co-authored-by: Derrick Williams <derrickaw@google.com> Co-authored-by: Enrique Calderon <71863693+ksobrenat32@users.noreply.github.com> Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

kristynsmith added 20 commits July 22, 2025 12:05

Update code_object_pickler.py to include consistent pickling for lambdas
74f690a

Create code_object_pickler_test.py
23fa4d8

Update code_object_pickler_test.py
86e12b2

Create after_module_add_function.py
d17d08a

Create after_module_add_lambda_variable.py
6b66de9

Create after_module_add_variable.py
a1106e6

Create after_module_remove_lambda_variable.py
bbe555c

Create after_module_remove_variable.py
4a8d350

Create after_module_with_classes.py
3941819

Create after_module_with_global_variable.py
c2ab713

Create after_module_with_nested_function.py
ca853c2

Create after_module_with_nested_function_2.py
8c26a6c

Create after_module_with_single_class.py
40b767c

Create before_module_with_classes.py
251b1d7

Create before_module_with_functions.py
3590211

Create before_module_with_lambdas.py
6169dbe

Create module_with_default_argument.py
03ba258

Create pickle_code_path_test.py
f1ce3ca

Update code_object_pickler.py
2e9ff8b

Update pickle_code_path_test.py
d940cfa

github-actionsBot added the python label Jul 22, 2025

kristynsmith added 9 commits July 22, 2025 14:06

Update code_object_pickler_test.py
0438ba4

Update pickle_code_path_test.py
30ece4b

Update after_module_add_function.py
3dc6856

Update after_module_add_lambda_variable.py
74cb523

Update after_module_add_variable.py
82e5656

Update after_module_remove_lambda_variable.py
0c0cf87

Update after_module_remove_variable.py
50011e4

Update after_module_with_classes.py
35f2dfe

Update after_module_with_global_variable.py
256a010

tvalentyn reviewed Aug 6, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/internal/code_object_pickler.py Outdated

tvalentyn reviewed Aug 6, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/internal/code_object_pickler.py Outdated

kristynsmith added 6 commits August 7, 2025 11:12

change type hints
cd83d96

fix typo that caused error
9660d41

fix formatting and lint errors
6410707

fix typehints
d49f850

fix formatting and lint
8f4344b

Update code_object_pickler.py
a4ebbd5

tvalentyn reviewed Aug 11, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/internal/code_object_pickler.py Outdated

kristynsmith closed this Aug 12, 2025

kristynsmith reopened this Aug 12, 2025

kristynsmith added 3 commits August 12, 2025 15:31

Update code_object_pickler.py
bac70eb

fix union error
7be70e6

import union
eca023d

tvalentyn reviewed Aug 13, 2025
View reviewed changes

tvalentyn reviewed Aug 14, 2025
View reviewed changes

kristynsmith added 2 commits August 14, 2025 11:47

update code_object_pickler.py docstring, move pickle_code_path_test.p…
ff59041
…y into code_object_pickler_test.py, rename files in test_data, rename tests in code_object_pickler_test.py

fix error
2a14db5

tvalentyn approved these changes Aug 14, 2025
View reviewed changes

kristynsmith added 3 commits August 14, 2025 14:07

rename functions in code_object_pickler.py
81f5a33

fix errors
a17c69d

fix formatting and lint errors
7ee69f1

tvalentyn approved these changes Aug 15, 2025
View reviewed changes

Comment threadsdks/python/apache_beam/internal/code_object_pickler.py Outdated

Update sdks/python/apache_beam/internal/code_object_pickler.py
a9838a1

tvalentyn merged commit afe94ad into apache:masterAug 15, 2025
81 of 86 checks passed

tvalentyn changed the title ~~Add support for lambda name pickling.~~Adds helper functions for computing and using code object identifiers to facilitate more efficient pickling of lambdas. (#35656)Aug 15, 2025

kristynsmith mentioned this pull request Aug 26, 2025
Fix typos and update test implementation from #35656 #35958
Merged
3 tasks

		@@ -60,15 +65,13 @@ def get_normalized_path(path):


		def get_code_path(callable: types.FunctionType):

Uh oh!

Conversation

kristynsmith commented Jul 22, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tvalentyn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tvalentynAug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants