Skip to content

Support Secret Manager in JdbcIO for Java, Python and YAML - #39834

Merged
shunping merged 4 commits into
apache:masterfrom
shunping:secret-manager-3
Aug 21, 2026
Merged

Support Secret Manager in JdbcIO for Java, Python and YAML#39834
shunping merged 4 commits into
apache:masterfrom
shunping:secret-manager-3

Conversation

@shunping

@shunpingshunping commented Aug 20, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Adds support for resolving database passwords securely using Secret Manager across Java, Python (cross-language), and Beam YAML for JdbcIO.

Follow-up to #39636 and #39806.

Changes

  • Leveraged Secret Manager classes in Java JDBCIO
  • Surfaced secret manager parameter in Python (xlang) and Yaml.
  • Added unit and integration tests.

github-advanced-security[bot]
github-advanced-securityBot found potential problems Aug 20, 2026
View reviewed changes
Comment threadsdks/python/apache_beam/yaml/integration_tests.py Fixed
Comment threadsdks/python/apache_beam/yaml/integration_tests.py Dismissed
Comment threadsdks/python/apache_beam/yaml/integration_tests.py Dismissed
github-advanced-security[bot]
github-advanced-securityBot found potential problems Aug 21, 2026
View reviewed changes
Comment threadsdks/python/apache_beam/yaml/integration_tests.py Dismissed
@shunping
shunping marked this pull request as ready for review August 21, 2026 01:40
@shunping

shunping commented Aug 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

r: @damccorm for java and @derrickaw for xlang and yaml.

@github-actions

github-actionsBot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

damccorm
damccorm reviewed Aug 21, 2026
View reviewed changes

@damccormdamccorm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think there are some improvements we can make to the documentation, but the code itself looks great

Comment thread.../java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcReadSchemaTransformProvider.java Outdated
public abstract String getReadQuery();

@SchemaFieldDescription(
"Secret Manager to use for fetching secret values. For example, GoogleCloudSecretManager.")

@damccormdamccormAug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a few things to improve the docs here:

  1. An example of the full secret manager string and how it is used (this would go in password, not here I guess).
  2. A set of allowed secret managers for the default expansion service.

It might make sense to abstract this out to a small doc since it will probably show up in several places.

@damccormdamccormAug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, the same feedback applies to the yaml doc below.

@shunpingshunpingAug 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added more description and examples for java, python and yaml docs.

@shunping
shunping requested a review from damccormAugust 21, 2026 15:58
damccorm
damccorm approved these changes Aug 21, 2026
View reviewed changes

@damccormdamccorm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@shunping

shunping commented Aug 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

PostCommit YAML Xlang Direct failed on master and it is unrelated to the change here.

@shunping
shunping merged commit 61ed38f into apache:masterAug 21, 2026
176 of 181 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@derrickawderrickawderrickaw approved these changes
@damccormdamccormdamccorm approved these changes
+1 more reviewer
@github-advanced-securitygithub-advanced-security[bot]github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

extensionsiojavajdbcpythonyaml

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@shunping@derrickaw@damccorm@github-advanced-security