Skip to content

[python] Add Secret management module in apache_beam.utils.secret - #39636

Merged
shunping merged 5 commits into
apache:masterfrom
shunping:secret-manager-1
Aug 18, 2026
Merged

[python] Add Secret management module in apache_beam.utils.secret#39636
shunping merged 5 commits into
apache:masterfrom
shunping:secret-manager-1

Conversation

@shunping

@shunpingshunping commented Aug 5, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Introduces apache_beam.utils.secret to centralize secret management in the Python SDK.

Changes

  • Derived a new Secret base class and implementations (RawSecret, GcpSecret, and GcpHsmGeneratedSecret) from the existing secret classes defined in util.py used by GBEK.
  • Added factory methods Secret.from_spec() and Secret.from_option_string().
  • Re-exported classes in apache_beam.transforms.util for backwards compatibility.
  • Added unit tests in secret_test.py and updated util_test.py.

github-advanced-security[bot]
github-advanced-securityBot found potential problems Aug 5, 2026
View reviewed changes
Comment threadsdks/python/apache_beam/utils/secret.py Fixed
Comment threadsdks/python/apache_beam/utils/secret.py Dismissed
@shunping

shunping commented Aug 5, 2026

Copy link
Copy Markdown
CollaboratorAuthor

r: @damccorm

@shunping
shunping marked this pull request as ready for review August 5, 2026 19:31
@github-actions

github-actionsBot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

damccorm
damccorm reviewed Aug 7, 2026
View reviewed changes
Comment threadsdks/python/apache_beam/utils/secret.py Outdated
Comment threadsdks/python/apache_beam/transforms/util_test.py Outdated
Comment threadsdks/python/apache_beam/transforms/util.py
@shunping
Refactor Secret classes and tests into apache_beam.utils.secret
fba1e7e
There is no functional changes in this commit. We also re-export
the secret classes in apache_beam.transforms.util for backward
compatibility.
@shunping
Add secret caching, string accessors, and __getstate__
d4c38b6
@shunping
shunping requested a review from damccormAugust 7, 2026 16:15
damccorm
damccorm reviewed Aug 17, 2026
View reviewed changes

@damccormdamccorm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - just a few more minor comments, otherwise LGTM

Comment threadsdks/python/apache_beam/utils/secret.py Outdated
Comment threadsdks/python/apache_beam/utils/secret.py

@abc.abstractmethod
def get_secret_bytes(self) -> bytes:
"""Returns the secret as a byte string."""

@damccormdamccormAug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we update this to recommend callers use get_bytes instead for caching? I think it is a good addition.

@shunpingshunpingAug 18, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that make sense, but I would prefer to do it in a separate PR. I want to avoid complicating this current PR so that if there are any issues, they remain easy to trace. Does that sound ok?

@damccormdamccormAug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, I'm just suggesting a comment-only change. But pushing to a future pr is fine if you'd prefer, I don't think this comment should be blocking anyways

@shunpingshunpingAug 18, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I thought you were talking about changing the references of get_secret_bytes to get_bytes in GBEK:

Comment threadsdks/python/apache_beam/utils/secret.py
"""
return self.get_bytes(cacheSecret=cacheSecret).decode("utf-8")

def get_bytes(self, cacheSecret: bool = False) -> bytes:

@damccormdamccormAug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can scope this out of this PR, but maybe we should allow users to specify a cache_ttl_seconds in init as well

@shunpingshunpingAug 18, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Let's add a feature request after this PR is submitted.

@shunping
shunping merged commit 52702a1 into apache:masterAug 18, 2026
100 checks passed
This was referenced Aug 19, 2026
Merged
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@damccormdamccormdamccorm approved these changes
+1 more reviewer
@github-advanced-securitygithub-advanced-security[bot]github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shunping@damccorm@github-advanced-security