Skip to content

[AMORO-4359][AMS] Support AWS Secrets Manager as a ConfigShade - #4360

Open
wangxianghu wants to merge 1 commit into
apache:masterfrom
wangxianghu:aws-sm-config-shade
Open

[AMORO-4359][AMS] Support AWS Secrets Manager as a ConfigShade#4360
wangxianghu wants to merge 1 commit into
apache:masterfrom
wangxianghu:aws-sm-config-shade

Conversation

@wangxianghu

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Close#4359.

AMS keeps sensitive values such as admin-password and database.password in
config.yaml. Today the only shade providers are base64 and the built-in
default, so operators still have to store the real secret (obfuscated at best) on disk. When AMS runs on AWS, the natural place for these secrets is AWS Secrets
Manager, with credentials resolved from the environment (EKS IRSA, ECS/EC2
instance profile, etc.) rather than checked into a file.

This PR adds an aws-smConfigShade implementation so that a sensitive config
value can be an AWS Secrets Manager ARN, and AMS resolves the real value at
startup:

ams: shade: identifier: aws-sm sensitive-keywords: database.username;database.password database: # both point to the same JSON secret; '#<field>' selects the value out of it username: arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:prod-amoro-db-mOhyOp#db.username password: arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:prod-amoro-db-mOhyOp#db.password 
  • The region is resolved from the ARN — no extra config.
  • A trailing # selects one field when the secret value is JSON;
    omit it when the secret value is the plaintext itself.
  • Failures (malformed ARN, fetch failure, missing field) fail fast so AMS never
    starts with a wrong secret.

Brief change log

  • Add AwsSecretsManagerConfigShade (identifier = aws-sm): resolves a secret
    ARN into its value, extracting a JSON field when # is given.
  • Add SecretReference: parses arn:aws:secretsmanager::㊙️[#],
    extracting the region from the ARN.
  • Add AwsSecretsManagerClient + DefaultAwsSecretsManagerClient: a thin,
    testable wrapper over the AWS SDK using the URL-connection HTTP client (no
    Netty), the default credentials provider chain, and per-process caching.
  • A single client is created lazily and memoized; all shaded secrets are
    expected to be in one region, and a cross-region reference fails fast.
  • Register AwsSecretsManagerConfigShade in the ConfigShade SPI file.
  • Add software.amazon.awssdk:secretsmanager (Netty/apache-client excluded) and
    url-connection-client dependencies.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

    Added TestSecretReference (ARN/region/#field parsing, whitespace, and
    every malformed-input rejection) and TestAwsSecretsManagerConfigShade
    (plaintext vs JSON-field decryption, dotted field names taken literally rather
    than as a nested path, client memoization, cross-region fail-fast, and the
    invalid-JSON / missing-field / malformed-ARN failure paths) — 23 tests, all
    passing.

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes )
  • If yes, how is the feature documented? (JavaDocs)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support AWS Secrets Manager as a ConfigShade

1 participant

@wangxianghu