Skip to content

fix(gcp): bound the state bucket backups prefix, not just noncurrent versions - #386

Merged
Cre-eD merged 1 commit into
mainfrom
feat/state-bucket-backups-lifecycle
Aug 21, 2026
Merged

fix(gcp): bound the state bucket backups prefix, not just noncurrent versions#386
Cre-eD merged 1 commit into
mainfrom
feat/state-bucket-backups-lifecycle

Conversation

@Cre-eD

@Cre-eD Cre-eD commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Why

The noncurrent-generation lifecycle rule bounds half of a state bucket's history. The half it leaves unbounded is the expensive one.

Pulumi writes each per-update backup under .pulumi/backups as its own live object. It never becomes noncurrent, so a DaysSinceNoncurrentTime rule never reaches it. Measured on a real state bucket: 11,947 such objects reaching back to the day the bucket was created, none ever eligible for deletion.

Storage is not the issue — state files are small, that prefix was 7.5 GB. The issue is that each of those objects carries the stack's data key wrapped under whichever KMS key version was primary when it was written. An unbounded backups prefix therefore keeps nearly every key version alive and billable. On that bucket it pinned roughly 7,000 versions, about $420/mo, none of it reclaimable while the prefix grows without limit.

That re-wrap was confirmed rather than assumed: sampling one stack's backups across five dates and hashing secrets_providers.state.encryptedkey gave five distinct ciphertexts.

The rule

StateHistoryLifecycle now emits two rules — the existing archived-only rule, plus an age-based rule scoped to the backups prefix.

The second one matches live objects, which is only safe while it is scoped. Unscoped it would delete the current state file of every stack in the bucket. So it is emitted only when the prefix constant is non-empty: a refactor that loses the scope produces no rule rather than a catastrophic one. Two tests assert that invariant — one per-rule, one across all rules independent of ordering or count, so a future third rule cannot slip in unscoped.

Why the guard had to change too

ShouldApplyNoncurrentVersionLifecycle applies a policy only to buckets with no lifecycle rules at all. That is deliberate and correct as a way of never editing an operator's policy — but it means this fix could never reach the buckets that have the problem, because any bucket this package has already touched now carries exactly one rule: the one it wrote.

ShouldApplyStateHistoryLifecycle adds a single case: a bucket whose rules were all written by this package is upgradable. Recognition is by exact rule shape, not by day count, since the count comes from operator config and may legitimately have changed since it was written.

The original property is intact, and tested:

  • a bucket carrying any rule this package did not write is left completely alone;
  • so is a bucket where the backups prefix is already bounded by other means, including by a broader prefix such as .pulumi/;
  • versioning off, retention 0, negative retention and nil attrs all still refuse to write.

ShouldApplyNoncurrentVersionLifecycle is kept for compatibility and marked deprecated with the reason it is too narrow to use alone.

Verification

  • Full package suite green, including the pre-existing lifecycle tests unchanged.
  • Every new guard mutation-checked — removed one at a time, confirmed a test fails:
    • dropping the prefix scoping → fails
    • guard accepting any existing policy → fails on "an operator's own rule is never touched"
    • ignoring an operator's own backups rule → fails
    • backups rule never emitted (the original half-fix) → fails
  • go build ./..., go vet, gofmt all clean.

Compatibility and blast radius

Additive. No exported symbol removed or changed in signature. New buckets get both rules at create time; existing buckets get the second rule only if this package wrote everything already there.

The one behaviour change worth calling out explicitly: a bucket whose only rule was written by an older version of this package will now be updated rather than skipped. That is the entire point of the change, but it does mean the write is no longer restricted to empty policies, so it is worth a reviewer's attention.

Anyone who wants the old behaviour can bound the prefix themselves or set retention to 0, both of which this now respects.

…rrent versions

The noncurrent-generation rule left half the history unbounded, and the half it
left is the expensive half.

Pulumi writes each per-update backup under .pulumi/backups as its own LIVE
object. It never becomes noncurrent, so a DaysSinceNoncurrentTime rule never
reaches it. Measured on a real state bucket 2026-08-20: 11,947 such objects
reaching back to the day the bucket was created, none of them ever eligible for
deletion.

Storage is not the problem; state files are small. The problem is that every one
of those objects carries the stack's data key wrapped under whichever KMS key
version was primary when it was written, so an unbounded backups prefix keeps
nearly every key version alive and billable. On that bucket it pinned roughly
7,000 versions, about $420/mo, none of it reclaimable while the prefix grows
without limit. Sampling one stack's backups across five dates gave five distinct
wrapped-key ciphertexts, confirming the per-write re-wrap rather than assuming
it.

StateHistoryLifecycle therefore emits two rules. The second is age-based and
matches LIVE objects, which is only safe because it is scoped to the backups
prefix, so it is emitted only when that prefix is non-empty: a refactor that
loses the scope produces no rule rather than a rule that deletes the current
state file of every stack in the bucket. Tests assert that invariant twice, once
per rule and once across all rules independent of ordering.

ShouldApplyStateHistoryLifecycle also had to change, because otherwise this fix
could never reach the buckets that have the problem. The existing guard applies
a policy only to buckets with no lifecycle rules at all, deliberately, so that
an operator's policy is never edited. But any bucket Simple Container has
already touched now carries exactly one rule, the one it wrote, so that guard
excludes precisely the population needing the second rule.

The new guard adds one case: a bucket whose rules were ALL written by this
package is upgradable. Recognition is by exact rule shape rather than by day
count, since the count comes from operator config and may legitimately have
changed since it was written. The original property is intact: a bucket carrying
any rule this package did not write is left completely alone, and so is a bucket
where the backups prefix is already bounded by other means, including by a
broader prefix such as .pulumi/.

ShouldApplyNoncurrentVersionLifecycle is kept for compatibility and marked
deprecated, with the reason it is too narrow to use on its own.

Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
@github-actions

Copy link
Copy Markdown

Semgrep Scan Results

Repository: api | Commit: 2fd9d82

Check Status Details
⚠️ Semgrep Warning 2 warning(s), 6 total

Scanned at 2026-08-21 04:07 UTC

@github-actions

Copy link
Copy Markdown

Security Scan Results

Repository: api | Commit: 2fd9d82

Check Status Details
✅ Secret Scan Pass No secrets detected
⚠️ Dependencies (Trivy) High 2 high, 4 total
⚠️ Dependencies (Grype) High 2 high, 4 total
📦 SBOM Generated 523 components (CycloneDX)

Scanned at 2026-08-21 04:08 UTC

@github-actions

Copy link
Copy Markdown

📊 Statement coverage

Measured on the documented included set (see docs/TESTING.md → Coverage scope). Observe-only — no regression gate is enforced yet.

Scope This PR main baseline Δ
Included set (Gold-tier denominator) 89.6% 89.6% +0.0 pp
Full set (whole repo, transparency) 28.9% 28.9% +0.0 pp

Baseline: main @ ec0f628

@Cre-eD
Cre-eD merged commit 0e40fff into main Aug 21, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants