fix(gcp): bound the state bucket backups prefix, not just noncurrent versions - #386
Merged
Merged
Conversation
…rrent versions The noncurrent-generation rule left half the history unbounded, and the half it left is the expensive half. Pulumi writes each per-update backup under .pulumi/backups as its own LIVE object. It never becomes noncurrent, so a DaysSinceNoncurrentTime rule never reaches it. Measured on a real state bucket 2026-08-20: 11,947 such objects reaching back to the day the bucket was created, none of them ever eligible for deletion. Storage is not the problem; state files are small. The problem is that every one of those objects carries the stack's data key wrapped under whichever KMS key version was primary when it was written, so an unbounded backups prefix keeps nearly every key version alive and billable. On that bucket it pinned roughly 7,000 versions, about $420/mo, none of it reclaimable while the prefix grows without limit. Sampling one stack's backups across five dates gave five distinct wrapped-key ciphertexts, confirming the per-write re-wrap rather than assuming it. StateHistoryLifecycle therefore emits two rules. The second is age-based and matches LIVE objects, which is only safe because it is scoped to the backups prefix, so it is emitted only when that prefix is non-empty: a refactor that loses the scope produces no rule rather than a rule that deletes the current state file of every stack in the bucket. Tests assert that invariant twice, once per rule and once across all rules independent of ordering. ShouldApplyStateHistoryLifecycle also had to change, because otherwise this fix could never reach the buckets that have the problem. The existing guard applies a policy only to buckets with no lifecycle rules at all, deliberately, so that an operator's policy is never edited. But any bucket Simple Container has already touched now carries exactly one rule, the one it wrote, so that guard excludes precisely the population needing the second rule. The new guard adds one case: a bucket whose rules were ALL written by this package is upgradable. Recognition is by exact rule shape rather than by day count, since the count comes from operator config and may legitimately have changed since it was written. The original property is intact: a bucket carrying any rule this package did not write is left completely alone, and so is a bucket where the backups prefix is already bounded by other means, including by a broader prefix such as .pulumi/. ShouldApplyNoncurrentVersionLifecycle is kept for compatibility and marked deprecated, with the reason it is too narrow to use on its own. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Cre-eD
requested review from
Laboratory,
smecsia and
universe-ops
as code owners
August 21, 2026 04:06
Semgrep Scan ResultsRepository:
Scanned at 2026-08-21 04:07 UTC |
Security Scan ResultsRepository:
Scanned at 2026-08-21 04:08 UTC |
📊 Statement coverageMeasured on the documented included set (see
Baseline: |
smecsia
approved these changes
Aug 21, 2026
universe-ops
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The noncurrent-generation lifecycle rule bounds half of a state bucket's history. The half it leaves unbounded is the expensive one.
Pulumi writes each per-update backup under
.pulumi/backupsas its own live object. It never becomes noncurrent, so aDaysSinceNoncurrentTimerule never reaches it. Measured on a real state bucket: 11,947 such objects reaching back to the day the bucket was created, none ever eligible for deletion.Storage is not the issue — state files are small, that prefix was 7.5 GB. The issue is that each of those objects carries the stack's data key wrapped under whichever KMS key version was primary when it was written. An unbounded backups prefix therefore keeps nearly every key version alive and billable. On that bucket it pinned roughly 7,000 versions, about $420/mo, none of it reclaimable while the prefix grows without limit.
That re-wrap was confirmed rather than assumed: sampling one stack's backups across five dates and hashing
secrets_providers.state.encryptedkeygave five distinct ciphertexts.The rule
StateHistoryLifecyclenow emits two rules — the existing archived-only rule, plus an age-based rule scoped to the backups prefix.The second one matches live objects, which is only safe while it is scoped. Unscoped it would delete the current state file of every stack in the bucket. So it is emitted only when the prefix constant is non-empty: a refactor that loses the scope produces no rule rather than a catastrophic one. Two tests assert that invariant — one per-rule, one across all rules independent of ordering or count, so a future third rule cannot slip in unscoped.
Why the guard had to change too
ShouldApplyNoncurrentVersionLifecycleapplies a policy only to buckets with no lifecycle rules at all. That is deliberate and correct as a way of never editing an operator's policy — but it means this fix could never reach the buckets that have the problem, because any bucket this package has already touched now carries exactly one rule: the one it wrote.ShouldApplyStateHistoryLifecycleadds a single case: a bucket whose rules were all written by this package is upgradable. Recognition is by exact rule shape, not by day count, since the count comes from operator config and may legitimately have changed since it was written.The original property is intact, and tested:
.pulumi/;0, negative retention andnilattrs all still refuse to write.ShouldApplyNoncurrentVersionLifecycleis kept for compatibility and marked deprecated with the reason it is too narrow to use alone.Verification
go build ./...,go vet,gofmtall clean.Compatibility and blast radius
Additive. No exported symbol removed or changed in signature. New buckets get both rules at create time; existing buckets get the second rule only if this package wrote everything already there.
The one behaviour change worth calling out explicitly: a bucket whose only rule was written by an older version of this package will now be updated rather than skipped. That is the entire point of the change, but it does mean the write is no longer restricted to empty policies, so it is worth a reviewer's attention.
Anyone who wants the old behaviour can bound the prefix themselves or set retention to
0, both of which this now respects.