Uh oh!
There was an error while loading. Please reload this page.
Expire stale refs in ExpireSnapshots - #3246
Conversation
73e24a0 to
8373d34Compare| MIN_SNAPSHOTS_TO_KEEP = "history.expire.min-snapshots-to-keep" | ||
| MIN_SNAPSHOTS_TO_KEEP_DEFAULT = 1 | ||
| MAX_REF_AGE_MS = "history.expire.max-ref-age-ms" |
There was a problem hiding this comment.
this is not used in this PR but I thought it could be useful for callers of ExpireSnapshot to get the default value for max-ref-age-ms from table properties (like in the Java implementation)
| for snapshot in self._transaction.table_metadata.snapshots: | ||
| if snapshot.timestamp_ms < expire_from and snapshot.snapshot_id not in protected_ids: | ||
| self._snapshot_ids_to_expire.add(snapshot.snapshot_id) | ||
| self._expire_older_than_ms = datetime_to_millis(dt) |
There was a problem hiding this comment.
moved most of the logic to _commit() to avoid order-dependence between older_than() and
remove_expired_refs() calls
| table_v2.catalog.commit_table.return_value = mock_response | ||
| # Remove any refs that protect the snapshots to be expired | ||
| table_v2.metadata = table_v2.metadata.model_copy( |
There was a problem hiding this comment.
removed because it duplicates old line 205?
b45c6fe to
dcfdd04Comparedcfdd04 to
ad0a92dCompareThis pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
Changes
remove_expired_refs(default_max_ref_age_ms)method onExpireSnapshots: removes branches/tags whose snapshot age exceeds theirmax_ref_age_ms(per-ref setting, with fallback to the method parameter);mainis never removedolder_than()is now lazy — threshold is evaluated in_commit()so that call order withremove_expired_refs()does not affect the result_get_protected_snapshot_ids()to exclude refs marked for removal, so their orphaned snapshots become eligible for age-based expiryMotivation
Mirrors Java's
RemoveSnapshots.computeRetainedRefs()behavior. Previously,ExpireSnapshotsnever removed stale refs even whenmax-ref-age-mswas configured on a branch or tag.This PR was AI-assisted.