Skip to content

[Jetstream] Compaction + orphan file cleanup #135

Description

@dudu-theman

Part of #138. Design doc: docs/design_docs/2026-07-13_bluesky_ingestion_jetstream.md

Periodic Iceberg compaction and unused/orphan file deletion driven by an AWS EventBridge rule, to keep merge-on-read query costs down.

Cost: EventBridge is essentially $0.00; Athena is required to run the compaction at $5.00/TB.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions