Skip to content

[Jetstream] Iceberg + Glue setup #132

Description

@dudu-theman

Part of #138. Design doc: docs/design_docs/2026-07-13_bluesky_ingestion_jetstream.md

Create Iceberg tables for Bluesky posts, likes, reposts, and follows, partitioned by created_at_day under the proposed s3://lab-data-integrations-interface/bluesky/raw/<type>/ layout.

  • One table per (platform, data_type), since column sets differ per type and users query one platform at a time.
  • Iceberg owns the schema; Glue just points at the Iceberg metadata file.
  • Chosen over partition projection because it works as well and sets us up better for deduplication.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions