diff --git a/docs/rfcs/0000-bigtable-backend.md b/docs/rfcs/0000-bigtable-backend.md new file mode 100644 index 00000000..f0b62af7 --- /dev/null +++ b/docs/rfcs/0000-bigtable-backend.md @@ -0,0 +1,217 @@ +# RFC-0003: Cloud Bigtable Storage Backend for ExtendDB + +- Status: Draft +- Author: @annguy3n +- Created: 2026-06-29 +- Tracking issue: #185 + +## Summary + +This RFC defines the design and implementation of a production-grade Cloud Bigtable storage backend for ExtendDB. It addresses critical architectural requirements for this backend: transaction isolation, stream emissions in transactions, Global Secondary Index (GSI) projections, GSI write consistency, Time-to-Live (TTL) eviction scalability, Optimistic Concurrency Control (OCC) row versioning, fenced two-phase commit (2PC) apply, application profile single-cluster routing, and secure Google Cloud Platform (GCP) connectivity. + +## Motivation + +A naive mapping of DynamoDB structures to Bigtable introduces several architectural challenges that must be addressed to ensure production-readiness: +1. **Transaction Isolation Leak & TOCTOU Races:** Non-transactional (single-row) conditional writes read an item and evaluate conditions without atomic concurrency control, leading to Time-of-Check to Time-of-Use (TOCTOU) races where concurrent writes can overwrite uncommitted state or bypass 2-Phase Commit (2PC) locks. +2. **2PC Race Condition (Read-then-Lock & Unfenced Apply):** Reading row data before placing locks exposes 2PC to TOCTOU races. Furthermore, applying 2PC Phase 2 mutations unconditionally without fencing against expired/aborted locks allows slow coordinators to overwrite newer state. +3. **Missing Transaction Streams:** Multi-row transactions (`TransactWriteItems`) do not emit DynamoDB Stream records. +4. **Inefficient TTL Eviction:** The TTL worker performs full-table scans to identify expired items, which degrades performance on large datasets. +5. **GSI Consistency Gap:** GSI shadow table write failures are logged without retry or repair, causing permanent base-to-shadow divergence. +6. **Multi-Cluster Replication & Conditional Mutations:** Cloud Bigtable rejects conditional mutations (`CheckAndMutateRow`) under multi-cluster routing with eventual consistency. +7. **Auth Configuration Constraints:** Lack of support for explicit credentials file configuration. + +This RFC provides the technical specification to resolve these challenges. + +## Detailed Design + +### 1. Table Mapping & Key Encoding + +#### A. Table Structure & Naming +* **Catalog:** System metadata, accounts, and policies are stored in a single table named `__extenddb_catalog__`. +* **Data Tables:** DynamoDB tables map to Bigtable tables named `t` (where `table_id_hex` is a 32-character hex UUID). Data attributes are stored in column family `d`. +* **GSIs:** Shadow tables are named `t_g` (where `idx_hash` is an 8-character hash of the index name). +* **Transaction Intents & OCC Versions:** Column family `m` in data tables stores lock/intent cells (`m:intent:`) and explicit OCC row versions (`m:v`). +* **TTL Markers:** Column family `t` (reserved). + +> [!NOTE] +> **Naming Constraints:** Table IDs in Cloud Bigtable are limited to 50 characters. To prevent overflow, GSIs are named `t_g` (43 characters total) rather than reproducing the full index name. + +#### B. Exact Decimal & Order-Preserving Key Encoding +Cloud Bigtable sorts row keys by raw byte lexicographical order. The maximum row key size supported by Cloud Bigtable is **4KB (4096 bytes)**, which is strictly enforced during key encoding (`row_key::encode_key`). + +Row keys are encoded using the following structured layout: +```text +Composite Key: [pk_tag:1] [pk_len:u32BE:4] [pk_bytes] [sk_tag:1] [sk_bytes] +PK-Only Key: [pk_tag:1] [pk_len:u32BE:4] [pk_bytes] +``` +Where `pk_tag` and `sk_tag` identify the DynamoDB scalar type (`0x53` for `S`, `0x4E` for `N`, `0x42` for `B`). + +* **Partition (HASH) Keys:** Partition keys use **length-prefixing** (`[pk_len:u32BE:4]`) preceding `[pk_bytes]`. Length prefixing is essential to delimit variable-length partition keys, prevent ambiguity/delimiter collision, and group all items sharing the same partition key contiguously within Bigtable's lexicographical keyspace. +* **Sort (RANGE) Keys:** Sort keys are stored as **raw order-preserving bytes without length prefixing**. Omitting the length prefix on sort keys is strictly mandatory: length-prefixing sort keys would sort primarily by byte-length rather than value, leading to severe sort inversion (for example, `"10"` of length 2 would sort before `"2"` of length 1, or shorter numbers would sort before longer numbers). +* **Number Encoding:** DynamoDB supports numeric values with up to 38 significant digits and requires exact numerical sorting (e.g., `-100 < -1.5 < -1 < 0 < 0.5 < 1 < 1.5 < 100`). Numbers are stored as exact arbitrary-precision decimals (never floating point floats) using an order-preserving byte encoding: + * **Negative numbers:** Sign byte `0x00`, followed by bit-inverted exponent and complement mantissa bytes. + * **Zero:** Exactly `[0x80]`. + * **Positive numbers:** Sign byte `0x81`, followed by big-endian exponent bytes and binary-coded decimal mantissa bytes. +* **GSI Index Row Keys:** Primary keys in GSI shadow tables follow the exact same encoding rules for the index partition and sort keys, formatted as `[gsi_pk_tag][gsi_pk_len][gsi_pk_bytes][gsi_sk_tag][gsi_sk_bytes][base_pk_tag][base_pk_len][base_pk_bytes][base_sk_tag][base_sk_bytes]`, ensuring full sort order preservation and primary key uniqueness. + +### 2. Transaction Rules & Isolation Guarantees + +#### A. Transaction Validation & Idempotency Rules +* **`ClientRequestToken` Idempotency:** `TransactWriteItems` enforces idempotency within a 10-minute sliding window. Requests with an active or recently committed token replay the committed result; conflicting requests with the same token but mismatched payload return `IdempotentParameterMismatchException`. +* **Duplicate Participant Rejection:** A single `TransactWriteItems` request cannot contain multiple operations on the same item. Any transaction with duplicate primary keys is rejected upfront with `ValidationException`. + +#### B. Lock-then-Read 2PC Flow +The `TransactWriteItems` coordinator acquires locks *before* reading data and evaluating condition expressions, and applies mutations via fenced CAS: + +```mermaid +sequenceDiagram + participant C as Coordinator (ExtendDB) + participant B as Bigtable + + C->>B: Phase 1: Open Transaction (Write PENDING to __extenddb_txn_log__) + Note over C,B: Place Intents (Locks) + loop For each participant row + C->>B: Place Intent (CheckAndMutateRow: fail if other fresh intent exists) + end + alt Any Lock Fails + C->>B: Rollback (Clear placed intents, drop txn row) + Note over C: Return TransactionConflict + end + Note over C,B: Read & Verify + C->>B: Read participant rows & capture row versions (m:v) + Note over C: Evaluate Condition Expressions + alt Any Condition Fails + C->>B: Rollback (Clear intents, drop txn row) + Note over C: Return TransactionCanceled + end + C->>B: Phase 2: Commit Transaction (Write COMMITTED to __extenddb_txn_log__) + Note over C,B: Fenced Apply Mutations + loop For each participant row + C->>B: Fenced Apply (CheckAndMutateRow: if m:intent: present, write d, set m:v, delete m:intent) + end + C->>B: Phase 3: Clean up (Write CLEANED to log, then drop txn row) +``` + +#### C. Read Isolation & Concurrency Guarantees +* **Read Committed (No Dirty Reads):** In Phase 1 / Phase 2, the transaction coordinator only writes intent lock markers (`m:intent:`) to column family `m`. Actual data mutations in column family `d` are **only applied in Phase 2 Apply** after the coordinator's `COMMITTED` record is durable in `__extenddb_txn_log__`. Standard reads (`GetItem`, `Query`, `Scan`, `BatchGetItem`) read solely from column family `d` and therefore never observe uncommitted candidate writes that could roll back. +* **Fenced Phase 2 Apply:** During Phase 2 Apply, participant mutations are applied via `CheckAndMutateRow` using a fencing predicate that asserts the coordinator's own intent marker `m:intent:` is still present on the participant row. The atomic mutation payload applies the data changes to family `d`, updates the row version `m:v`, and deletes `m:intent:` in a single atomic RPC. This guarantees that if an intent was timed out or aborted by the recovery sweeper, a slow/stalled coordinator cannot execute an unfenced write over newer state. +* **Serializable `TransactGetItems`:** `TransactGetItems` observes the committed set atomically. Before reading data rows, it checks for active intent markers in column family `m`. If an active lock is present, it coordinates with `__extenddb_txn_log__` to observe the snapshot at the transaction commit boundary, preventing dirty or partial reads. +* **Cross-Row Visibility for Standard Reads:** Standard `Query` or `Scan` operations may observe some participant rows updated before others across separate Bigtable tables during the commit apply window. This is fully compliant with DynamoDB's `Read Committed` isolation model, as every observed row reflects a durable, committed state. + +#### D. Optimistic Concurrency Control (OCC) Row Versioning for Single-Row Writes +All single-row writes (`PutItem`, `UpdateItem`, `DeleteItem`) maintain an explicit row version in column family `m` under qualifier `v` (`u64` monotonic counter or epoch timestamp) and execute via atomic `CheckAndMutateRow` to prevent TOCTOU race conditions and protect 2PC transactions: +* **Version Tracking (`m:v`):** Every row write writes data attributes into column family `d` and increments/sets the row version `m:v`. +* **Conditional Writes (`PutItem`, `UpdateItem`, `DeleteItem` with `ConditionExpression`):** + 1. **Read & Version Capture:** The engine reads the current row from Bigtable, extracting the existing item and capturing `read_version = m:v` (`Option`). + 2. **In-Memory Condition Evaluation:** `evaluate_condition` checks the condition expression against the read image. If false, `StorageError::ConditionFailed` is returned immediately. + 3. **Atomic OCC CAS Mutation:** The write is submitted via `CheckAndMutateRow`: + * **Existing Row (`read_version = Some(v)`):** Predicate filter uses a composite condition: `(no active intent in m) AND (m:v == v)`. Mutations (write `d`, set `m:v = v + 1`, or delete `d` and `m:v`) are placed in `true_mutations`. If the row was modified concurrently or locked, `predicate_matched == false`, and the engine returns `TransactionConflict`. + * **New Row Insert (`read_version = None`):** Predicate filter matches if `(active intent exists in m) OR (m:v exists) OR (family d exists)`. Mutations are placed in `false_mutations`. If the row already exists or is locked, `predicate_matched == true`, and the engine returns `TransactionConflict`. +* **Unconditional Writes:** + * **Predicate Filter:** Matches if an active 2PC intent exists in family `m` (`^intent:.*$` with timestamp $\ge$ `now - intent_timeout`). + * **Mutations:** In `false_mutations`, updates family `d` and sets `m:v = now_micros` (or incremented counter). +* **Deletes:** Use `DeleteFromFamily(d)` and `DeleteFromColumn(m:v)` instead of `DeleteFromRow` to preserve any concurrent 2PC intent markers in family `m`. + +#### E. Cloud Bigtable App Profile & Single-Cluster Routing Requirements +Cloud Bigtable requires single-cluster routing (or multi-cluster routing with single-row transactions enabled) to execute conditional mutations (`CheckAndMutateRow`). Multi-cluster instance routing configured with eventual consistency does not allow conditional mutations and rejects `CheckAndMutateRow` requests with `FailedPrecondition`. +* **Configuration (`app_profile_id`):** `BigtableStorageConfig` provides an optional `app_profile_id` field. When connecting to multi-cluster Bigtable instances, administrators must configure an application profile routed to a single cluster to ensure atomic OCC CAS mutations and 2PC intent locks operate correctly. + +#### F. Garbage Collection (GC) Rules +To prevent disk bloat and ensure fast lookups, tables provisioned by ExtendDB configure explicit Garbage Collection rules on all column families: +* **Data Column Family (`d`):** `MaxVersions(1)` retains only the latest cell version, compacting away historical cell values. +* **Metadata Column Family (`m`):** `MaxVersions(1)` retains only the latest row version `m:v` and intent markers. +* **System Tables:** Catalog table (`__extenddb_catalog__`), transaction log table (`__extenddb_txn_log__`), and TTL index table (`__extenddb_ttl_index__`) are all configured with `MaxVersions(1)`. + +#### G. Recovery Sweeper +A background worker periodically scans `__extenddb_txn_log__` for transactions older than `intent_timeout`: +* **`PENDING` State:** Clear intents on participant rows and delete the coordinator row (rollback). +* **`COMMITTED` State:** Re-apply mutations to participant rows via fenced `apply_fenced_mutation`, write stream records, clear intents, and delete the coordinator row (roll-forward). + +### 3. Stream Emissions in Transactions + +To ensure stream record atomicity: +1. **Phase 1 (Prepare):** Fetch the full `TableDescription` (including `StreamSpecification` and `latest_stream_arn`) and pre-generate the `StreamRecord` payloads. +2. **Phase 2 (Log Enrichment & Commit):** Store the pre-generated stream record payloads and participant mutation payloads in the coordinator row of `__extenddb_txn_log__` alongside `COMMITTED`, and apply stream records to `__extenddb_streams__` concurrently with data mutations. +3. **Recovery:** The recovery sweeper uses the payloads stored in the log to roll forward stream writes on crash. + +### 4. Scalable TTL Indexing & Streams Integration + +Avoid full-table scans by introducing a sharded TTL index table. + +#### A. TTL Index Table Schema +* **Table Name:** `__extenddb_ttl_index__` +* **Row Key format (Binary):** `[shard_id:1] [expiry_timestamp_be:8] [account_id_len:1] [account_id] [table_name_len:1] [table_name] [encoded_base_row_key]` + * `shard_id` (1 byte): `hash(base_row_key) % 16` (prevents write hotspotting). + * `expiry_timestamp_be` (8 bytes): Big-Endian `u64` representing the TTL epoch second. + * `encoded_base_row_key` (variable): Raw row key of the target item. +* **Payload:** None (empty values). +* **GC Rule:** `MaxVersions(1)` on family `d`. + +#### B. Maintenance +* **Writes:** Insert an empty row in `__extenddb_ttl_index__` when writing an item with a TTL attribute. +* **Updates/Deletes:** Delete the old index entry using the item's prior image. + +#### C. Sweeper Flow & Stream Emission +1. Perform parallel range scans across all 16 shards from `start_key = [S] [0]` to `end_key = [S] [current_time_epoch_s + 1]`. +2. For each expired entry, read the base row and verify its current TTL matches the index entry. +3. If valid, execute `delete_item` on the base table. +4. **TTL Stream Records:** When DynamoDB Streams are enabled on the table, TTL deletions emit a `REMOVE` stream record containing `userIdentity.type = "Service"` and `userIdentity.principalId = "dynamodb.amazonaws.com"`, matching native DynamoDB behavior. +5. If invalid (stale index), delete the index entry. + +### 5. GSI Projections & Consistency + +#### A. Projection Validation & ConsistentRead Rejection +Enforce DynamoDB GSI projection behavior: +* **Read Path:** Queries on GSIs with `KEYS_ONLY` or `INCLUDE` projections return only the projected attributes. +* **Validation:** If the client requests `Select: ALL_ATTRIBUTES` (or requests non-projected attributes) on a non-`ALL` GSI, return `ValidationException`. Do not perform base table fetches. +* **ConsistentRead Rejection:** Queries on GSIs with `ConsistentRead = true` are rejected with `ValidationException` (matching DynamoDB API rules). +* **Defaulting:** If `Select` is omitted in a GSI query, default to `Select::AllProjectedAttributes`. + +#### B. Background Reconciler +Implement a reconciler as an internal background worker spawned via `ServerRuntimeHooks::spawn_workers`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows, preventing phantom records from returning during divergence windows. + +### 6. GCP Credentials & Bigtable Configuration + +Extend `BigtableStorageConfig` to support explicit service account files and app profiles: + +```toml +[storage.bigtable] +project_id = "my-project" +instance_id = "my-instance" +data_instance_id = "my-data-instance" # Optional (separates catalog/data) +app_profile_id = "default" # Optional (requires single-cluster routing for CheckAndMutateRow) +credentials_path = "/path/to/sa-key.json" # Optional (service account JSON) +pool_size = 20 +``` + +* **Behavior:** If `credentials_path` is specified, the server programmatically acquires tokens from the service account key. Configure `tonic` channels with `ClientTlsConfig` using native system roots. +* **App Profile:** If `app_profile_id` is specified, data client requests use the designated profile to ensure single-cluster routing for conditional mutations. + +### 7. Future Work & Non-Goals + +* **Backup / PITR / Export (`BackupEngine`):** `BigtableEngine` implements the `extenddb_storage::BackupEngine` supertrait with explicit `StorageError::Internal("Operation not supported")` stubs. Point-in-Time Recovery (PITR), native Bigtable table backups, and Cloud Storage (GCS) exports require dedicated lifecycle orchestration and are deferred to a dedicated follow-up RFC. Note that Cloud Bigtable's native backup retention windows and export capabilities have distinct operational characteristics from DynamoDB's fixed 35-day PITR window. + +## Testing Strategy + +1. **Unit Tests (Emulator-based):** + * Verify exact decimal number key encoding/decoding and numerical sorting up to 38 significant digits. + * Verify row key 4KB size enforcement and partition length prefixing vs raw sort key ordering. + * Verify OCC row versioning (`m:v`) decoding and composite `CheckAndMutateRow` filter construction. + * Verify 2PC coordinator state transitions, fenced apply predicate, and idempotency token deduplication. + * Assert `ValidationException` is returned for invalid GSI attribute requests and `ConsistentRead = true` on GSIs. + * Assert `BackupEngine` methods return `StorageError::Internal("Operation not supported")`. +2. **Failure Integration Tests (Real Bigtable & Emulator):** + * Inject crashes during the 2PC flow and assert that the recovery sweeper rolls forward/back using fenced mutations and preserves stream record writes. + * Run concurrent mixed workloads to verify transactional serializability and OCC conflict detection on single-row writes. + * Verify GSI reconciler heals base-to-shadow mismatches. + * Verify TTL sweeper emits `REMOVE` stream records with the TTL principal and ignores updated items with old index entries. +3. **E2E & Chaos Tests:** + * Run the Python integration suite (`devtools/run-tests --extenddb --pytest`) against production Bigtable. + * Inject network partitions during `TransactWriteItems` to verify 2PC resilience. + +## Alternatives Considered + +* **Unprotected Single-Row Writes:** Bypassing intent checks for single-row writes allows dirty writes, violating serializability. Rejected. +* **Scan-Based TTL Sweeper:** Scanning the entire base table to evict items degrades performance at scale. Rejected. +* **Length-Prefixed Sort Keys:** Length-prefixing sort keys in composite primary keys caused sort inversion where shorter string/numeric representations sorted before longer ones regardless of value. Rejected in favor of raw order-preserving sort key bytes. +* **Cloud Spanner Backend:** Cloud Spanner supports multi-row transactions natively, eliminating the need for application-layer 2PC. This is a viable alternative to be explored as a separate backend connector.