feat(storage): add Route 53 backend (TXT records, base64-encoded items, the works)

[quinnypig]

Forward

I have been arguing in public that Route 53 is a database since 2020. There is a blog post. There are charity shirts. Strangers send me screenshots of YAML files that use TXT records as feature flags. The bit predates this repository's first commit by a comfortable margin.

ExtendDB shipped at v0.1.0 with a pluggable-backend trait, exactly one backend implementation, and a --backend flag that lists Cassandra without irony. I cannot, in good conscience, walk past that.

This PR adds a Route 53 storage backend.

What's in here

A new extenddb-storage-route53 crate behind a route53 cargo feature. Items are encoded as TXT records under a configured hosted zone. Partition keys become subdomain labels. The JSON-serialized item body is base64-encoded and split across the 255-byte strings of a single TXT resource record, with overflow spilling into sibling records keyed by sequence number.

The encoding module (crates/storage-route53/src/encoding.rs) is real Rust with round-trip tests. cargo test -p extenddb-storage-route53 passes.

The Bootstrapper trait is registered. Every method returns OpError::Internal annotated with the Route 53 API call a real implementation would issue: CreateHostedZone, ChangeResourceRecordSets, ListResourceRecordSets, DeleteHostedZone, GetChange. A future implementer has a clear porting map.

Why this is not as deranged as it sounds

Route 53 has properties that, taken in the right order, make the substrate look reasonable:

• One of the few AWS services with a documented 100% query-time SLA, which exceeds the SLA on DynamoDB's own read path.
• ChangeResourceRecordSets returns PENDING and then INSYNC once the change is on every authoritative nameserver. That is a stronger durability story than DynamoDB's documented "strongly consistent reads" contract, which says nothing about multi-region propagation.
• The DNS TTL on a TXT record is the storage TTL. ExtendDB's ttl_deletion_target_seconds setting maps directly onto it. Items vanish from caching resolvers within the TTL without a background sweeper.
• ConsistentRead=false becomes "resolve via any caching DNS resolver." For workloads with a high cache-hit rate, the per-read cost approaches zero. That describes a depressingly large fraction of production DynamoDB tables.
• Reads do not require a SigV4 signature. Any DNS client on the public internet can resolve them. The README's "Multi-cloud" line acquires an additional clause: "or any cloud that can resolve DNS."

Route 53 also has properties that, in any other order, make the substrate look indefensible:

• ChangeResourceRecordSets is rate-limited to five per second per region per AWS account. That is the effective write capacity unit. The batch endpoint amortizes this somewhat.
• The default hosted-zone record limit is 10,000. Raises require a support ticket, which is billed separately.
• 1,000 hosted zones per AWS account by default, which caps the number of "databases" you can run before you have to call your account manager.
• DynamoDB's 400 KB item-size limit is comfortably larger than what fits in a hosted zone for any non-trivial cardinality. Big items should be in S3 with a TXT pointer.

I am genuinely unsure which list is more interesting. I have included both for the reviewer's enjoyment.

Pricing

Component	DynamoDB on-demand	Route 53 backend
Storage	Per GB-month	Flat per hosted zone per month
Writes	Per WCU-second	Per ChangeResourceRecordSets call
Reads (consistent)	Per RCU	Per Route 53 API call
Reads (eventually consistent)	Per RCU	Per million DNS queries; cache hits are free

For read-heavy workloads with high temporal locality this is the rare AWS configuration where unit economics improve with scale rather than punishing it. For write-heavy workloads it is a downgrade. Choosing between the two is left to operations teams who have read this PR and found themselves nodding for reasons they cannot articulate.

Streams

Route 53 exposes a GetChange API that returns the propagation state of a pending change. The streams implementation polls GetChange for each outstanding change ID and emits a stream record when the change flips to INSYNC. ApproximateCreationDateTime maps to SubmittedAt. Shard iteration becomes "list pending changes for this zone in submission order." The mapping is left to the reviewer.

Build matrix

Build	Behavior
cargo build	Unchanged. route53 not registered.
cargo build --features route53	extenddb init --backend route53 reaches the bootstrapper and returns the Route-53-flavored error from the first method called.
cargo test -p extenddb-storage-route53	Two encoding round-trip tests; both pass.

Before / after

Before:

$ extenddb init --backend route53Error: Internal("Unknown backend: route53. Available backends: postgres")

After (built with --features route53):

$ extenddb init --backend route53Error: Internal("is_catalog_initialized: Route 53 backend is registeredbut the relevant operation is not yet implemented. Use --backendpostgres, or wire this method to the corresponding Route 53 API call(referenced inline below). Maps to Route 53 ListResourceRecordSets.")

The user now has both an error and a porting map to the AWS API call a real implementation would invoke.

What's still missing

Piece	State
Cargo crate + workspace member	Done
Bootstrapper impl (registered, stubbed with sourced errors)	Done
encoding module (chunking, base64, round-trip tests)	Done
OperationsEngineRegistration	Not in this PR
StorageConfigRegistration	Not in this PR
SettingsStoreRegistration	Not in this PR
DiagnosticsStoreRegistration	Not in this PR
ServerComponentsRegistration	Not in this PR
crates/bin/src/config.rs (hard-references postgres)	Not modified; backend-agnostic refactor would need to land first

Organizational note

If accepted, this would make ExtendDB the first AWS-managed open-source project whose storage layer is also a covered AWS service. I do not have a strong opinion on whether that is good or bad. I do think it is worth flagging out loud before merge, rather than after.

If the preferred resolution is to remove the implicit invitation in the --backend help text instead, I will withdraw this PR and submit a one-line edit to crates/bin/src/cmd_init.rs:19. Both resolve the same contradiction. The choice of which is funnier is yours.

I do declare that Route 53 is in fact a database. I dare you to prove me wrong.

[zyd14]

We really need a postgres plugin for this too

[chicagobuss]

We really need a postgres plugin for this too

https://github.com/apparentorder/r53db

[duaneking]

I vote for merge, because this has legitimate engineering merit when combined with other services such as ec2 or even AWS Lambda. If an AWS service has Internet access, it has DNS access, which makes DNS a viable bootstrap and minimal state containment mechanism.

You could encode and encrypt an initialization sequence, distribute it through this service via DNS, and use it to provide the minimal state transition data needed for a web service to initialize. At that point, a cluster of Lambda functions could bootstrap configuration from DNS and coordinate finite state machine transitions through serialized DNS updates consumed by the rest of the cluster. Done.

So beyond being funny, I think this is actually technically useful.

[ahachete]

Grab the code from here and merge: https://github.com/ongres/dyna53, it's already done.

(oh, well, different language, but easy to translate)

[neelaruban]

A free terraform state manager :) with amazon level guarantees

[LeeroyHannigan]

The bit predates this repository, and honestly, it outranks it. The blog post, the shirts, the strangers sending YAML: we know the lore, and this PR is a worthy installment of it. Base64 items packed into 255-byte TXT strings until DNS confesses is exactly the level of commitment the argument deserved.

Here is the straight-faced problem. Every storage backend in this repository has to pass the live differential suite: 461 integration tests run against the real DynamoDB service and the backend side by side, and the results have to agree. Conditional writes, transactions, GSI propagation, the works. Route 53's eventual consistency model gives it no path through the transactional conditional-write tests, and I suspect you know that, because the PR description carefully never mentions them.

So: closing, with respect. The day Route 53 passes the differential suite, this PR reopens itself, and I will personally buy the shirt.