Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/on-pr-quality.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,6 @@ jobs:

kafka:
uses: ./.github/workflows/integration-kafka.yaml

distributed-cli:
uses: ./.github/workflows/integration-distributed-cli.yaml
19 changes: 18 additions & 1 deletion distributed_cli/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,7 @@ fn harness_main_rs(entrypoint: &str, mode: HarnessMode) -> String {
let envelope = distributed::DistributedManifestEnvelope::new(manifest);
let statements = envelope
.project
.sql_statements(distributed::TableSqlDialect::{dialect})
.sql_statements(distributed::table::TableSqlDialect::{dialect})
.expect("manifest SQL should render");
if !statements.is_empty() {{
println!("{{}}", statements.join("\n\n"));
Expand Down Expand Up @@ -1031,6 +1031,23 @@ mod tests {
assert!(cargo_toml.contains("name = \"dsvc-manifest-harness-schema-postgres\""));
}

#[test]
fn schema_harness_uses_public_table_module_sql_dialect() {
let main_rs = harness_main_rs(
"orders_service::distributed_manifest",
HarnessMode::SchemaSql(SchemaDialect::Postgres),
);

assert!(
main_rs.contains("distributed::table::TableSqlDialect::Postgres"),
"main.rs: {main_rs}"
);
assert!(
!main_rs.contains("distributed::TableSqlDialect"),
"main.rs: {main_rs}"
);
}

#[test]
fn atlas_spec_uses_secret_ref_by_default() {
let spec = atlas_spec_from_flags(&schema_args(), "CREATE TABLE orders (id text);".into())
Expand Down
23 changes: 17 additions & 6 deletions docs/async-transports.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ bus.subscribe(service.clone(), RunOptions::idempotent()).await?; // fan-out
let namespace = "orders-prod";
// let bus = NatsBus::connect("nats://localhost:4222").namespace(namespace).await?;
// let bus = PostgresBus::new(pool);
// let bus = SqliteBus::new(pool);
// let bus = RabbitBus::connect("amqp://localhost:5672/%2f").namespace(namespace).await?;
// let bus = KafkaBus::connect("localhost:9092").namespace(namespace).await?;
```
Expand All @@ -137,8 +138,9 @@ of one service deployment use the same value; independent event consumers use
different values so each gets its own event copy. Direct `Handlers` or manual
`listen`/`subscribe` calls can set the group with `bus.group(..)` or
`Handlers::named(..)`. `namespace` scopes streams, subjects, topics, queues, or
exchanges on a shared broker. `PostgresBus` does not take `namespace` because the
database/schema behind `pool` already scopes its bus tables.
exchanges on a shared broker. `PostgresBus` and `SqliteBus` do not take
`namespace` because the database/schema/file behind `pool` already scopes their
bus tables.

Topology names are validated before broker use. Keep groups/service names to
portable deployment IDs (`A-Z`, `a-z`, `0-9`, `_`, `-`); namespaces may also use
Expand All @@ -155,6 +157,7 @@ handlers use distinct `group`s when each service needs its own copy:
| `InMemoryBus` | (always) | named queue, popped once | retained log + per-subscriber cursor |
| `NatsBus` | `nats` | shared durable `{group}_cmd` on the stream | durable `{group}_evt` per group |
| `PostgresBus` | `postgres` | `bus_queue`, `FOR UPDATE SKIP LOCKED` | `bus_log` + `bus_offset` per `group` (Kafka-style) |
| `SqliteBus` | `sqlite` | `bus_queue`, atomic `UPDATE ... RETURNING` lease claim | `bus_log` + `bus_offset` per `group` |
| `RabbitBus` | `rabbitmq` | default exchange → durable queue `{ns}.cmd.{name}` | topic exchange → queue `{ns}.evt.{group}` per group |
| `KafkaBus` | `kafka` | shared consumer group `{ns}.{group}.cmd` | consumer group per service `{ns}.{group}.evt` |
| `KnativeBus` | `http` | POST CloudEvent → `{target}-commands` broker-ingress | POST → own `{source}-events` broker; consume via generated Triggers |
Expand All @@ -171,6 +174,12 @@ the uniform drain-to-idle `run_source` model the facade shares; its `bus_log` +
`bus_offset` fan-out gives single-DB transactional effectively-once (the offset
advances with the effects). See `specs/transport-bus-facade`.

`SqliteBus` is the same single-database pattern scaled down to a local SQLite
file: `bus_queue` is claimed with a conditional `UPDATE ... RETURNING` because
SQLite has no `FOR UPDATE SKIP LOCKED`, and `bus_log`/`bus_offset` provide
fan-out. It is intended for local durable transport, tests, demos, and small
single-node deployments, not as a high-throughput broker replacement.

## Testing

The reusable conformance harness (`tests/transport_conformance/`) proves the
Expand All @@ -183,6 +192,7 @@ docker compose up -d # postgres, rabbitmq, kafka, nats (see compose.yaml)

DATABASE_URL=postgres://sourced:sourced@localhost:5432/distributed \
cargo test --test postgres_transport --features postgres
cargo test --test sqlite_transport --features sqlite
NATS_URL=nats://localhost:4222 cargo test --test nats_transport --features nats
AMQP_URL=amqp://guest:guest@localhost:5672/%2f \
cargo test --test rabbitmq_transport --features rabbitmq
Expand All @@ -198,9 +208,10 @@ on push to `main`.
## Status

Implemented and verified: the core contracts, the source runner, the publisher /
outbox dispatcher, the conformance harness, the Postgres / NATS / RabbitMQ /
Kafka adapters, the Knative ingress, and the **bus facade** (`Bus` +
`BusConsumer` with `InMemoryBus` / `NatsBus` / `PostgresBus` / `RabbitBus` /
`KafkaBus` / `KnativeBus`, each with real-broker competing-vs-fan-out tests).
outbox dispatcher, the conformance harness, the Postgres / SQLite / NATS /
RabbitMQ / Kafka adapters, the Knative ingress, and the **bus facade** (`Bus` +
`BusConsumer` with `InMemoryBus` / `NatsBus` / `PostgresBus` / `SqliteBus` /
`RabbitBus` / `KafkaBus` / `KnativeBus`, each with competing-vs-fan-out
integration tests against its broker or local database).
Still open: migrating the in-repo examples to showcase these APIs. See
`tasks/transport-docs-examples-cutover`.
4 changes: 4 additions & 0 deletions src/bus/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ mod router;
mod run_options;
mod runner;
mod source;
#[cfg(feature = "sqlite")]
mod sqlite_bus;
mod stable_id;
mod topology;

Expand Down Expand Up @@ -151,6 +153,8 @@ pub use router::MessageRouter;
pub use run_options::{ConsumerDeliveryMode, InboxHook, NoInbox, RunOptions};
pub use runner::run_source;
pub use source::{AsyncMessageSource, ReceivedMessage};
#[cfg(feature = "sqlite")]
pub use sqlite_bus::{SqliteBus, SqliteLogReceived, SqliteQueueReceived};
pub use stable_id::{validate_stable_message_id, StableMessageIdError, MAX_STABLE_MESSAGE_ID_LEN};
pub use topology::{
resolve_consumer_group, validate_consumer_group, validate_namespace, BusTopologyConfig,
Expand Down
Loading
Loading