diff --git a/.changeset/external-lookup-message-queue-families-retired.md b/.changeset/external-lookup-message-queue-families-retired.md new file mode 100644 index 0000000000..667a1250c7 --- /dev/null +++ b/.changeset/external-lookup-message-queue-families-retired.md @@ -0,0 +1,58 @@ +--- +"@objectstack/spec": major +--- + +refactor(spec)!: retire the external-lookup and message-queue config families — two dead declarations whose only distinctive feature was an inline-credential sink (#8075) + +`ExternalDataSource(Schema)`, `ExternalFieldMapping(Schema|Parsed)`, +`ExternalLookup(Schema|Parsed)` (the whole of `data/external-lookup.zod.ts`) and +`MessageQueueConfig(Schema|Parsed)`, `MessageQueueProvider(Schema)`, +`TopicConfig(Schema|Parsed)`, `ConsumerConfig(Schema|Parsed)`, +`DeadLetterQueue(Schema|Parsed)` (the whole of `system/message-queue.zod.ts`) are +REMOVED under ADR-0049 enforce-or-remove — 8 defs, 22 exported names, reference docs +with them. + +Both families are the #8075 census verdict (fork (b), accepted 2026-08-12): +security-shaped declared surface with inline-credential sinks and **zero consumers**. +`ExternalDataSourceSchema.authentication.config` was a record whose own docblock +example wrote `"clientSecret": "..."` inline; `MessageQueueConfigSchema.sasl` required +an inline `password` whenever present. Neither schema was reachable from any +metadata-type binding, stack collection or `/meta` door, and neither had a single +import outside `packages/spec` repo-wide (corpus-reach control passing in the same +run). The consumed near-namesake `kernel/EventMessageQueueConfig` deliberately carries +NO credential field — so the consumed MQ shape had no credential key and the +credential-bearing MQ shape had no consumer. A dead schema minus one field is still a +dead schema, so the whole declarations go, not just the credential faces (#3950). + +The #7990 Option-B reopen trigger ("a third measured artefact-type surface") is NOT +met — nothing ever persisted these; this is the ADR-0049 leg of the triage-agreed +fork. + +FROM → TO: + +| removed | use instead | +|---|---| +| `ExternalLookup` / `ExternalDataSource` (+ `authentication.config` inline secrets) | `object.external` (`ObjectExternalBindingSchema`, ADR-0015/0062) names a datasource by reference; connection credentials live in datasource config (`data/datasource.zod.ts`, `data/driver/`), never inline in object metadata. `data/external-catalog.zod.ts` is that path's catalog surface and is untouched | +| `ExternalFieldMapping` | **nothing** — it existed only to serve `ExternalLookup.fieldMappings`. The base `shared/FieldMapping` and `integration/ConnectorFieldMapping` are untouched | +| `MessageQueueConfig` (+ `sasl.username`/`sasl.password`) / `MessageQueueProvider` / `TopicConfig` / `ConsumerConfig` / `DeadLetterQueue` | the live MQ surface is `kernel/EventMessageQueueConfig` (`EventBusConfig.messageQueue`) — topic, pattern, format, batching; **no credential field by design**. Broker connection + SASL credentials are runtime deployment configuration, not authorable metadata. `kernel/DeadLetterQueueEntry` (the event bus's per-event DLQ record) is untouched | + +**The fix:** delete the import. Nothing was ever deployed under either family — that +is the finding, not a consolation — so there is no data migration; `tsc` reports +TS2305 at every import of a retired name. Either capability returns via the ENFORCE +route of ADR-0049 through a new ADR: the executor / broker admin service first, the +vocabulary second. + +**Subsumed:** the #5552 `data/ExternalFieldMapping:transform` `retiredKey()` tombstone +and its `RETIRED_KEYS_BY_MAJOR[17]` entry — both land in the unreleased protocol 17, +so composed, the key retirement is absorbed by the def retirement (the +`WidgetManifest.performance` way): there is no longer a mapping shape to author the +key INTO. The `shared/FieldMapping` tombstone and the +`integration/ConnectorFieldMapping` spelling still reject `transform` with the #5552 +prescription; the `field-mapping-transform-removed` D2 conversion still rewrites +`connectors[].fieldMappings[].transform`. + +The retirement kit — route 3: no tombstone, no D2 conversion. +`RETIRED_DEFS_BY_MAJOR[17]` (8 defs) plus the D3 `SemanticMigration` +`external-lookup-message-queue-families-retired` are the declaration. + + diff --git a/content/docs/getting-started/quick-reference.mdx b/content/docs/getting-started/quick-reference.mdx index 0989ba0fea..1c4d5475fe 100644 --- a/content/docs/getting-started/quick-reference.mdx +++ b/content/docs/getting-started/quick-reference.mdx @@ -23,7 +23,7 @@ Categories that have no section here at all are named under [Categories Without a Section](#categories-without-a-section) — that curation is stated, not left implicit. -## Data Protocol (17 of 30 schemas) +## Data Protocol (16 of 29 schemas) Core business logic and data modeling schemas. @@ -43,7 +43,6 @@ Core business logic and data modeling schemas. | **[SQL Driver](/docs/references/data/driver-sql)** | `driver-sql.zod.ts` | SQLDriverConfig, SQLDialect | SQL-specific driver | | **[NoSQL Driver](/docs/references/data/driver-nosql)** | `driver-nosql.zod.ts` | NoSQLDriverConfig | NoSQL-specific driver | | **[Document](/docs/references/data/document)** | `document.zod.ts` | Document | Document-oriented data | -| **[External Lookup](/docs/references/data/external-lookup)** | `external-lookup.zod.ts` | ExternalLookup | External data lookups | | **[Postgres Driver](/docs/references/data/driver-postgres)** | `driver/postgres.zod.ts` | PostgresConfig | PostgreSQL configuration | | **[Mongo Driver](/docs/references/data/driver-mongo)** | `driver/mongo.zod.ts` | MongoConfig | MongoDB configuration | @@ -89,7 +88,7 @@ Plugin architecture, manifests, and kernel runtime. | **[Metadata Loader](/docs/references/kernel/metadata-loader)** | `metadata-loader.zod.ts` | MetadataLoaderContract | Metadata loading | | **[Package Registry](/docs/references/kernel/package-registry)** | `package-registry.zod.ts` | InstalledPackage, InstallPackageRequest | Package resolution | -## System Protocol (18 of 37 schemas) +## System Protocol (17 of 36 schemas) Runtime environment, logging, jobs, caching, and observability. @@ -103,7 +102,6 @@ Runtime environment, logging, jobs, caching, and observability. | **[HTTP Server](/docs/references/system/http-server)** | `http-server.zod.ts` | RouteHandlerMetadata, MiddlewareConfig | Route + middleware metadata | | **[Job](/docs/references/system/job)** | `job.zod.ts` | Job, JobSchedule | Background job queue | | **[Logging](/docs/references/system/logging)** | `logging.zod.ts` | LoggingConfig | Structured logging | -| **[Message Queue](/docs/references/system/message-queue)** | `message-queue.zod.ts` | MessageQueueConfig, TopicConfig | Message queuing | | **[Metadata Persistence](/docs/references/system/metadata-persistence)** | `metadata-persistence.zod.ts` | MetadataHistoryRecord, MetadataDiffResult | Metadata storage | | **[Metrics](/docs/references/system/metrics)** | `metrics.zod.ts` | Metrics | Application metrics | | **[Migration](/docs/references/system/migration)** | `migration.zod.ts` | Migration | Schema migration | diff --git a/content/docs/references/data/external-lookup.mdx b/content/docs/references/data/external-lookup.mdx deleted file mode 100644 index bb77e2ead6..0000000000 --- a/content/docs/references/data/external-lookup.mdx +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: External Lookup -description: External Lookup protocol schemas ---- - -{/* ⚠️ AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */} - -External Data Source Schema - -Configuration for connecting to external data systems. -Similar to Salesforce External Objects for real-time data integration. - -@example -```json -{ - "id": "salesforce-accounts", - "name": "Salesforce Account Data", - "type": "rest-api", - "endpoint": "https://api.salesforce.com/services/data/v58.0", - "authentication": { - "type": "oauth2", - "config": { - "clientId": "...", - "clientSecret": "...", - "tokenUrl": "https://login.salesforce.com/services/oauth2/token" - } - } -} -``` - - -**Source:** `packages/spec/src/data/external-lookup.zod.ts` - - -## TypeScript Usage - -```typescript -import { ExternalDataSourceSchema, ExternalFieldMappingSchema, ExternalLookupSchema } from '@objectstack/spec/data'; -import type { ExternalDataSource, ExternalFieldMapping, ExternalLookup } from '@objectstack/spec/data'; - -// Validate data -const result = ExternalDataSourceSchema.parse(data); -``` - ---- - -## ExternalDataSource - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **id** | `string` | ✅ | Data source ID | -| **name** | `string` | ✅ | Data source name | -| **type** | `Enum<'odata' \| 'rest-api' \| 'graphql' \| 'custom'>` | ✅ | Protocol type | -| **endpoint** | `string` | ✅ | API endpoint URL | -| **authentication** | `{ type: Enum<'oauth2' \| 'api-key' \| 'basic' \| 'none'>; config: Record }` | ✅ | Authentication | - - ---- - -## ExternalFieldMapping - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **source** | `string` | ✅ | Source field name | -| **target** | `string` | ✅ | Target field name | -| **transform** | `never` | optional | [REMOVED] `FieldMapping.transform` — authored as `connector.fieldMappings[].transform` and `externalLookup.fieldMappings[].transform` — was removed in @objectstack/spec 17.0.0 (#5552, ADR-0049), and the whole `FieldMappingTransform` union went with it (`constant` / `cast` / `lookup` / `javascript` / `map`) — no runtime ever executed any of the five, and the `javascript` member advertised `dialect: "js"`, a dialect retired in #3278. Delete the key. The transform pipeline that IS enforced is the import mapping's: `mapping.fieldMapping[].transform` (a string enum — `none`/`constant`/`map`/`split`/`join`/`lookup` — with its settings in `params`), applied by the REST import path, which rejects `javascript` with a 400 rather than pretending to run it. Run `os migrate meta --from 16` to rewrite existing sources automatically. | -| **defaultValue** | `any` | optional | Default if source is null/undefined | -| **type** | `string` | optional | Field type | -| **readonly** | `boolean` | ✅ | Read-only field | - - ---- - -## ExternalLookup - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **fieldName** | `string` | ✅ | Field name | -| **dataSource** | `{ id: string; name: string; type: Enum<'odata' \| 'rest-api' \| 'graphql' \| 'custom'>; endpoint: string; … }` | ✅ | External data source | -| **query** | `{ endpoint: string; method: Enum<'GET' \| 'POST'>; parameters?: Record }` | ✅ | Query configuration | -| **fieldMappings** | `{ source: string; target: string; defaultValue?: any; type?: string; … }[]` | ✅ | Field mappings | -| **caching** | `{ enabled: boolean; ttl: number; strategy: Enum<'lru' \| 'lfu' \| 'ttl'> }` | optional | Caching configuration | -| **fallback** | `{ enabled: boolean; defaultValue?: any; showError: boolean }` | optional | Fallback configuration | -| **rateLimit** | `{ requestsPerSecond: number; burstSize?: number }` | optional | Rate limiting | -| **retry** | `{ maxRetries: number; initialDelayMs: number; maxDelayMs: number; backoffMultiplier: number; … }` | optional | Retry configuration with exponential backoff | -| **transform** | `{ request?: object; response?: object }` | optional | Request/response transformation pipeline | -| **pagination** | `{ type: Enum<'offset' \| 'cursor' \| 'page'>; pageSize: number; maxPages?: number }` | optional | Pagination configuration for external data | - - ---- - diff --git a/content/docs/references/data/index.mdx b/content/docs/references/data/index.mdx index f98ecea006..ad98cb1db4 100644 --- a/content/docs/references/data/index.mdx +++ b/content/docs/references/data/index.mdx @@ -23,7 +23,6 @@ This section contains all protocol schemas for the data layer of ObjectStack. - diff --git a/content/docs/references/data/meta.json b/content/docs/references/data/meta.json index ab512bcd57..ec5a53503d 100644 --- a/content/docs/references/data/meta.json +++ b/content/docs/references/data/meta.json @@ -20,7 +20,6 @@ "driver-nosql", "driver-sql", "external-catalog", - "external-lookup", "---Documents & Seed---", "document", "feed", diff --git a/content/docs/references/index.mdx b/content/docs/references/index.mdx index 252fee6447..e554ac9100 100644 --- a/content/docs/references/index.mdx +++ b/content/docs/references/index.mdx @@ -1,6 +1,6 @@ --- title: Protocol Reference -description: Every schema published by @objectstack/spec — 1578 schemas across 14 protocol modules +description: Every schema published by @objectstack/spec — 1570 schemas across 14 protocol modules --- {/* ⚠️ AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */} @@ -23,7 +23,7 @@ counts are sums of the rows they head. Regenerate with | [API Protocol](/docs/references/api) | 28 | 412 | REST/GraphQL contracts, endpoints, routing, realtime, batch, discovery. | | [Automation Protocol](/docs/references/automation) | 13 | 68 | Flows and their nodes, approvals, ETL pipelines, webhooks, state machines, execution records. | | [Cloud Protocol](/docs/references/cloud) | 11 | 94 | Environments, packages and versions, marketplace, developer portal, tenancy. | -| [Data Protocol](/docs/references/data) | 30 | 166 | Objects, fields, queries, filters, datasources and drivers — the ObjectQL layer. | +| [Data Protocol](/docs/references/data) | 29 | 163 | Objects, fields, queries, filters, datasources and drivers — the ObjectQL layer. | | [Identity Protocol](/docs/references/identity) | 5 | 28 | Users and accounts, organizations, positions, API keys, SCIM provisioning. | | [Integration Protocol](/docs/references/integration) | 1 | 27 | The single connector protocol (ADR-0097) — catalog descriptors and provider-bound instances. | | [Kernel Protocol](/docs/references/kernel) | 31 | 176 | Plugin lifecycle and manifests, capabilities and security, metadata loading, service registry. | @@ -31,9 +31,9 @@ counts are sums of the rows they head. Regenerate with | [Security Protocol](/docs/references/security) | 5 | 27 | Permission sets, row-level security, sharing rules, tenancy posture. | | [Shared Protocol](/docs/references/shared) | 8 | 31 | Primitives used across every protocol — identifiers, HTTP, expressions, error maps, enums. | | [Studio Protocol](/docs/references/studio) | 3 | 35 | Studio designer metadata — the authoring surfaces for the protocols above. | -| [System Protocol](/docs/references/system) | 37 | 292 | The runtime environment — logging, jobs, cache, metrics, notifications, i18n and compliance. | +| [System Protocol](/docs/references/system) | 36 | 287 | The runtime environment — logging, jobs, cache, metrics, notifications, i18n and compliance. | | [UI Protocol](/docs/references/ui) | 16 | 148 | Apps, pages, views, dashboards, reports, actions and themes — the ObjectUI layer. | -| **Total** | **200** | **1578** | 14 protocol modules | +| **Total** | **198** | **1570** | 14 protocol modules | --- @@ -146,7 +146,7 @@ Environments, packages and versions, marketplace, developer portal, tenancy. ## Data Protocol -**Source:** `packages/spec/src/data/` · **Import:** `@objectstack/spec/data` · **30 pages, 166 schemas** +**Source:** `packages/spec/src/data/` · **Import:** `@objectstack/spec/data` · **29 pages, 163 schemas** Objects, fields, queries, filters, datasources and drivers — the ObjectQL layer. @@ -169,7 +169,6 @@ Objects, fields, queries, filters, datasources and drivers — the ObjectQL laye | [`driver/sqlite.zod.ts`](/docs/references/data/driver-sqlite) | `SqliteConfig`, `SqliteWasmConfig`, `SqliteWasmPersistMode` | | [`driver/turso.zod.ts`](/docs/references/data/driver-turso) | `TursoConfig`, `TursoTransportMode` | | [`external-catalog.zod.ts`](/docs/references/data/external-catalog) | `ExternalCatalog`, `ExternalColumn`, `ExternalTable` | -| [`external-lookup.zod.ts`](/docs/references/data/external-lookup) | `ExternalDataSource`, `ExternalFieldMapping`, `ExternalLookup` | | [`feed.zod.ts`](/docs/references/data/feed) | `FeedFilterMode`, `FeedItemType` | | [`field.zod.ts`](/docs/references/data/field) | `CurrencyConfig`, `CurrencyValue`, `Field`, `FieldType`, `LocationCoordinates`, `SelectOption`, `UniqueScope` | | [`field-value.zod.ts`](/docs/references/data/field-value) | `Address`, `AddressValue`, `CalendarDateValue`, `ClockTimeValue`, `FileLikeValue`, `FileReferenceIdValue`, `FileValue`, `InstantValue`, `LocationValue`, `ReferenceIdValue` | @@ -318,7 +317,7 @@ Studio designer metadata — the authoring surfaces for the protocols above. ## System Protocol -**Source:** `packages/spec/src/system/` · **Import:** `@objectstack/spec/system` · **37 pages, 292 schemas** +**Source:** `packages/spec/src/system/` · **Import:** `@objectstack/spec/system` · **36 pages, 287 schemas** The runtime environment — logging, jobs, cache, metrics, notifications, i18n and compliance. @@ -343,7 +342,6 @@ The runtime environment — logging, jobs, cache, metrics, notifications, i18n a | [`job.zod.ts`](/docs/references/system/job) | `CronSchedule`, `IntervalSchedule`, `Job`, `JobExecution`, `JobExecutionStatus`, `OnceSchedule`, `RetryPolicy`, `Schedule` | | [`license.zod.ts`](/docs/references/system/license) | `Feature`, `License`, `LicenseMetricType`, `Plan` | | [`logging.zod.ts`](/docs/references/system/logging) | `ConsoleDestinationConfig`, `ExtendedLogLevel`, `ExternalServiceDestinationConfig`, `FileDestinationConfig`, `HttpDestinationConfig`, `LogDestination`, `LogDestinationType`, `LogEnrichmentConfig`, `LogEntry`, `LogFormat`, `LogLevel`, `LoggerConfig`, `LoggingConfig`, `StructuredLogEntry` | -| [`message-queue.zod.ts`](/docs/references/system/message-queue) | `ConsumerConfig`, `DeadLetterQueue`, `MessageQueueConfig`, `MessageQueueProvider`, `TopicConfig` | | [`metadata-persistence.zod.ts`](/docs/references/system/metadata-persistence) | `MetadataCollectionInfo`, `MetadataDiffResult`, `MetadataFallbackStrategy`, `MetadataFormat`, `MetadataHistoryQueryOptions`, `MetadataHistoryQueryResult`, `MetadataHistoryRecord`, `MetadataHistoryRetentionPolicy`, `MetadataLoadOptions`, `MetadataLoadResult`, `MetadataLoaderContract`, `MetadataManagerConfig`, `MetadataRecord`, `MetadataSaveOptions`, `MetadataSaveResult`, `MetadataScope`, `MetadataSource`, `MetadataState`, `MetadataStats`, `MetadataWatchEvent`, `PackagePublishResult` | | [`metrics.zod.ts`](/docs/references/system/metrics) | `HistogramBucketConfig`, `MetricAggregationConfig`, `MetricAggregationType`, `MetricDataPoint`, `MetricDefinition`, `MetricExportConfig`, `MetricLabels`, `MetricType`, `MetricUnit`, `MetricsConfig`, `ServiceLevelIndicator`, `ServiceLevelObjective`, `TimeSeries`, `TimeSeriesDataPoint` | | [`migration.zod.ts`](/docs/references/system/migration) | `AddFieldOperation`, `ChangeSet`, `CreateObjectOperation`, `DataMigrationFlag`, `DeleteObjectOperation`, `ExecuteSqlOperation`, `MigrationDependency`, `MigrationJournalEvent`, `MigrationOperation`, `ModifyFieldOperation`, `RemoveFieldOperation`, `RenameObjectOperation` | diff --git a/content/docs/references/shared/mapping.mdx b/content/docs/references/shared/mapping.mdx index 6b0e718ee4..abb981e314 100644 --- a/content/docs/references/shared/mapping.mdx +++ b/content/docs/references/shared/mapping.mdx @@ -7,14 +7,16 @@ description: Mapping protocol schemas Base Field Mapping Protocol -Shared by: Connector, External Lookup +Shared by: Connector This module provides the canonical field mapping schema used across ObjectStack for data synchronization. **Use Cases:** - Integration connectors ([integration/connector.zod.ts](/docs/references/integration/connector)) -- External lookups ([data/external-lookup.zod.ts](/docs/references/data/external-lookup)) +- (`data/external-lookup.zod.ts`'s `ExternalFieldMappingSchema` extended + this base too, until the whole external-lookup family was retired in + #8075 — ADR-0049, zero consumers.) @example Basic field mapping ```typescript diff --git a/content/docs/references/system/index.mdx b/content/docs/references/system/index.mdx index 1b7d6c1222..f2b84b82d3 100644 --- a/content/docs/references/system/index.mdx +++ b/content/docs/references/system/index.mdx @@ -25,7 +25,6 @@ This section contains all protocol schemas for the system layer of ObjectStack. - diff --git a/content/docs/references/system/message-queue.mdx b/content/docs/references/system/message-queue.mdx deleted file mode 100644 index c43c627abb..0000000000 --- a/content/docs/references/system/message-queue.mdx +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Message Queue -description: Message Queue protocol schemas ---- - -{/* ⚠️ AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */} - -Message queue protocol for async communication -Supports Kafka, RabbitMQ, AWS SQS, Redis Pub/Sub - - -**Source:** `packages/spec/src/system/message-queue.zod.ts` - - -## TypeScript Usage - -```typescript -import { ConsumerConfigSchema, DeadLetterQueueSchema, MessageQueueConfigSchema, MessageQueueProviderSchema, TopicConfigSchema } from '@objectstack/spec/system'; -import type { ConsumerConfig, DeadLetterQueue, MessageQueueConfig, MessageQueueProvider, TopicConfig } from '@objectstack/spec/system'; - -// Validate data -const result = ConsumerConfigSchema.parse(data); -``` - ---- - -## ConsumerConfig - -Consumer group configuration for topic consumption - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **groupId** | `string` | ✅ | Consumer group identifier | -| **autoOffsetReset** | `Enum<'earliest' \| 'latest'>` | ✅ | Where to start reading when no offset exists | -| **enableAutoCommit** | `boolean` | ✅ | Automatically commit consumed offsets | -| **maxPollRecords** | `number` | ✅ | Maximum records returned per poll | - - ---- - -## DeadLetterQueue - -Dead letter queue configuration for unprocessable messages - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **enabled** | `boolean` | ✅ | Enable dead letter queue for failed messages | -| **maxRetries** | `number` | ✅ | Maximum delivery attempts before sending to DLQ | -| **queueName** | `string` | ✅ | Name of the dead letter queue | - - ---- - -## MessageQueueConfig - -Top-level message queue configuration - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **provider** | `Enum<'kafka' \| 'rabbitmq' \| 'aws-sqs' \| 'redis-pubsub' \| 'google-pubsub' \| 'azure-service-bus'>` | ✅ | Message queue backend provider | -| **topics** | `{ name: string; partitions: number; replicationFactor: number; retentionMs?: number; … }[]` | ✅ | List of topic configurations | -| **consumers** | `{ groupId: string; autoOffsetReset: Enum<'earliest' \| 'latest'>; enableAutoCommit: boolean; maxPollRecords: number }[]` | optional | Consumer group configurations | -| **deadLetterQueue** | `{ enabled: boolean; maxRetries: number; queueName: string }` | optional | Dead letter queue for failed messages | -| **ssl** | `boolean` | ✅ | Enable SSL/TLS for broker connections | -| **sasl** | `{ mechanism: Enum<'plain' \| 'scram-sha-256' \| 'scram-sha-512'>; username: string; password: string }` | optional | SASL authentication configuration | - - ---- - -## MessageQueueProvider - -Supported message queue backend provider - -### Allowed Values - -* `kafka` -* `rabbitmq` -* `aws-sqs` -* `redis-pubsub` -* `google-pubsub` -* `azure-service-bus` - - ---- - -## TopicConfig - -Configuration for a message queue topic - -### Properties - -| Property | Type | Required | Description | -| :--- | :--- | :--- | :--- | -| **name** | `string` | ✅ | Topic name identifier | -| **partitions** | `number` | ✅ | Number of partitions for parallel consumption | -| **replicationFactor** | `number` | ✅ | Number of replicas for fault tolerance | -| **retentionMs** | `number` | optional | Message retention period in milliseconds | -| **compressionType** | `Enum<'none' \| 'gzip' \| 'snappy' \| 'lz4'>` | ✅ | Message compression algorithm | - - ---- - diff --git a/content/docs/references/system/meta.json b/content/docs/references/system/meta.json index fbff8d1e47..d2bd8a631d 100644 --- a/content/docs/references/system/meta.json +++ b/content/docs/references/system/meta.json @@ -19,7 +19,6 @@ "core-services", "http-server", "job", - "message-queue", "metadata-persistence", "notification", "object-storage", diff --git a/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md b/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md index 67f97f89b9..96230e37e4 100644 --- a/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md +++ b/docs/audits/2026-07-unknown-key-strictness-ledger.counts.md @@ -21,16 +21,16 @@ regenerate. | Measure | Value | |---|---| | Triaged directories | 5 | -| Object sites in them | 438 | -| Still-open (strip) sites | 144 | -| Files carrying at least one | 25 | +| Object sites in them | 426 | +| Still-open (strip) sites | 132 | +| Files carrying at least one | 24 | Remaining strip sites by class: | Bucket | Sites | |---|---| | authorable — the ruling's forced scope | 4 | -| unresolved — needs a per-schema verdict | 34 | +| unresolved — needs a per-schema verdict | 22 | | wire / open — out of forced scope | 104 | | no door — no carrier, ADR-0049 territory | 1 | | no gate — carrier live, no parse | 0 | @@ -45,11 +45,11 @@ The `strict` column is the one the campaign schedules against; it counts both th | Dir | Sites | strict | passthrough | catchall | strip | |---|---|---|---|---|---| | `ui/` | 161 | 150 | 5 | 0 | 6 | -| `data/` | 165 | 62 | 1 | 0 | 102 | +| `data/` | 153 | 62 | 1 | 0 | 90 | | `automation/` | 65 | 42 | 0 | 0 | 23 | | `security/` | 20 | 7 | 0 | 0 | 13 | | `studio/` | 27 | 27 | 0 | 0 | 0 | -| **total** | **438** | **288** | **6** | **0** | **144** | +| **total** | **426** | **288** | **6** | **0** | **132** | ## File-level triage — site counts @@ -97,7 +97,6 @@ classify and is not listed (it becomes reportable the day it grows its first sit | `driver/sqlite.zod.ts` | 2 | | `driver/turso.zod.ts` | 2 | | `external-catalog.zod.ts` | 4 | -| `external-lookup.zod.ts` | 12 | | `field-value.zod.ts` | 3 | | `field.zod.ts` | 10 | | `filter.zod.ts` | 11 | @@ -109,7 +108,7 @@ classify and is not listed (it becomes reportable the day it grows its first sit | `seed-loader.zod.ts` | 12 | | `seed.zod.ts` | 1 | | `validation.zod.ts` | 6 | -| **total** | **165** | +| **total** | **153** | ### `automation/` — sites @@ -178,7 +177,7 @@ over it is here. ### `data/` — open -**102 strip of 165**, in 15 file(s). +**90 strip of 153**, in 14 file(s). | File | Strip | Sites | |---|---|---| @@ -189,7 +188,6 @@ over it is here. | `driver-sql.zod.ts` | 2 | 2 | | `driver.zod.ts` | 9 | 9 | | `external-catalog.zod.ts` | 4 | 4 | -| `external-lookup.zod.ts` | 12 | 12 | | `field-value.zod.ts` | 2 | 3 | | `field.zod.ts` | 2 | 10 | | `filter.zod.ts` | 10 | 11 | @@ -197,12 +195,12 @@ over it is here. | `object.zod.ts` | 1 | 20 | | `query.zod.ts` | 4 | 5 | | `seed-loader.zod.ts` | 12 | 12 | -| **total** | **102** | **165** | +| **total** | **90** | **153** | | Bucket | Sites | |---|---| | authorable — the ruling's forced scope | 3 | -| unresolved — needs a per-schema verdict | 34 | +| unresolved — needs a per-schema verdict | 22 | | wire / open — out of forced scope | 65 | | no door — no carrier, ADR-0049 territory | 0 | | no gate — carrier live, no parse | 0 | @@ -269,4 +267,4 @@ directory rather than per file. | `kernel/` | 296 | | `qa/` | 6 | | `shared/` | 20 | -| `system/` | 365 | +| `system/` | 360 | diff --git a/docs/audits/2026-07-unknown-key-strictness-ledger.md b/docs/audits/2026-07-unknown-key-strictness-ledger.md index c2563b4af7..b8960e557b 100644 --- a/docs/audits/2026-07-unknown-key-strictness-ledger.md +++ b/docs/audits/2026-07-unknown-key-strictness-ledger.md @@ -689,7 +689,6 @@ column does not move and the `strip` column falls by the count of what left. |---|---|---| | `object.zod.ts` | authorable | top-level already guarded (#1535); inner blocks strict as of 批 20 — except `IndexSchema`, held on a measured console drift (see the `data/` remaining-strip row) | | `data-engine.zod.ts` | wire (p) | engine contract shapes (was 14 — `DataEngineBatchRequestSchema` retired with `IDataEngine.batch?`, #4618) | -| `external-lookup.zod.ts` | mixed (p) | authored config + wire results | | `seed-loader.zod.ts` | mixed (p) | seed file shapes are authored; loader state is runtime | | `field.zod.ts` | authorable | partially strict | | `filter.zod.ts` | open | query dialect — user data flows through the predicate values; validated semantically elsewhere | @@ -1174,7 +1173,6 @@ triage row record which one was taken. |---|---|---| | `object.zod.ts` | authorable | **13 of 14 closed at #4001 批 20; the 14th is HELD, and that hold is the batch's finding.** The registered type's top level was already closed (#1535/#4519/#4522) — these were the inner blocks under it, and the asymmetry is why the file mattered: an author who has SEEN the root reject a typo reads a clean parse of `lifecycle: { maxAge: '30d' }` as acceptance. Closed: `ObjectAccessConfig`, `Lifecycle` + all four sub-blocks (`retention`/`ttl`/`storage`/`archive`), `ObjectFieldGroup`, `ObjectExternalBinding`, `userActions`, `systemFields`, `activityMilestones`, `publicSharing`, `ObjectExtension`. Reachability was measured, not assumed: a BFS from all 24 metadata-type roots plus `ObjectStackSchema` (4810 nodes, 25 roots) resolves every one `direct` **by identity** — none rests on the `derived-clone` bridge #5056 found can mark a dead shape reachable — with `ObjectSchema`/`PageSchema` as positive controls and 批 14's `EmbedConfigSchema` UNREACHABLE (overlap 0.00) **in the same run**. The parse door was probed separately and at each path: `ObjectSchema.safeParse` REJECTS at the top level today and SILENTLY STRIPPED at all thirteen nested paths, so each closure converts a measured silent strip, not a hypothesis. Doors: `saveMetaItem`'s 422 (`metadata-protocol/protocol.ts` — `getMetadataTypeSchema('object')`), `ObjectSchema.create()`, `defineObjectExtension()`, and `registry.validate()` (diagnostic-only by #3903 design). ⚠️ **`IndexSchema` is deliberately NOT closed — the #5114 class, caught before it shipped rather than after.** objectui's console ships its own hand-copied JSON-Schema for this shape (`metadata-admin/EmbeddedItemEditor.tsx` → `FALLBACK_SCHEMAS.index`), because `index` is an embedded-only sub-type the framework publishes no schema for; that copy has drifted and offers **`where`** for the partial-index predicate where this schema declares **`partial`** (and `brin` in an enum that has no `brin`). The editor splices its form output into `object.indexes[]` and PUTs the WHOLE object, and `saveMetaItem` keeps the body verbatim while validating it — so closing this one shape would 422 a control the console itself renders. Unlike #5073's `allowAddTab` the capability is not merely un-gated but already DEAD in both directions: `driver-sql`'s `syncDeclaredIndexes` consumes `name`/`fields`/`unique` only, so neither `where` NOR `partial` reaches any DDL. That is why the hold is not just "fix the producer first": pointing an author at `partial` today would be a guidance entry claiming more than the platform delivers (finding 18), so the close is gated on BOTH the objectui rename (**#5247**) and an ADR-0049 answer for `type`/`partial` (**#5248**). Recorded in three places (the `IndexSchema` JSDoc + `object-strictness-batch20.test.ts` §4 + this row). One caveat shipped knowingly: `systemFields` is a `false | {…}` union, so its rejection is an `invalid_union` whose own message is the bare *"Invalid input"* — the #5014 flattening. 批 18's `discriminatedUnion` fix is unavailable (one arm is a literal, so there is no discriminant), so the behaviour is pinned honestly rather than papered over; every other site in the file is a plain object and surfaces its prescription directly. Curation is anchored to named siblings and each claim is asserted: the dominant failure here is FLATTENING (`maxAge`/`expireAfter`/`shards` written one level too high, where §3.5's own refine then rejects the object for the WRONG key), so `lifecycle` carries wrong-layer pointers DOWN into its four sub-blocks; `userActions` points at `ui/view.zod.ts`'s identically-named block, whose vocabulary is completely disjoint; `systemFields.owner` points at `ownership`, a key the block's own field doc names but the shape never declared; `external.allowWrites` names the ADR-0015 double opt-in and mirrors `datasource.zod.ts`'s own `writable → allowWrites` alias in the opposite direction; `fieldGroups[].fields` states the direction of the membership edge (declared on the FIELD), and its three DEPRECATED collapse aliases stay ACCEPTED — closing a shape must not turn a documented deprecation into a rejection. Highest author volume in the repo | | `data-engine.zod.ts` | wire | **out of scope** — engine request/response contracts | -| `external-lookup.zod.ts` | mixed (p) | `ExternalDataSource` + `.authentication` and the `ExternalLookup` tree are authored config; needs the per-schema read the ledger never did | | `seed-loader.zod.ts` | mixed (p) | Split is real: `SeedLoaderConfig` / `SeedIdentity` (+`.user`/`.org`) / `ReferenceResolution` are authored; `SeedLoadResult` / `SeedLoaderResult` (+`.summary`) / `ReferenceResolutionError` / `ObjectDependencyNode` / `ObjectDependencyGraph` / `SeedLoaderRequest` are loader runtime | | `filter.zod.ts` | open | **out of scope** — query dialect; user data flows through, validated semantically elsewhere | | `driver-nosql.zod.ts` | wire | **out of scope** | @@ -1191,9 +1189,11 @@ triage row record which one was taken. **Authorable strip in `data/`:** [the counts file](./2026-07-unknown-key-strictness-ledger.counts.md#data--open) splits this directory three ways, and the middle bucket is the one to read: `object` and `field` are -**firm** authorable; `external-lookup`, `seed-loader`, `analytics` and `field-value` are +**firm** authorable; `seed-loader`, `analytics` and `field-value` are **unresolved** — they still carry `mixed (p)`, so the ledger is saying "nobody has done the -per-schema read" rather than "these are ready". The rest is wire/open and out of the +per-schema read" rather than "these are ready". (`external-lookup` carried `mixed (p)` too +until #8075 retired the whole file under ADR-0049 — its per-schema read arrived as a +zero-consumer verdict, and the strictness question died with the shapes.) The rest is wire/open and out of the ruling's forced scope; that count fell by one when #4721 closed `query.zod.ts`'s `SortNodeSchema`, the one row in this directory where the per-schema read moved a site OUT of `open` rather than confirming it, and by one more when **#4001 batch B** closed diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index 92c27a2cbb..a44a0b5e7f 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -409,6 +409,9 @@ ONE AUTHOR-REACHABLE SURFACE reaches this indirectly and is why it is not purely - **`export-field-meta-constraints-retired`** — `@objectstack/rest: ExportFieldMeta.required / .system / .readonly / .hasDefault / .min / .max / .minLength / .maxLength (the map built by `buildFieldMetaMap`, reached as `PreparedImport.metaMap` from `prepareImportRequest`)` → the object schema you already hold — read `fields[name].required` / `.system` / `.readonly` / `.defaultValue` / `.min` / `.max` / `.minLength` / `.maxLength` off the same `ObjectSchema` you passed to `buildFieldMetaMap`, which is where the ENGINE reads them and therefore the only copy that cannot drift - Why not automatic: ADR-0049 enforce-or-remove. These eight were never a source of truth: `buildFieldMetaMap(schema)` DERIVED each one from the very `schema` its caller passed in, so the map carried a second copy of facts the caller already held. They existed for exactly one consumer — the import dry run's hand-copied pre-check mirror (`firstMissingRequiredField` / `firstConstraintViolation`, framework#3956) — and #4633 ruling D retired that mirror (PR #6532): the dry run now asks `DataProtocol.validateData` for the engine's verdict, which reads the object's own schema. That left all eight computed on every import and read by NOTHING, which is the declared-and-unread shape ADR-0049 exists for; a constraint vocabulary standing next to the presentation one with no enforcer behind it is precisely the thing an AI-authored consumer mistakes for a contract. Verified zero-reader before removal, per key and by type, across this repo (`packages/rest` itself, and all five in-repo dependents of `@objectstack/rest`: runtime, cli, verify, plugin-auth, plugin-dev) and the `objectui` sibling; plugin-auth's identity import forwards `prepared.metaMap` into `runImport` but reads only the presentation keys through `coerceRow`. Why this needs a ledger entry despite that sweep: it is the `findStream` (#4484) / `IStorageService.list` (#5540) / `actor-user-roles-to-positions` (#6011) disposition — a published TS surface with NO spec schema, so there is no `retiredKey()` tombstone and no parse rejection that could carry a prescription, and the ledger is the only channel that reaches an upgrader. It is if anything blinder than those three: the keys shipped in a FINAL release (`@objectstack/rest` 14.5.0) and have been published in every release since, and because they were OPTIONAL keys on an interface that itself survives, a JavaScript consumer reading `meta.required` after the upgrade gets `undefined` with no error at all — tsc reports at the read site only for a typed consumer. Why D3 semantic and not a D2 conversion: there is nothing to convert. No authored or stored metadata changes shape — `required` / `min` / `maxLength` and the rest remain fully authorable on a field definition and fully enforced by the engine, which is where they always lived. The only place these eight are ever spelled is inside a consumer's own TypeScript, so no `objectstack migrate meta` transform can reach them. ADR-0049 / ADR-0087, #6536 (the sweep PR #6532 deliberately deferred). - Done when: No code of yours reads any of the eight off a `buildFieldMetaMap` / `prepareImportRequest` result. Grep your sources for `.required` / `.hasDefault` / `.minLength` / `.maxLength` / `.min` / `.max` / `.system` / `.readonly` on an `ExportFieldMeta`-typed value; each hit moves to the object schema you already passed in. ⚠️ Prove it against a RUN, not against tsc: these were optional keys, so an untyped or `any`-typed read compiles clean and silently becomes `undefined` — assert that the constraint your code acts on is still observed on a real import, not merely that the build is green. Note `hasDefault` has no one-to-one replacement key: it was the derived predicate `defaultValue != null`, mirroring the engine's `applyFieldDefaults` gate, so read `fields[name].defaultValue` and apply that same `!= null` test yourself. +- **`external-lookup-message-queue-families-retired`** — `data.externalLookup / data.externalDataSource / data.externalFieldMapping (the whole of data/external-lookup.zod.ts — 3 defs, 8 exported names) and system.messageQueue (the whole of system/message-queue.zod.ts — MessageQueueConfig, MessageQueueProvider, TopicConfig, ConsumerConfig, DeadLetterQueue — 5 defs, 14 exported names)` → (removed — there is no replacement key, because there was never a key: neither family was reachable from any metadata-type binding, stack collection or /meta door, so no document could carry either. For external data: `object.external` (`ObjectExternalBindingSchema`, ADR-0015/0062) names a datasource by reference and connection credentials live in the datasource config — never inline in object metadata; `data/external-catalog.zod.ts` is that federated path's catalog surface and is untouched. For message queues: the LIVE surface is `kernel/events/integrations.zod.ts`'s `EventMessageQueueConfig` (`EventBusConfig.messageQueue`), which deliberately carries NO credential field — broker connection and SASL credentials are runtime deployment configuration, not authorable metadata. Either capability returns via the ENFORCE route of ADR-0049 through a new ADR — the executor / broker admin service first, the vocabulary second) + - Why not automatic: Both families are the #8075 census verdict (fork (b), accepted 2026-08-12): security-shaped declared surface with inline-credential sinks and ZERO consumers. `ExternalDataSourceSchema.authentication.config` is a record of unknown whose own docblock example wrote `"clientSecret": "..."` inline, and `MessageQueueConfigSchema.sasl.password` was a required inline broker credential — the #7990 class (cleartext-at-rest credential sinks), except that unlike #7990's two measured surfaces nothing ever persisted these: no metadata-type binding (kernel/metadata-type-schemas.ts imports neither module), no stack collection, no object/field embedding (`object.external` binds `ObjectExternalBindingSchema` — remoteName/remoteSchema/writable/columnMap, no authentication), and zero imports outside packages/spec repo-wide, with the corpus-reach control (`DatasourceSchema` under identical exclusions) returning hits in the same run. The consumed MQ near-namesake `kernel/EventMessageQueueConfig` deliberately has no credential key, so the consumed shape had no credential and the credential-bearing shape had no consumer. A dead schema minus one field is still a dead schema, so the whole declarations go, not just the credential faces (#3950: an exported schema with no consumer reads as a capability to whoever finds it — here it read as an invitation to author secrets in cleartext). With no carrier key there is nothing to tombstone and no source or `sys_metadata` row for a D2 conversion to rewrite: route 3, the #4834 / #4988 / #5055 / #6486 shape — RETIRED_DEFS_BY_MAJOR plus this entry ARE the declaration. ⚠️ The #5552 `data/ExternalFieldMapping:transform` tombstone (one of that retirement's three spellings) is SUBSUMED by the def retirement, the WidgetManifest.performance way: it goes with the shape that carried it. The base `shared/FieldMapping` tombstone and the `integration/ConnectorFieldMapping` spelling are untouched and still reject `transform` with the #5552 prescription. ⚠️ The #7990 Option-B reopen trigger ("a third measured artefact-type surface") is NOT met by this census — that ruling's parked class-level write-boundary guard stays parked; this is the ADR-0049 leg of the fork the triage pre-agreed. + - Done when: No code imports `ExternalLookup(Schema|Parsed)`, `ExternalDataSource(Schema)`, `ExternalFieldMapping(Schema|Parsed)`, `MessageQueueConfig(Schema|Parsed)`, `MessageQueueProvider(Schema)`, `TopicConfig(Schema|Parsed)`, `ConsumerConfig(Schema|Parsed)` or `DeadLetterQueue(Schema|Parsed)` from `@objectstack/spec`, `@objectstack/spec/data` or `@objectstack/spec/system` — every one is TS2305 after upgrade, on every public entry (pinned by resolved symbol identity in `data/external-lookup-retirement.test.ts` and `system/message-queue-retirement.test.ts`). No metadata document needs editing, because none could ever carry one of these shapes. `kernel/EventMessageQueueConfig` (with its inline provider enum and no credential key), `data/external-catalog.zod.ts`, `object.external` and `kernel/DeadLetterQueueEntry` survive unchanged. - **`filter-regex-options-retired`** — `data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request` → $icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement - Why not automatic: Like `driver-aggregate-undeclared-key-aliases-removed` and `driver-sql-distinct-bare-filter-typed`, this entry records a LENIENCY being withdrawn rather than a declared surface: `$regex` was never in `FILTER_OPERATORS` and never a key on `StringOperatorSchema`. That is measured, not assumed — `git log -S'$regex'` over `packages/spec/src` returns only doc comments describing how `$contains` LOWERS to MongoDB (`Contains substring - SQL: LIKE %?% | MongoDB: $regex`), plus #5701 itself, which added the name solely as `RETIRED_FILTER_OPERATORS` prescription data. ⚠️ But it differs from those two in the one way that decides the disposition, so a reader should not have to infer it: those were driver CALL ARGUMENTS, code and never stack metadata, whereas a filter IS stored metadata. `FilterConditionSchema` is an OPEN RECORD (`z.record(z.string(), z.unknown())`) because a filter key is a field name, so a stored `{ name: { $regex: 'acme.*' } }` parses GREEN and always will — a `retiredKey()` tombstone cannot exist on an open map, which is exactly why the ledger has to carry this. What such a stack used to get was four different answers from four backends: `driver-sql` and Turso's remote transport compiled it to a LIKE-escaped SUBSTRING (so `a.b` matched only the literal `a.b` and the regex was silently never a regex), `driver-memory` and objectql's `having` ran it as a real `RegExp` (so the same filter also matched `axb`, and an INVALID pattern was caught and answered `false` — zero rows, in silence), and `driver-mongodb` refused it with a bare `Error` carrying no `code` and no `status`. It is now refused everywhere with INVALID_FILTER / 400 naming the replacement. There is deliberately NO D2 conversion and this sits in `semantic` rather than among the mechanical transforms: rewriting `$regex` to `$icontains` is NOT lossless in either direction — a regex metacharacter becomes a literal — so an auto-applied rewrite would silently change which rows a dashboard, report or permission filter selects, a wrong number rather than a missing one. Choosing the substring the pattern MEANT is a judgment about the query, not a transform. ⚠️ This entry covers BOTH HALVES of the #4706 ruling (B), not just the driver one: the contract half (#5701 — the `$icontains` declaration, the `$contains` family pinned case-sensitive, and the `RETIRED_FILTER_OPERATORS` prescriptions) landed before the ADR-0087 disposition gate (#6148) existed and so was never asked for a ledger entry; the driver half (#5702) is where the refusal became executable. One surface, one entry, registered from the half that made it observable. ADR-0049 / ADR-0087, #4706 / #5701 / #5702. - Done when: No stored filter and no request `where` spells `$regex` or `$options` — grep the stack for both. Each one is rewritten by asking what the pattern MEANT, not by transliterating it: a bare substring pattern becomes `$icontains` (or `$contains` when the match must stay case-sensitive), and its metacharacters are dropped rather than escaped, because they were never honoured as a regex on the SQL family in the first place. ⚠️ Expect the answer to CHANGE on any stack that ran on `driver-memory`, `driver-mongodb` or objectql `having`, where the pattern really was evaluated as a regular expression; on the SQL family the rewritten filter returns what it always returned. A pattern that genuinely needs alternation, anchoring or character classes has no filter-level replacement — move that predicate into a formula field or a server-side view, or open an issue for it. Verify by loading the stack: a surviving `$regex` or `$options` is answered INVALID_FILTER / 400 with a message naming the replacement, on every backend. diff --git a/packages/spec/PROTOCOL_MAP.md b/packages/spec/PROTOCOL_MAP.md index ec08bb5a2d..c63d35bff4 100644 --- a/packages/spec/PROTOCOL_MAP.md +++ b/packages/spec/PROTOCOL_MAP.md @@ -34,7 +34,6 @@ This document serves as the **Grand Map** of the ObjectStack specification. It l | [`seed-loader.zod.ts`](src/data/seed-loader.zod.ts) | | **Seed Loader**. How seed rows are resolved and applied. | | [`analytics.zod.ts`](src/data/analytics.zod.ts) | | **Data Analytics**. Aggregation and multidimensional analysis types. | | [`document.zod.ts`](src/data/document.zod.ts) | | **Document**. Unstructured document storage protocol. | -| [`external-lookup.zod.ts`](src/data/external-lookup.zod.ts) | | **External Lookup**. Virtual lookups to external API data. | | [`filter.zod.ts`](src/data/filter.zod.ts) | | **Filter**. Low-level filter syntax definitions. | | [`hook.zod.ts`](src/data/hook.zod.ts) | | **Triggers/Hooks**. Database trigger definitions (before/after insert/update). | | [`mapping.zod.ts`](src/data/mapping.zod.ts) | | **Data Mapping**. Rules for transforming data between schemas. | @@ -131,7 +130,6 @@ This document serves as the **Grand Map** of the ObjectStack specification. It l | [`metrics.zod.ts`](src/system/metrics.zod.ts) | | **Observability**. Prometheus/OpenTelemetry metrics. | | [`tracing.zod.ts`](src/system/tracing.zod.ts) | | **Tracing**. Distributed tracing configuration. | | [`object-storage.zod.ts`](src/system/object-storage.zod.ts) | | **File Storage**. S3 bucket and upload configurations. | -| [`message-queue.zod.ts`](src/system/message-queue.zod.ts) | | **Message Queue**. Internal event bus settings. | | [`search-engine.zod.ts`](src/system/search-engine.zod.ts) | | **Search**. Elasticsearch/Meilisearch configuration. | | [`notification.zod.ts`](src/system/notification.zod.ts) | | **Notifications**. System-wide notification preferences. | | [`translation.zod.ts`](src/system/translation.zod.ts) | | **i18n**. Internationalization and localization. | diff --git a/packages/spec/api-surface/data.json b/packages/spec/api-surface/data.json index 38cc957e37..e1043dcff8 100644 --- a/packages/spec/api-surface/data.json +++ b/packages/spec/api-surface/data.json @@ -218,17 +218,9 @@ "ExternalColumn (type)", "ExternalColumnParsed (type)", "ExternalColumnSchema (const)", - "ExternalDataSource (type)", - "ExternalDataSourceSchema (const)", "ExternalDatasourceSettings (type)", "ExternalDatasourceSettingsParsed (type)", "ExternalDatasourceSettingsSchema (const)", - "ExternalFieldMapping (type)", - "ExternalFieldMappingParsed (type)", - "ExternalFieldMappingSchema (const)", - "ExternalLookup (type)", - "ExternalLookupParsed (type)", - "ExternalLookupSchema (const)", "ExternalTable (type)", "ExternalTableParsed (type)", "ExternalTableSchema (const)", diff --git a/packages/spec/api-surface/system.json b/packages/spec/api-surface/system.json index aef086ce3d..79324d2b8b 100644 --- a/packages/spec/api-surface/system.json +++ b/packages/spec/api-surface/system.json @@ -136,9 +136,6 @@ "ConsoleDestinationConfig (type)", "ConsoleDestinationConfigParsed (type)", "ConsoleDestinationConfigSchema (const)", - "ConsumerConfig (type)", - "ConsumerConfigParsed (type)", - "ConsumerConfigSchema (const)", "CoreServiceName (type)", "CounterOperation (type)", "CounterOperationSchema (const)", @@ -170,9 +167,6 @@ "DatabaseLevelIsolationStrategySchema (const)", "DatabaseProvider (type)", "DatabaseProviderSchema (const)", - "DeadLetterQueue (type)", - "DeadLetterQueueParsed (type)", - "DeadLetterQueueSchema (const)", "DeleteObjectOperation (type)", "DeployBundle (type)", "DeployBundleParsed (type)", @@ -340,11 +334,6 @@ "MaskingVisibilityRule (type)", "MaskingVisibilityRuleParsed (type)", "MaskingVisibilityRuleSchema (const)", - "MessageQueueConfig (type)", - "MessageQueueConfigParsed (type)", - "MessageQueueConfigSchema (const)", - "MessageQueueProvider (type)", - "MessageQueueProviderSchema (const)", "MetadataCollectionInfo (type)", "MetadataCollectionInfoSchema (const)", "MetadataDiffResult (type)", @@ -699,9 +688,6 @@ "TimeSeriesDataPoint (type)", "TimeSeriesDataPointSchema (const)", "TimeSeriesSchema (const)", - "TopicConfig (type)", - "TopicConfigParsed (type)", - "TopicConfigSchema (const)", "TraceContext (type)", "TraceContextParsed (type)", "TraceContextPropagation (type)", diff --git a/packages/spec/authorable-defaults/data.json b/packages/spec/authorable-defaults/data.json index 57ee560540..6c32c33cd2 100644 --- a/packages/spec/authorable-defaults/data.json +++ b/packages/spec/authorable-defaults/data.json @@ -38,7 +38,6 @@ "data/ExternalDatasourceSettings:allowWrites = false", "data/ExternalDatasourceSettings:queryTimeoutMs = 30000", "data/ExternalDatasourceSettings:validation = {\"checkOnBoot\":true,\"onMismatch\":\"fail\"}", - "data/ExternalFieldMapping:readonly = true", "data/Field:autonumberFormat = \"{0000}\"", "data/Field:deleteBehavior = \"set_null\"", "data/Field:externalId = false", diff --git a/packages/spec/authorable-defaults/system.json b/packages/spec/authorable-defaults/system.json index 06e9380b13..7d90866fec 100644 --- a/packages/spec/authorable-defaults/system.json +++ b/packages/spec/authorable-defaults/system.json @@ -57,9 +57,6 @@ "system/ConsoleDestinationConfig:colors = true", "system/ConsoleDestinationConfig:prettyPrint = false", "system/ConsoleDestinationConfig:stream = \"stdout\"", - "system/ConsumerConfig:autoOffsetReset = \"latest\"", - "system/ConsumerConfig:enableAutoCommit = true", - "system/ConsumerConfig:maxPollRecords = 500", "system/CronSchedule:timezone = \"UTC\"", "system/CursorStyle:opacity = 1", "system/CursorStyle:pulseOnUpdate = true", @@ -67,8 +64,6 @@ "system/DataClassificationPolicy:requireAudit = false", "system/DataClassificationPolicy:requireEncryption = false", "system/DataClassificationPolicy:requireMasking = false", - "system/DeadLetterQueue:enabled = false", - "system/DeadLetterQueue:maxRetries = 3", "system/DeployBundle:flows = []", "system/DeployBundle:objects = []", "system/DeployBundle:permissions = []", @@ -147,7 +142,6 @@ "system/MaskingVisibilityRule:auditUnmask = true", "system/MaskingVisibilityRule:defaultMasked = true", "system/MaskingVisibilityRule:requireApproval = false", - "system/MessageQueueConfig:ssl = false", "system/MetadataHistoryQueryOptions:includeMetadata = true", "system/MetadataHistoryRetentionPolicy:autoCleanup = false", "system/MetadataHistoryRetentionPolicy:cleanupIntervalHours = 24", @@ -253,9 +247,6 @@ "system/TenantUsage:currentStorageBytes = 0", "system/TenantUsage:currentUsers = 0", "system/TenantUsage:deploymentsToday = 0", - "system/TopicConfig:compressionType = \"none\"", - "system/TopicConfig:partitions = 1", - "system/TopicConfig:replicationFactor = 1", "system/TraceContext:remote = false", "system/TraceContext:sampled = true", "system/TraceContext:traceFlags = 1", diff --git a/packages/spec/authorable-surface/data.json b/packages/spec/authorable-surface/data.json index b5c17f27d5..3798cc954f 100644 --- a/packages/spec/authorable-surface/data.json +++ b/packages/spec/authorable-surface/data.json @@ -320,32 +320,11 @@ "data/ExternalColumn:primaryKey", "data/ExternalColumn:sqlType", "data/ExternalColumn:suggestedFieldType", - "data/ExternalDataSource:authentication", - "data/ExternalDataSource:endpoint", - "data/ExternalDataSource:id", - "data/ExternalDataSource:name", - "data/ExternalDataSource:type", "data/ExternalDatasourceSettings:allowWrites", "data/ExternalDatasourceSettings:allowedSchemas", "data/ExternalDatasourceSettings:credentialsRef", "data/ExternalDatasourceSettings:queryTimeoutMs", "data/ExternalDatasourceSettings:validation", - "data/ExternalFieldMapping:defaultValue", - "data/ExternalFieldMapping:readonly", - "data/ExternalFieldMapping:source", - "data/ExternalFieldMapping:target", - "data/ExternalFieldMapping:transform [RETIRED]", - "data/ExternalFieldMapping:type", - "data/ExternalLookup:caching", - "data/ExternalLookup:dataSource", - "data/ExternalLookup:fallback", - "data/ExternalLookup:fieldMappings", - "data/ExternalLookup:fieldName", - "data/ExternalLookup:pagination", - "data/ExternalLookup:query", - "data/ExternalLookup:rateLimit", - "data/ExternalLookup:retry", - "data/ExternalLookup:transform", "data/ExternalTable:columns", "data/ExternalTable:indexes", "data/ExternalTable:remoteName", diff --git a/packages/spec/authorable-surface/system.json b/packages/spec/authorable-surface/system.json index 08550612d5..34ae44afd5 100644 --- a/packages/spec/authorable-surface/system.json +++ b/packages/spec/authorable-surface/system.json @@ -259,10 +259,6 @@ "system/ConsoleDestinationConfig:colors", "system/ConsoleDestinationConfig:prettyPrint", "system/ConsoleDestinationConfig:stream", - "system/ConsumerConfig:autoOffsetReset", - "system/ConsumerConfig:enableAutoCommit", - "system/ConsumerConfig:groupId", - "system/ConsumerConfig:maxPollRecords", "system/CounterOperation:delta", "system/CounterOperation:replicaId", "system/CounterOperation:timestamp", @@ -306,9 +302,6 @@ "system/DatabaseLevelIsolationStrategy:database", "system/DatabaseLevelIsolationStrategy:encryption", "system/DatabaseLevelIsolationStrategy:strategy", - "system/DeadLetterQueue:enabled", - "system/DeadLetterQueue:maxRetries", - "system/DeadLetterQueue:queueName", "system/DeleteObjectOperation:objectName", "system/DeleteObjectOperation:type", "system/DeployBundle:flows", @@ -628,12 +621,6 @@ "system/MaskingVisibilityRule:defaultMasked", "system/MaskingVisibilityRule:requireApproval", "system/MaskingVisibilityRule:unmaskRoles", - "system/MessageQueueConfig:consumers", - "system/MessageQueueConfig:deadLetterQueue", - "system/MessageQueueConfig:provider", - "system/MessageQueueConfig:sasl", - "system/MessageQueueConfig:ssl", - "system/MessageQueueConfig:topics", "system/MetadataCollectionInfo:count", "system/MetadataCollectionInfo:namespaces", "system/MetadataCollectionInfo:type", @@ -1265,11 +1252,6 @@ "system/TimeSeriesDataPoint:labels", "system/TimeSeriesDataPoint:timestamp", "system/TimeSeriesDataPoint:value", - "system/TopicConfig:compressionType", - "system/TopicConfig:name", - "system/TopicConfig:partitions", - "system/TopicConfig:replicationFactor", - "system/TopicConfig:retentionMs", "system/TraceContext:parentSpanId", "system/TraceContext:remote", "system/TraceContext:sampled", diff --git a/packages/spec/export-origins/data.json b/packages/spec/export-origins/data.json index 074d214cd3..46753b9b51 100644 --- a/packages/spec/export-origins/data.json +++ b/packages/spec/export-origins/data.json @@ -218,17 +218,9 @@ "ExternalColumn": "src/data/external-catalog.zod.ts#ExternalColumn (type)", "ExternalColumnParsed": "src/data/external-catalog.zod.ts#ExternalColumnParsed (type)", "ExternalColumnSchema": "src/data/external-catalog.zod.ts#ExternalColumnSchema (const)", - "ExternalDataSource": "src/data/external-lookup.zod.ts#ExternalDataSource (type)", - "ExternalDataSourceSchema": "src/data/external-lookup.zod.ts#ExternalDataSourceSchema (const)", "ExternalDatasourceSettings": "src/data/datasource.zod.ts#ExternalDatasourceSettings (type)", "ExternalDatasourceSettingsParsed": "src/data/datasource.zod.ts#ExternalDatasourceSettingsParsed (type)", "ExternalDatasourceSettingsSchema": "src/data/datasource.zod.ts#ExternalDatasourceSettingsSchema (const)", - "ExternalFieldMapping": "src/data/external-lookup.zod.ts#ExternalFieldMapping (type)", - "ExternalFieldMappingParsed": "src/data/external-lookup.zod.ts#ExternalFieldMappingParsed (type)", - "ExternalFieldMappingSchema": "src/data/external-lookup.zod.ts#ExternalFieldMappingSchema (const)", - "ExternalLookup": "src/data/external-lookup.zod.ts#ExternalLookup (type)", - "ExternalLookupParsed": "src/data/external-lookup.zod.ts#ExternalLookupParsed (type)", - "ExternalLookupSchema": "src/data/external-lookup.zod.ts#ExternalLookupSchema (const)", "ExternalTable": "src/data/external-catalog.zod.ts#ExternalTable (type)", "ExternalTableParsed": "src/data/external-catalog.zod.ts#ExternalTableParsed (type)", "ExternalTableSchema": "src/data/external-catalog.zod.ts#ExternalTableSchema (const)", diff --git a/packages/spec/export-origins/system.json b/packages/spec/export-origins/system.json index 8be122e10c..80c314be33 100644 --- a/packages/spec/export-origins/system.json +++ b/packages/spec/export-origins/system.json @@ -136,9 +136,6 @@ "ConsoleDestinationConfig": "src/system/logging.zod.ts#ConsoleDestinationConfig (type)", "ConsoleDestinationConfigParsed": "src/system/logging.zod.ts#ConsoleDestinationConfigParsed (type)", "ConsoleDestinationConfigSchema": "src/system/logging.zod.ts#ConsoleDestinationConfigSchema (const)", - "ConsumerConfig": "src/system/message-queue.zod.ts#ConsumerConfig (type)", - "ConsumerConfigParsed": "src/system/message-queue.zod.ts#ConsumerConfigParsed (type)", - "ConsumerConfigSchema": "src/system/message-queue.zod.ts#ConsumerConfigSchema (const)", "CoreServiceName": "src/system/core-services.zod.ts#CoreServiceName (type)", "CounterOperation": "src/system/collaboration.zod.ts#CounterOperation (type)", "CounterOperationSchema": "src/system/collaboration.zod.ts#CounterOperationSchema (const)", @@ -170,9 +167,6 @@ "DatabaseLevelIsolationStrategySchema": "src/system/tenant.zod.ts#DatabaseLevelIsolationStrategySchema (const)", "DatabaseProvider": "src/system/tenant.zod.ts#DatabaseProvider (type)", "DatabaseProviderSchema": "src/system/tenant.zod.ts#DatabaseProviderSchema (const)", - "DeadLetterQueue": "src/system/message-queue.zod.ts#DeadLetterQueue (type)", - "DeadLetterQueueParsed": "src/system/message-queue.zod.ts#DeadLetterQueueParsed (type)", - "DeadLetterQueueSchema": "src/system/message-queue.zod.ts#DeadLetterQueueSchema (const)", "DeleteObjectOperation": "src/system/migration.zod.ts#DeleteObjectOperation (type)", "DeployBundle": "src/system/deploy-bundle.zod.ts#DeployBundle (type)", "DeployBundleParsed": "src/system/deploy-bundle.zod.ts#DeployBundleParsed (type)", @@ -340,11 +334,6 @@ "MaskingVisibilityRule": "src/system/security-context.zod.ts#MaskingVisibilityRule (type)", "MaskingVisibilityRuleParsed": "src/system/security-context.zod.ts#MaskingVisibilityRuleParsed (type)", "MaskingVisibilityRuleSchema": "src/system/security-context.zod.ts#MaskingVisibilityRuleSchema (const)", - "MessageQueueConfig": "src/system/message-queue.zod.ts#MessageQueueConfig (type)", - "MessageQueueConfigParsed": "src/system/message-queue.zod.ts#MessageQueueConfigParsed (type)", - "MessageQueueConfigSchema": "src/system/message-queue.zod.ts#MessageQueueConfigSchema (const)", - "MessageQueueProvider": "src/system/message-queue.zod.ts#MessageQueueProvider (type)", - "MessageQueueProviderSchema": "src/system/message-queue.zod.ts#MessageQueueProviderSchema (const)", "MetadataCollectionInfo": "src/system/metadata-persistence.zod.ts#MetadataCollectionInfo (type)", "MetadataCollectionInfoSchema": "src/system/metadata-persistence.zod.ts#MetadataCollectionInfoSchema (const)", "MetadataDiffResult": "src/system/metadata-persistence.zod.ts#MetadataDiffResult (type)", @@ -699,9 +688,6 @@ "TimeSeriesDataPoint": "src/system/metrics.zod.ts#TimeSeriesDataPoint (type)", "TimeSeriesDataPointSchema": "src/system/metrics.zod.ts#TimeSeriesDataPointSchema (const)", "TimeSeriesSchema": "src/system/metrics.zod.ts#TimeSeriesSchema (const)", - "TopicConfig": "src/system/message-queue.zod.ts#TopicConfig (type)", - "TopicConfigParsed": "src/system/message-queue.zod.ts#TopicConfigParsed (type)", - "TopicConfigSchema": "src/system/message-queue.zod.ts#TopicConfigSchema (const)", "TraceContext": "src/system/tracing.zod.ts#TraceContext (type)", "TraceContextParsed": "src/system/tracing.zod.ts#TraceContextParsed (type)", "TraceContextPropagation": "src/system/tracing.zod.ts#TraceContextPropagation (type)", diff --git a/packages/spec/json-schema.manifest/data.json b/packages/spec/json-schema.manifest/data.json index f5b63933bf..58e03a5eb3 100644 --- a/packages/spec/json-schema.manifest/data.json +++ b/packages/spec/json-schema.manifest/data.json @@ -71,10 +71,7 @@ "data/ExpressionBody", "data/ExternalCatalog", "data/ExternalColumn", - "data/ExternalDataSource", "data/ExternalDatasourceSettings", - "data/ExternalFieldMapping", - "data/ExternalLookup", "data/ExternalTable", "data/FeedFilterMode", "data/FeedItemType", diff --git a/packages/spec/json-schema.manifest/system.json b/packages/spec/json-schema.manifest/system.json index 640169f655..df910b6964 100644 --- a/packages/spec/json-schema.manifest/system.json +++ b/packages/spec/json-schema.manifest/system.json @@ -52,7 +52,6 @@ "system/ComplianceEncryptionRequirement", "system/ComplianceFramework", "system/ConsoleDestinationConfig", - "system/ConsumerConfig", "system/CoreServiceName", "system/CounterOperation", "system/CoverageBreakdownEntry", @@ -67,7 +66,6 @@ "system/DataMigrationFlag", "system/DatabaseLevelIsolationStrategy", "system/DatabaseProvider", - "system/DeadLetterQueue", "system/DeleteObjectOperation", "system/DeployBundle", "system/DeployDiff", @@ -134,8 +132,6 @@ "system/LoggerConfig", "system/LoggingConfig", "system/MaskingVisibilityRule", - "system/MessageQueueConfig", - "system/MessageQueueProvider", "system/MetadataCollectionInfo", "system/MetadataDiffResult", "system/MetadataFallbackStrategy", @@ -271,7 +267,6 @@ "system/TextCRDTState", "system/TimeSeries", "system/TimeSeriesDataPoint", - "system/TopicConfig", "system/TraceContext", "system/TraceContextPropagation", "system/TraceFlags", diff --git a/packages/spec/scripts/build-docs.ts b/packages/spec/scripts/build-docs.ts index 09505bd6d0..05e1a3ecac 100644 --- a/packages/spec/scripts/build-docs.ts +++ b/packages/spec/scripts/build-docs.ts @@ -494,7 +494,9 @@ const SECTION_GROUPS: Record data: [ { section: 'Objects & Fields', pages: ['object', 'field', 'validation', 'hook', 'hook-body', 'mapping'] }, { section: 'Query & Analytics', pages: ['query', 'filter', 'data-engine', 'analytics', 'date-macros'] }, - { section: 'Datasources & Drivers', pages: ['datasource', 'driver', 'driver-sql', 'driver-nosql', 'external-catalog', 'external-lookup'] }, + // `external-lookup` retired whole at #8075 (ADR-0049) — `external-catalog` + // is the surviving federated-external-data surface. + { section: 'Datasources & Drivers', pages: ['datasource', 'driver', 'driver-sql', 'driver-nosql', 'external-catalog'] }, { section: 'Documents & Seed', pages: ['document', 'seed', 'seed-loader', 'feed'] }, ], integration: [ @@ -529,7 +531,9 @@ const SECTION_GROUPS: Record // `metadata-loader` removed at #4696 — that file lives in `kernel/`, and the // two schemas `system/` re-exports from it are documented on // `system/metadata-persistence`, the file that re-exports them. - { section: 'Services & Infrastructure', pages: ['core-services', 'http-server', 'cache', 'message-queue', 'object-storage', 'search-engine', 'worker', 'job', 'notification', 'translation', 'metadata-persistence'] }, + // `message-queue` retired whole at #8075 (ADR-0049) — the live MQ surface + // is kernel's `EventMessageQueueConfig`, documented with the event bus. + { section: 'Services & Infrastructure', pages: ['core-services', 'http-server', 'cache', 'object-storage', 'search-engine', 'worker', 'job', 'notification', 'translation', 'metadata-persistence'] }, { section: 'Observability', pages: ['logging', 'metrics', 'tracing', 'audit'] }, { section: 'Security & Compliance', pages: ['encryption', 'security-context', 'incident-response', 'supplier-security', 'disaster-recovery', 'change-management', 'training'] }, { section: 'Content & Collaboration', pages: ['doc', 'book', 'collaboration'] }, diff --git a/packages/spec/scripts/lib/renamed-defs.ts b/packages/spec/scripts/lib/renamed-defs.ts index e240bb0846..5153357a96 100644 --- a/packages/spec/scripts/lib/renamed-defs.ts +++ b/packages/spec/scripts/lib/renamed-defs.ts @@ -74,7 +74,8 @@ export const RENAMED_DEFS: Readonly> = { // #4703 / ADR-0112 D9a — `FieldMapping` was published by THREE defs at once. // The two domain-specific sides take a domain prefix; `shared/FieldMapping` - // is the BASE that this rename's target (and `data/ExternalFieldMapping`) + // is the BASE that this rename's target (and, until #8075's whole-family + // retirement, `data/ExternalFieldMapping`) // extend, so it keeps the bare name and is deliberately absent from this // table. Note what an `extend` means for the invariants below: the target // def's key set is a superset of the base's, so every carried key is found, diff --git a/packages/spec/scripts/renamed-defs.test.ts b/packages/spec/scripts/renamed-defs.test.ts index 265a100d9a..47d746ed94 100644 --- a/packages/spec/scripts/renamed-defs.test.ts +++ b/packages/spec/scripts/renamed-defs.test.ts @@ -169,10 +169,10 @@ describe('the committed RENAMED_DEFS table', () => { }); it('leaves the shared BASE alone — `shared/FieldMapping` keeps the bare name', () => { - // `integration/ConnectorFieldMapping` and `data/ExternalFieldMapping` both - // `.extend()` it. Renaming the base would move keys under two other defs - // and change nothing about the collision, which was between the two - // domain-specific sides and the base's own name. + // `integration/ConnectorFieldMapping` (and `data/ExternalFieldMapping`, + // until #8075 retired that family whole) `.extend()` it. Renaming the base + // would move keys under other defs and change nothing about the collision, + // which was between the two domain-specific sides and the base's own name. expect(RENAMED_DEFS['shared/FieldMapping']).toBeUndefined(); }); diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index cb17779347..166d8dbf9d 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -700,6 +700,13 @@ "toMajor": 17, "rationale": "ADR-0049 enforce-or-remove. These eight were never a source of truth: `buildFieldMetaMap(schema)` DERIVED each one from the very `schema` its caller passed in, so the map carried a second copy of facts the caller already held. They existed for exactly one consumer — the import dry run's hand-copied pre-check mirror (`firstMissingRequiredField` / `firstConstraintViolation`, framework#3956) — and #4633 ruling D retired that mirror (PR #6532): the dry run now asks `DataProtocol.validateData` for the engine's verdict, which reads the object's own schema. That left all eight computed on every import and read by NOTHING, which is the declared-and-unread shape ADR-0049 exists for; a constraint vocabulary standing next to the presentation one with no enforcer behind it is precisely the thing an AI-authored consumer mistakes for a contract. Verified zero-reader before removal, per key and by type, across this repo (`packages/rest` itself, and all five in-repo dependents of `@objectstack/rest`: runtime, cli, verify, plugin-auth, plugin-dev) and the `objectui` sibling; plugin-auth's identity import forwards `prepared.metaMap` into `runImport` but reads only the presentation keys through `coerceRow`. Why this needs a ledger entry despite that sweep: it is the `findStream` (#4484) / `IStorageService.list` (#5540) / `actor-user-roles-to-positions` (#6011) disposition — a published TS surface with NO spec schema, so there is no `retiredKey()` tombstone and no parse rejection that could carry a prescription, and the ledger is the only channel that reaches an upgrader. It is if anything blinder than those three: the keys shipped in a FINAL release (`@objectstack/rest` 14.5.0) and have been published in every release since, and because they were OPTIONAL keys on an interface that itself survives, a JavaScript consumer reading `meta.required` after the upgrade gets `undefined` with no error at all — tsc reports at the read site only for a typed consumer. Why D3 semantic and not a D2 conversion: there is nothing to convert. No authored or stored metadata changes shape — `required` / `min` / `maxLength` and the rest remain fully authorable on a field definition and fully enforced by the engine, which is where they always lived. The only place these eight are ever spelled is inside a consumer's own TypeScript, so no `objectstack migrate meta` transform can reach them. ADR-0049 / ADR-0087, #6536 (the sweep PR #6532 deliberately deferred)." }, + { + "surface": "data.externalLookup / data.externalDataSource / data.externalFieldMapping (the whole of data/external-lookup.zod.ts — 3 defs, 8 exported names) and system.messageQueue (the whole of system/message-queue.zod.ts — MessageQueueConfig, MessageQueueProvider, TopicConfig, ConsumerConfig, DeadLetterQueue — 5 defs, 14 exported names)", + "replacement": "(removed — there is no replacement key, because there was never a key: neither family was reachable from any metadata-type binding, stack collection or /meta door, so no document could carry either. For external data: `object.external` (`ObjectExternalBindingSchema`, ADR-0015/0062) names a datasource by reference and connection credentials live in the datasource config — never inline in object metadata; `data/external-catalog.zod.ts` is that federated path's catalog surface and is untouched. For message queues: the LIVE surface is `kernel/events/integrations.zod.ts`'s `EventMessageQueueConfig` (`EventBusConfig.messageQueue`), which deliberately carries NO credential field — broker connection and SASL credentials are runtime deployment configuration, not authorable metadata. Either capability returns via the ENFORCE route of ADR-0049 through a new ADR — the executor / broker admin service first, the vocabulary second)", + "migrationId": "external-lookup-message-queue-families-retired", + "toMajor": 17, + "rationale": "Both families are the #8075 census verdict (fork (b), accepted 2026-08-12): security-shaped declared surface with inline-credential sinks and ZERO consumers. `ExternalDataSourceSchema.authentication.config` is a record of unknown whose own docblock example wrote `\"clientSecret\": \"...\"` inline, and `MessageQueueConfigSchema.sasl.password` was a required inline broker credential — the #7990 class (cleartext-at-rest credential sinks), except that unlike #7990's two measured surfaces nothing ever persisted these: no metadata-type binding (kernel/metadata-type-schemas.ts imports neither module), no stack collection, no object/field embedding (`object.external` binds `ObjectExternalBindingSchema` — remoteName/remoteSchema/writable/columnMap, no authentication), and zero imports outside packages/spec repo-wide, with the corpus-reach control (`DatasourceSchema` under identical exclusions) returning hits in the same run. The consumed MQ near-namesake `kernel/EventMessageQueueConfig` deliberately has no credential key, so the consumed shape had no credential and the credential-bearing shape had no consumer. A dead schema minus one field is still a dead schema, so the whole declarations go, not just the credential faces (#3950: an exported schema with no consumer reads as a capability to whoever finds it — here it read as an invitation to author secrets in cleartext). With no carrier key there is nothing to tombstone and no source or `sys_metadata` row for a D2 conversion to rewrite: route 3, the #4834 / #4988 / #5055 / #6486 shape — RETIRED_DEFS_BY_MAJOR plus this entry ARE the declaration. ⚠️ The #5552 `data/ExternalFieldMapping:transform` tombstone (one of that retirement's three spellings) is SUBSUMED by the def retirement, the WidgetManifest.performance way: it goes with the shape that carried it. The base `shared/FieldMapping` tombstone and the `integration/ConnectorFieldMapping` spelling are untouched and still reject `transform` with the #5552 prescription. ⚠️ The #7990 Option-B reopen trigger (\"a third measured artefact-type surface\") is NOT met by this census — that ruling's parked class-level write-boundary guard stays parked; this is the ADR-0049 leg of the fork the triage pre-agreed." + }, { "surface": "data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request", "replacement": "$icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement", @@ -1689,6 +1696,13 @@ "toMajor": 17, "rationale": "ADR-0049 enforce-or-remove. These eight were never a source of truth: `buildFieldMetaMap(schema)` DERIVED each one from the very `schema` its caller passed in, so the map carried a second copy of facts the caller already held. They existed for exactly one consumer — the import dry run's hand-copied pre-check mirror (`firstMissingRequiredField` / `firstConstraintViolation`, framework#3956) — and #4633 ruling D retired that mirror (PR #6532): the dry run now asks `DataProtocol.validateData` for the engine's verdict, which reads the object's own schema. That left all eight computed on every import and read by NOTHING, which is the declared-and-unread shape ADR-0049 exists for; a constraint vocabulary standing next to the presentation one with no enforcer behind it is precisely the thing an AI-authored consumer mistakes for a contract. Verified zero-reader before removal, per key and by type, across this repo (`packages/rest` itself, and all five in-repo dependents of `@objectstack/rest`: runtime, cli, verify, plugin-auth, plugin-dev) and the `objectui` sibling; plugin-auth's identity import forwards `prepared.metaMap` into `runImport` but reads only the presentation keys through `coerceRow`. Why this needs a ledger entry despite that sweep: it is the `findStream` (#4484) / `IStorageService.list` (#5540) / `actor-user-roles-to-positions` (#6011) disposition — a published TS surface with NO spec schema, so there is no `retiredKey()` tombstone and no parse rejection that could carry a prescription, and the ledger is the only channel that reaches an upgrader. It is if anything blinder than those three: the keys shipped in a FINAL release (`@objectstack/rest` 14.5.0) and have been published in every release since, and because they were OPTIONAL keys on an interface that itself survives, a JavaScript consumer reading `meta.required` after the upgrade gets `undefined` with no error at all — tsc reports at the read site only for a typed consumer. Why D3 semantic and not a D2 conversion: there is nothing to convert. No authored or stored metadata changes shape — `required` / `min` / `maxLength` and the rest remain fully authorable on a field definition and fully enforced by the engine, which is where they always lived. The only place these eight are ever spelled is inside a consumer's own TypeScript, so no `objectstack migrate meta` transform can reach them. ADR-0049 / ADR-0087, #6536 (the sweep PR #6532 deliberately deferred)." }, + { + "surface": "data.externalLookup / data.externalDataSource / data.externalFieldMapping (the whole of data/external-lookup.zod.ts — 3 defs, 8 exported names) and system.messageQueue (the whole of system/message-queue.zod.ts — MessageQueueConfig, MessageQueueProvider, TopicConfig, ConsumerConfig, DeadLetterQueue — 5 defs, 14 exported names)", + "replacement": "(removed — there is no replacement key, because there was never a key: neither family was reachable from any metadata-type binding, stack collection or /meta door, so no document could carry either. For external data: `object.external` (`ObjectExternalBindingSchema`, ADR-0015/0062) names a datasource by reference and connection credentials live in the datasource config — never inline in object metadata; `data/external-catalog.zod.ts` is that federated path's catalog surface and is untouched. For message queues: the LIVE surface is `kernel/events/integrations.zod.ts`'s `EventMessageQueueConfig` (`EventBusConfig.messageQueue`), which deliberately carries NO credential field — broker connection and SASL credentials are runtime deployment configuration, not authorable metadata. Either capability returns via the ENFORCE route of ADR-0049 through a new ADR — the executor / broker admin service first, the vocabulary second)", + "migrationId": "external-lookup-message-queue-families-retired", + "toMajor": 17, + "rationale": "Both families are the #8075 census verdict (fork (b), accepted 2026-08-12): security-shaped declared surface with inline-credential sinks and ZERO consumers. `ExternalDataSourceSchema.authentication.config` is a record of unknown whose own docblock example wrote `\"clientSecret\": \"...\"` inline, and `MessageQueueConfigSchema.sasl.password` was a required inline broker credential — the #7990 class (cleartext-at-rest credential sinks), except that unlike #7990's two measured surfaces nothing ever persisted these: no metadata-type binding (kernel/metadata-type-schemas.ts imports neither module), no stack collection, no object/field embedding (`object.external` binds `ObjectExternalBindingSchema` — remoteName/remoteSchema/writable/columnMap, no authentication), and zero imports outside packages/spec repo-wide, with the corpus-reach control (`DatasourceSchema` under identical exclusions) returning hits in the same run. The consumed MQ near-namesake `kernel/EventMessageQueueConfig` deliberately has no credential key, so the consumed shape had no credential and the credential-bearing shape had no consumer. A dead schema minus one field is still a dead schema, so the whole declarations go, not just the credential faces (#3950: an exported schema with no consumer reads as a capability to whoever finds it — here it read as an invitation to author secrets in cleartext). With no carrier key there is nothing to tombstone and no source or `sys_metadata` row for a D2 conversion to rewrite: route 3, the #4834 / #4988 / #5055 / #6486 shape — RETIRED_DEFS_BY_MAJOR plus this entry ARE the declaration. ⚠️ The #5552 `data/ExternalFieldMapping:transform` tombstone (one of that retirement's three spellings) is SUBSUMED by the def retirement, the WidgetManifest.performance way: it goes with the shape that carried it. The base `shared/FieldMapping` tombstone and the `integration/ConnectorFieldMapping` spelling are untouched and still reject `transform` with the #5552 prescription. ⚠️ The #7990 Option-B reopen trigger (\"a third measured artefact-type surface\") is NOT met by this census — that ruling's parked class-level write-boundary guard stays parked; this is the ADR-0049 leg of the fork the triage pre-agreed." + }, { "surface": "data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request", "replacement": "$icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement", diff --git a/packages/spec/src/conversions/registry.ts b/packages/spec/src/conversions/registry.ts index 68be2f669b..014a92ff45 100644 --- a/packages/spec/src/conversions/registry.ts +++ b/packages/spec/src/conversions/registry.ts @@ -4468,7 +4468,14 @@ const connectorRateLimitConfigRemoved: MetadataConversion = { * source that can carry the key. `ExternalLookupSchema` is not referenced by * any stack collection or metadata type, so there is no external-lookup * document for the walker to visit; its authorable key is retired by the same - * tombstone and needs no transform. + * tombstone and needs no transform. (That zero-consumer finding then became + * the whole family's verdict: #8075 retired `data/external-lookup.zod.ts` + * outright — route 3, D3 `external-lookup-message-queue-families-retired` — + * and the `data/ExternalFieldMapping:transform` retired-keys entry was + * subsumed by the def retirement, the WidgetManifest.performance way. The + * `externalLookup.…` clause in this entry's `surface` stays: it correctly + * documents what the v16→17 change list removed, and it composes with the def + * retirement into "delete the whole value".) * * `retiredFromLoadPath`: the key claims a transformation that never ran, the * `connector-rate-limit-config-removed` shape exactly. Absorbing it silently at diff --git a/packages/spec/src/data/external-lookup-retirement.test.ts b/packages/spec/src/data/external-lookup-retirement.test.ts new file mode 100644 index 0000000000..4b9e1e0e41 --- /dev/null +++ b/packages/spec/src/data/external-lookup-retirement.test.ts @@ -0,0 +1,161 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; + +import { + EXPORT_ENTRY_POINTS, + exportNamesOf, + holdersOf, +} from '../../scripts/lib/export-origins-testkit'; + +// ─── [#8075] the external-lookup family is RETIRED ────────────────────────── +// +// ADR-0049 enforce-or-remove, fork (b) of the #8075 census (accepted +// 2026-08-12): `data/external-lookup.zod.ts` is deleted whole — 3 emitted defs +// (`data/ExternalDataSource`, `data/ExternalFieldMapping`, +// `data/ExternalLookup`), 8 exported names, reference docs with them. +// +// The measurement that decided it (issue #8075, report comment; spot-verified +// on the merged tree before this removal, control passing in the SAME run): +// +// 1. STATIC — zero imports of any export outside `packages/spec` repo-wide, +// while the corpus-reach control (`DatasourceSchema` under identical +// exclusions) returns hits. In-package, the only non-test consumer of +// `ExternalDataSourceSchema` / `ExternalFieldMappingSchema` was +// `ExternalLookupSchema` in the same module, itself consumed by nothing. +// 2. DOORS — no metadata-type binding (kernel/metadata-type-schemas.ts +// imports neither module), no stack collection, no object/field +// embedding: `object.external` binds `ObjectExternalBindingSchema` +// (remoteName / remoteSchema / writable / columnMap — no authentication; +// the ADR-0015/0062 federated path routes credentials through datasource +// config). The #5552 conversion's docblock had already recorded that no +// external-lookup document exists for the conversion walker to visit. +// 3. The security face: `ExternalDataSourceSchema.authentication.config` +// was a `z.record(z.string(), z.unknown())` whose own docblock example +// wrote `"clientSecret": "..."` inline — an invitation to author OAuth +// secrets in cleartext metadata, with no consumer to ever read them. +// +// ## Why route 3, and why there is nothing to tombstone +// +// With no carrier key there is no shape on which a `retiredKey()` tombstone +// could sit, and no author document for an ADR-0087 D2 conversion to rewrite — +// a prescription nobody can receive is noise. The declared record is the D3 +// `SemanticMigration` `external-lookup-message-queue-families-retired` plus +// the `RETIRED_DEFS_BY_MAJOR[17]` entries the manifest-deletion gate reads. +// +// The #5552 `data/ExternalFieldMapping:transform` tombstone (one of that +// retirement's three spellings) is SUBSUMED rather than deleted-in-isolation: +// it goes with the shape that carried it, which is strictly stronger, because +// there is no longer a mapping shape to author the key INTO. The base +// `shared/FieldMapping` tombstone and the `integration/ConnectorFieldMapping` +// spelling are untouched — `shared/mapping.test.ts` still pins those. +// +// Form follows #4988 / #5055: resolved symbol identity over every public entry +// via the build-time `export-origins/` artifact, plus the file-deletion probe +// in the #4988 direction (whole-file retirement, no surviving occupant). +describe('[#8075] data/ external-lookup family retirement', () => { + /** The 8 names the three retired defs exported (3 schema consts + 5 types). */ + const RETIRED_NAMES = [ + 'ExternalDataSourceSchema', 'ExternalDataSource', + 'ExternalFieldMappingSchema', 'ExternalFieldMapping', 'ExternalFieldMappingParsed', + 'ExternalLookupSchema', 'ExternalLookup', 'ExternalLookupParsed', + ] as const; + + /** + * Names that must SURVIVE on `./data`. The first three are the live + * federated-external-data path the retirement's prescription points at; the + * catalog trio lives one file over from the deleted module and is exactly + * what a too-wide "finish everything external" sweep would take. + */ + const MUST_SURVIVE_DATA = [ + 'ObjectExternalBindingSchema', + 'DatasourceSchema', + 'ImportFieldMappingSchema', + 'ExternalCatalogSchema', + 'ExternalTableSchema', + 'ExternalColumnSchema', + ] as const; + + it('every retired name has ZERO holders on any public entry; the survivors still stand', () => { + // Anti-vacuity: the baseline must cover the real surface. + for (const needed of ['.', './data', './shared', './system']) { + expect(EXPORT_ENTRY_POINTS, `exports map must include ${needed}`).toContain(needed); + } + expect(exportNamesOf('./data').length, './data must export a non-trivial surface').toBeGreaterThan(100); + + // ── ABSENCE (every entry, not just ./data) ──────────────────────────── + for (const name of RETIRED_NAMES) { + expect(holdersOf(name), `${name} must have zero holders after #8075`).toEqual([]); + } + + // ── SURVIVAL ────────────────────────────────────────────────────────── + const dataNames = exportNamesOf('./data'); + for (const name of MUST_SURVIVE_DATA) { + expect(dataNames, `${name} must SURVIVE this retirement`).toContain(name); + } + // The base the deleted `ExternalFieldMappingSchema` used to extend keeps + // its bare name on ./shared — the retirement takes the extender, never + // the base. + expect(exportNamesOf('./shared')).toContain('FieldMappingSchema'); + }); + + it('the module is gone from disk, and nothing imports it any more', async () => { + const fs = await import('node:fs'); + const path = await import('node:path'); + const { fileURLToPath } = await import('node:url'); + const srcRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); + + for (const f of ['external-lookup.zod.ts', 'external-lookup.test.ts']) { + expect(fs.existsSync(path.join(srcRoot, 'data', f)), `data/${f} must be deleted`).toBe(false); + } + // Anti-vacuity: the sibling that was measured live-adjacent and KEPT must + // still be on disk, so "false" above cannot mean "wrong directory". + expect(fs.existsSync(path.join(srcRoot, 'data', 'external-catalog.zod.ts'))).toBe(true); + + const importers: string[] = []; + const walk = (dir: string) => { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) walk(full); + else if (entry.name.endsWith('.ts')) { + const src = fs.readFileSync(full, 'utf-8'); + if (/(?:import|export)[^;]*['"][^'"]*\/external-lookup\.zod(?:\.js)?['"]/.test(src)) { + importers.push(path.relative(srcRoot, full)); + } + } + } + }; + walk(srcRoot); + expect(importers, 'a resurrected import means the retirement is being undone — re-read #8075').toEqual([]); + }); + + it('runtime namespace agrees with the compiler view', async () => { + const data = await import('./index'); + for (const name of RETIRED_NAMES) { + expect(name in data, `data must not export ${name}`).toBe(false); + } + for (const name of MUST_SURVIVE_DATA) { + expect(name in data, `${name} must SURVIVE at runtime`).toBe(true); + } + }); + + it('the live external path still carries NO inline credential face', async () => { + // The retirement's argument in one assertion: the surviving federated + // binding (`object.external`) exposes no `authentication` and no secret + // slot — credentials belong to datasource config, not object metadata. If + // someone re-adds an inline credential face here, this pin asks for the + // #7990 / #8075 analysis to be re-run, not for a quiet green. + const { ObjectExternalBindingSchema } = await import('./object.zod'); + const parsed = ObjectExternalBindingSchema.safeParse({ + datasource: 'warehouse', + remoteName: 'accounts', + authentication: { type: 'oauth2', config: { clientSecret: 'x' } }, + }); + // strictObject: an inline `authentication` block is REJECTED, not stored. + expect(parsed.success).toBe(false); + if (!parsed.success) { + expect(parsed.error.issues.some((i) => i.code === 'unrecognized_keys' + && (i as { keys?: string[] }).keys?.includes('authentication'))).toBe(true); + } + }); +}); diff --git a/packages/spec/src/data/external-lookup.test.ts b/packages/spec/src/data/external-lookup.test.ts deleted file mode 100644 index d33ab6ba73..0000000000 --- a/packages/spec/src/data/external-lookup.test.ts +++ /dev/null @@ -1,817 +0,0 @@ -import { describe, it, expect } from 'vitest'; -import { - ExternalDataSourceSchema, - ExternalFieldMappingSchema, - ExternalLookupSchema, - type ExternalLookup, - type ExternalDataSource, - type ExternalFieldMapping, -} from './external-lookup.zod'; - -describe('ExternalDataSourceSchema', () => { - it('should validate complete external data source', () => { - const validSource: ExternalDataSource = { - id: 'salesforce-accounts', - name: 'Salesforce Account Data', - type: 'rest-api', - endpoint: 'https://api.salesforce.com/services/data/v58.0', - authentication: { - type: 'oauth2', - config: { - clientId: 'client_123', - clientSecret: 'secret_456', - tokenUrl: 'https://login.salesforce.com/services/oauth2/token', - }, - }, - }; - - expect(() => ExternalDataSourceSchema.parse(validSource)).not.toThrow(); - }); - - it('should accept all data source types', () => { - const types = ['odata', 'rest-api', 'graphql', 'custom'] as const; - - types.forEach((type) => { - const source = { - id: `source-${type}`, - name: `${type} Source`, - type, - endpoint: 'https://api.example.com', - authentication: { - type: 'none' as const, - config: {}, - }, - }; - - expect(() => ExternalDataSourceSchema.parse(source)).not.toThrow(); - }); - }); - - it('should accept all authentication types', () => { - const authTypes = ['oauth2', 'api-key', 'basic', 'none'] as const; - - authTypes.forEach((authType) => { - const source = { - id: `auth-${authType}`, - name: 'Test Source', - type: 'rest-api' as const, - endpoint: 'https://api.example.com', - authentication: { - type: authType, - config: {}, - }, - }; - - expect(() => ExternalDataSourceSchema.parse(source)).not.toThrow(); - }); - }); - - it('should validate API key authentication', () => { - const source = { - id: 'api-key-source', - name: 'API Key Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'api-key', - config: { - apiKey: 'sk-1234567890', - headerName: 'X-API-Key', - }, - }, - }; - - expect(() => ExternalDataSourceSchema.parse(source)).not.toThrow(); - }); - - it('should validate basic authentication', () => { - const source = { - id: 'basic-auth-source', - name: 'Basic Auth Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'basic', - config: { - username: 'user', - password: 'pass', - }, - }, - }; - - expect(() => ExternalDataSourceSchema.parse(source)).not.toThrow(); - }); - - it('should reject invalid endpoint URL', () => { - const invalidSource = { - id: 'invalid-source', - name: 'Invalid Source', - type: 'rest-api', - endpoint: 'not-a-url', - authentication: { - type: 'none', - config: {}, - }, - }; - - expect(() => ExternalDataSourceSchema.parse(invalidSource)).toThrow(); - }); -}); - -describe('ExternalFieldMappingSchema', () => { - it('should validate complete field mapping', () => { - const validMapping: ExternalFieldMapping = { - source: 'AccountName', - target: 'name', - type: 'text', - readonly: true, - }; - - expect(() => ExternalFieldMappingSchema.parse(validMapping)).not.toThrow(); - }); - - it('should accept minimal field mapping', () => { - const minimalMapping = { - source: 'ExternalField', - target: 'local_field', - type: 'text', - }; - - expect(() => ExternalFieldMappingSchema.parse(minimalMapping)).not.toThrow(); - }); - - it('should default readonly to true', () => { - const mapping = { - source: 'Field1', - target: 'field_1', - type: 'text', - }; - - const parsed = ExternalFieldMappingSchema.parse(mapping); - expect(parsed.readonly).toBe(true); - }); - - it('should accept writable field mapping', () => { - const writableMapping = { - source: 'Status', - target: 'status', - type: 'text', - readonly: false, - }; - - expect(() => ExternalFieldMappingSchema.parse(writableMapping)).not.toThrow(); - }); - - it('should accept various field types', () => { - const types = ['text', 'number', 'boolean', 'date', 'datetime', 'lookup']; - - types.forEach((type) => { - const mapping = { - source: 'Field', - target: 'field', - type, - }; - - expect(() => ExternalFieldMappingSchema.parse(mapping)).not.toThrow(); - }); - }); -}); - -describe('ExternalLookupSchema', () => { - it('should validate complete external lookup', () => { - const validLookup: ExternalLookup = { - fieldName: 'external_account', - dataSource: { - id: 'salesforce-api', - name: 'Salesforce', - type: 'rest-api', - endpoint: 'https://api.salesforce.com/services/data/v58.0', - authentication: { - type: 'oauth2', - config: { clientId: 'client_123' }, - }, - }, - query: { - endpoint: '/sobjects/Account', - method: 'GET', - parameters: { limit: 100 }, - }, - fieldMappings: [ - { - source: 'Name', - target: 'account_name', - type: 'text', - readonly: true, - }, - { - source: 'Industry', - target: 'industry', - type: 'text', - readonly: true, - }, - ], - caching: { - enabled: true, - ttl: 300, - strategy: 'ttl', - }, - fallback: { - enabled: true, - showError: true, - }, - rateLimit: { - requestsPerSecond: 10, - burstSize: 20, - }, - }; - - expect(() => ExternalLookupSchema.parse(validLookup)).not.toThrow(); - }); - - it('should accept minimal external lookup', () => { - const minimalLookup = { - fieldName: 'external_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [ - { - source: 'Field1', - target: 'field_1', - type: 'text', - }, - ], - }; - - expect(() => ExternalLookupSchema.parse(minimalLookup)).not.toThrow(); - }); - - it('should default query method to GET', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - }; - - const parsed = ExternalLookupSchema.parse(lookup); - expect(parsed.query.method).toBe('GET'); - }); - - it('should accept POST query method', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/search', - method: 'POST' as const, - parameters: { - query: 'search term', - }, - }, - fieldMappings: [], - }; - - expect(() => ExternalLookupSchema.parse(lookup)).not.toThrow(); - }); - - it('should default caching to enabled with 300s TTL', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - caching: {}, - }; - - const parsed = ExternalLookupSchema.parse(lookup); - expect(parsed.caching?.enabled).toBe(true); - expect(parsed.caching?.ttl).toBe(300); - expect(parsed.caching?.strategy).toBe('ttl'); - }); - - it('should accept all cache strategies', () => { - const strategies = ['lru', 'lfu', 'ttl'] as const; - - strategies.forEach((strategy) => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - caching: { - strategy, - }, - }; - - expect(() => ExternalLookupSchema.parse(lookup)).not.toThrow(); - }); - }); - - it('should validate custom cache TTL', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - caching: { - enabled: true, - ttl: 600, - strategy: 'ttl' as const, - }, - }; - - const parsed = ExternalLookupSchema.parse(lookup); - expect(parsed.caching?.ttl).toBe(600); - }); - - it('should default fallback to enabled with showError true', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - fallback: {}, - }; - - const parsed = ExternalLookupSchema.parse(lookup); - expect(parsed.fallback?.enabled).toBe(true); - expect(parsed.fallback?.showError).toBe(true); - }); - - it('should accept custom fallback value', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - fallback: { - enabled: true, - defaultValue: 'N/A', - showError: false, - }, - }; - - const parsed = ExternalLookupSchema.parse(lookup); - expect(parsed.fallback?.defaultValue).toBe('N/A'); - expect(parsed.fallback?.showError).toBe(false); - }); - - it('should validate rate limiting', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - rateLimit: { - requestsPerSecond: 5, - burstSize: 10, - }, - }; - - expect(() => ExternalLookupSchema.parse(lookup)).not.toThrow(); - }); - - it('should accept rate limit without burst size', () => { - const lookup = { - fieldName: 'test_field', - dataSource: { - id: 'source-1', - name: 'Source', - type: 'rest-api', - endpoint: 'https://api.example.com', - authentication: { - type: 'none', - config: {}, - }, - }, - query: { - endpoint: '/data', - }, - fieldMappings: [], - rateLimit: { - requestsPerSecond: 10, - }, - }; - - expect(() => ExternalLookupSchema.parse(lookup)).not.toThrow(); - }); - - it('should validate OData external lookup', () => { - const odataLookup = { - fieldName: 'odata_products', - dataSource: { - id: 'odata-service', - name: 'OData Product Service', - type: 'odata' as const, - endpoint: 'https://services.odata.org/V4/Northwind/Northwind.svc', - authentication: { - type: 'none' as const, - config: {}, - }, - }, - query: { - endpoint: '/Products', - method: 'GET' as const, - parameters: { - $filter: "ProductName eq 'Chai'", - $select: 'ProductID,ProductName,UnitPrice', - }, - }, - fieldMappings: [ - { - source: 'ProductID', - target: 'product_id', - type: 'number', - readonly: true, - }, - { - source: 'ProductName', - target: 'product_name', - type: 'text', - readonly: true, - }, - { - source: 'UnitPrice', - target: 'unit_price', - type: 'currency', - readonly: true, - }, - ], - }; - - expect(() => ExternalLookupSchema.parse(odataLookup)).not.toThrow(); - }); - - it('should validate GraphQL external lookup', () => { - const graphqlLookup = { - fieldName: 'graphql_users', - dataSource: { - id: 'graphql-api', - name: 'GraphQL API', - type: 'graphql' as const, - endpoint: 'https://api.example.com/graphql', - authentication: { - type: 'api-key' as const, - config: { - apiKey: 'key_123', - headerName: 'Authorization', - }, - }, - }, - query: { - endpoint: '', - method: 'POST' as const, - parameters: { - query: '{ users { id name email } }', - }, - }, - fieldMappings: [ - { - source: 'id', - target: 'user_id', - type: 'text', - readonly: true, - }, - { - source: 'name', - target: 'user_name', - type: 'text', - readonly: true, - }, - { - source: 'email', - target: 'user_email', - type: 'email', - readonly: true, - }, - ], - caching: { - enabled: true, - ttl: 180, - strategy: 'lru' as const, - }, - }; - - expect(() => ExternalLookupSchema.parse(graphqlLookup)).not.toThrow(); - }); - - it('should validate complete Salesforce-like external lookup', () => { - const salesforceLookup: ExternalLookup = { - fieldName: 'salesforce_contacts', - dataSource: { - id: 'salesforce-prod', - name: 'Salesforce Production', - type: 'rest-api', - endpoint: 'https://na1.salesforce.com/services/data/v58.0', - authentication: { - type: 'oauth2', - config: { - clientId: 'client_id', - clientSecret: 'client_secret', - tokenUrl: 'https://login.salesforce.com/services/oauth2/token', - scope: 'api', - }, - }, - }, - query: { - endpoint: '/query', - method: 'GET', - parameters: { - q: 'SELECT Id, Name, Email, Phone FROM Contact WHERE IsActive = true LIMIT 1000', - }, - }, - fieldMappings: [ - { - source: 'Id', - target: 'salesforce_id', - type: 'text', - readonly: true, - }, - { - source: 'Name', - target: 'contact_name', - type: 'text', - readonly: true, - }, - { - source: 'Email', - target: 'email', - type: 'email', - readonly: true, - }, - { - source: 'Phone', - target: 'phone', - type: 'phone', - readonly: true, - }, - ], - caching: { - enabled: true, - ttl: 600, - strategy: 'ttl', - }, - fallback: { - enabled: true, - defaultValue: null, - showError: true, - }, - rateLimit: { - requestsPerSecond: 5, - burstSize: 15, - }, - }; - - expect(() => ExternalLookupSchema.parse(salesforceLookup)).not.toThrow(); - }); -}); - -describe('External Lookup Retry Configuration', () => { - const baseLookup = { - fieldName: 'external_data', - dataSource: { - id: 'test-api', - name: 'Test API', - type: 'rest-api' as const, - endpoint: 'https://api.example.com', - authentication: { type: 'api-key' as const, config: { key: 'test' } }, - }, - query: { endpoint: '/data' }, - fieldMappings: [{ source: 'name', target: 'name' }], - }; - - it('should accept lookup with retry configuration', () => { - const lookup = { - ...baseLookup, - retry: { - maxRetries: 3, - initialDelayMs: 1000, - maxDelayMs: 30000, - backoffMultiplier: 2, - retryableStatusCodes: [429, 500, 502, 503, 504], - }, - }; - - expect(() => ExternalLookupSchema.parse(lookup)).not.toThrow(); - }); - - it('should apply retry defaults', () => { - const result = ExternalLookupSchema.parse({ - ...baseLookup, - retry: {}, - }); - - expect(result.retry?.maxRetries).toBe(3); - expect(result.retry?.initialDelayMs).toBe(1000); - expect(result.retry?.maxDelayMs).toBe(30000); - expect(result.retry?.backoffMultiplier).toBe(2); - expect(result.retry?.retryableStatusCodes).toEqual([429, 500, 502, 503, 504]); - }); - - it('should accept custom retryable status codes', () => { - const result = ExternalLookupSchema.parse({ - ...baseLookup, - retry: { retryableStatusCodes: [408, 429, 503] }, - }); - - expect(result.retry?.retryableStatusCodes).toEqual([408, 429, 503]); - }); - - it('should reject negative maxRetries', () => { - expect(() => ExternalLookupSchema.parse({ - ...baseLookup, - retry: { maxRetries: -1 }, - })).toThrow(); - }); -}); - -describe('External Lookup Transform Configuration', () => { - const baseLookup = { - fieldName: 'external_data', - dataSource: { - id: 'test-api', - name: 'Test API', - type: 'rest-api' as const, - endpoint: 'https://api.example.com', - authentication: { type: 'api-key' as const, config: { key: 'test' } }, - }, - query: { endpoint: '/data' }, - fieldMappings: [{ source: 'name', target: 'name' }], - }; - - it('should accept lookup with transform pipeline', () => { - const lookup = { - ...baseLookup, - transform: { - request: { - headers: { 'X-Custom-Header': 'value' }, - queryParams: { format: 'json' }, - }, - response: { - dataPath: '$.data.results', - totalPath: '$.meta.total', - }, - }, - }; - - expect(() => ExternalLookupSchema.parse(lookup)).not.toThrow(); - }); - - it('should accept partial transform config', () => { - const result = ExternalLookupSchema.parse({ - ...baseLookup, - transform: { - response: { dataPath: '$.items' }, - }, - }); - - expect(result.transform?.response?.dataPath).toBe('$.items'); - expect(result.transform?.request).toBeUndefined(); - }); -}); - -describe('External Lookup Pagination Configuration', () => { - const baseLookup = { - fieldName: 'external_data', - dataSource: { - id: 'test-api', - name: 'Test API', - type: 'rest-api' as const, - endpoint: 'https://api.example.com', - authentication: { type: 'api-key' as const, config: { key: 'test' } }, - }, - query: { endpoint: '/data' }, - fieldMappings: [{ source: 'name', target: 'name' }], - }; - - it('should accept lookup with pagination', () => { - const result = ExternalLookupSchema.parse({ - ...baseLookup, - pagination: { - type: 'cursor', - pageSize: 50, - maxPages: 10, - }, - }); - - expect(result.pagination?.type).toBe('cursor'); - expect(result.pagination?.pageSize).toBe(50); - }); - - it('should apply pagination defaults', () => { - const result = ExternalLookupSchema.parse({ - ...baseLookup, - pagination: {}, - }); - - expect(result.pagination?.type).toBe('offset'); - expect(result.pagination?.pageSize).toBe(100); - }); - - it('should accept all pagination types', () => { - const types = ['offset', 'cursor', 'page'] as const; - types.forEach(type => { - expect(() => ExternalLookupSchema.parse({ - ...baseLookup, - pagination: { type }, - })).not.toThrow(); - }); - }); -}); diff --git a/packages/spec/src/data/external-lookup.zod.ts b/packages/spec/src/data/external-lookup.zod.ts deleted file mode 100644 index 734f59193e..0000000000 --- a/packages/spec/src/data/external-lookup.zod.ts +++ /dev/null @@ -1,315 +0,0 @@ -// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. - -import { z } from 'zod'; -import { FieldMappingSchema as BaseFieldMappingSchema } from '../shared/mapping.zod'; - -/** - * External Data Source Schema - * - * Configuration for connecting to external data systems. - * Similar to Salesforce External Objects for real-time data integration. - * - * @example - * ```json - * { - * "id": "salesforce-accounts", - * "name": "Salesforce Account Data", - * "type": "rest-api", - * "endpoint": "https://api.salesforce.com/services/data/v58.0", - * "authentication": { - * "type": "oauth2", - * "config": { - * "clientId": "...", - * "clientSecret": "...", - * "tokenUrl": "https://login.salesforce.com/services/oauth2/token" - * } - * } - * } - * ``` - */ -import { lazySchema } from '../shared/lazy-schema'; -export const ExternalDataSourceSchema = lazySchema(() => z.object({ - /** - * Unique identifier for the external data source - */ - id: z.string().describe('Data source ID'), - - /** - * Human-readable name of the data source - */ - name: z.string().describe('Data source name'), - - /** - * Protocol type for connecting to the data source - */ - type: z.enum(['odata', 'rest-api', 'graphql', 'custom']).describe('Protocol type'), - - /** - * Base URL endpoint for the external system - */ - endpoint: z.string().url().describe('API endpoint URL'), - - /** - * Authentication configuration - */ - authentication: z.object({ - /** - * Authentication method - */ - type: z.enum(['oauth2', 'api-key', 'basic', 'none']).describe('Auth type'), - - /** - * Authentication-specific configuration - * Structure varies based on auth type - */ - config: z.record(z.string(), z.unknown()).describe('Auth configuration'), - }).describe('Authentication'), -})); - -/** - * Field Mapping Schema for External Lookups - * - * Extends the base field mapping with external lookup specific features. - * Uses the canonical field mapping protocol from shared/mapping.zod.ts. - * - * @see {@link BaseFieldMappingSchema} for the base field mapping schema - * - * @example - * ```json - * { - * "source": "AccountName", - * "target": "name", - * "readonly": true - * } - * ``` - */ -export const ExternalFieldMappingSchema = lazySchema(() => BaseFieldMappingSchema.extend({ - /** - * Field data type - */ - type: z.string().optional().describe('Field type'), - - /** - * Whether the field is read-only - * @default true - */ - readonly: z.boolean().optional().default(true).describe('Read-only field'), -})); - -/** - * External Lookup Schema - * - * Real-time data lookup protocol for external systems. - * Enables querying external data sources without replication. - * Inspired by Salesforce External Objects and OData protocols. - * - * @example - * ```json - * { - * "fieldName": "external_account", - * "dataSource": { - * "id": "salesforce-api", - * "name": "Salesforce", - * "type": "rest-api", - * "endpoint": "https://api.salesforce.com/services/data/v58.0", - * "authentication": { - * "type": "oauth2", - * "config": {"clientId": "..."} - * } - * }, - * "query": { - * "endpoint": "/sobjects/Account", - * "method": "GET", - * "parameters": {"limit": 100} - * }, - * "fieldMappings": [ - * { - * "externalField": "Name", - * "localField": "account_name", - * "type": "text", - * "readonly": true - * } - * ], - * "caching": { - * "enabled": true, - * "ttl": 300, - * "strategy": "ttl" - * }, - * "fallback": { - * "enabled": true, - * "showError": true - * }, - * "rateLimit": { - * "requestsPerSecond": 10, - * "burstSize": 20 - * } - * } - * ``` - */ -export const ExternalLookupSchema = lazySchema(() => z.object({ - /** - * Name of the field that uses external lookup - */ - fieldName: z.string().describe('Field name'), - - /** - * External data source configuration - */ - dataSource: ExternalDataSourceSchema.describe('External data source'), - - /** - * Query configuration for fetching external data - */ - query: z.object({ - /** - * API endpoint path (relative to base endpoint) - */ - endpoint: z.string().describe('Query endpoint path'), - - /** - * HTTP method for the query - * @default 'GET' - */ - method: z.enum(['GET', 'POST']).optional().default('GET').describe('HTTP method'), - - /** - * Query parameters or request body - */ - parameters: z.record(z.string(), z.unknown()).optional().describe('Query parameters'), - }).describe('Query configuration'), - - /** - * Mapping between external and local fields - */ - fieldMappings: z.array(ExternalFieldMappingSchema).describe('Field mappings'), - - /** - * Cache configuration for external data - */ - caching: z.object({ - /** - * Whether caching is enabled - * @default true - */ - enabled: z.boolean().optional().default(true).describe('Cache enabled'), - - /** - * Time-to-live in seconds - * @default 300 - */ - ttl: z.number().optional().default(300).describe('Cache TTL (seconds)'), - - /** - * Cache eviction strategy - * @default 'ttl' - */ - strategy: z.enum(['lru', 'lfu', 'ttl']).optional().default('ttl').describe('Cache strategy'), - }).optional().describe('Caching configuration'), - - /** - * Fallback behavior when external system is unavailable - */ - fallback: z.object({ - /** - * Whether fallback is enabled - * @default true - */ - enabled: z.boolean().optional().default(true).describe('Fallback enabled'), - - /** - * Default value to use when external system fails - */ - defaultValue: z.unknown().optional().describe('Default fallback value'), - - /** - * Whether to show error message to user - * @default true - */ - showError: z.boolean().optional().default(true).describe('Show error to user'), - }).optional().describe('Fallback configuration'), - - /** - * Rate limiting to prevent overwhelming external system - */ - rateLimit: z.object({ - /** - * Maximum requests per second - */ - requestsPerSecond: z.number().describe('Requests per second limit'), - - /** - * Burst size for handling spikes - */ - burstSize: z.number().optional().describe('Burst size'), - }).optional().describe('Rate limiting'), - - /** - * Retry configuration with exponential backoff - * - * @example - * ```json - * { - * "maxRetries": 3, - * "initialDelayMs": 1000, - * "maxDelayMs": 30000, - * "backoffMultiplier": 2, - * "retryableStatusCodes": [429, 500, 502, 503, 504] - * } - * ``` - */ - retry: z.object({ - /** Maximum number of retry attempts */ - maxRetries: z.number().min(0).default(3).describe('Maximum retry attempts'), - /** Initial delay before first retry (ms) */ - initialDelayMs: z.number().default(1000).describe('Initial retry delay in milliseconds'), - /** Maximum delay between retries (ms) */ - maxDelayMs: z.number().default(30000).describe('Maximum retry delay in milliseconds'), - /** Backoff multiplier for exponential backoff */ - backoffMultiplier: z.number().default(2).describe('Exponential backoff multiplier'), - /** HTTP status codes that trigger a retry */ - retryableStatusCodes: z.array(z.number()).default([429, 500, 502, 503, 504]) - .describe('HTTP status codes that are retryable'), - }).optional().describe('Retry configuration with exponential backoff'), - - /** - * Request/response transformation pipeline - * - * Allows transforming request parameters and response data - * before they are processed by the external lookup system. - */ - transform: z.object({ - /** Transform request parameters before sending */ - request: z.object({ - /** Header transformations (key-value additions) */ - headers: z.record(z.string(), z.string()).optional().describe('Additional request headers'), - /** Query parameter transformations */ - queryParams: z.record(z.string(), z.string()).optional().describe('Additional query parameters'), - }).optional().describe('Request transformation'), - /** Transform response data after receiving */ - response: z.object({ - /** JSONPath expression to extract data from response */ - dataPath: z.string().optional().describe('JSONPath to extract data (e.g., "$.data.results")'), - /** JSONPath expression to extract total count for pagination */ - totalPath: z.string().optional().describe('JSONPath to extract total count (e.g., "$.meta.total")'), - }).optional().describe('Response transformation'), - }).optional().describe('Request/response transformation pipeline'), - - /** Pagination support for external data sources */ - pagination: z.object({ - /** Pagination type */ - type: z.enum(['offset', 'cursor', 'page']).default('offset').describe('Pagination type'), - /** Page size */ - pageSize: z.number().default(100).describe('Items per page'), - /** Maximum pages to fetch */ - maxPages: z.number().optional().describe('Maximum number of pages to fetch'), - }).optional().describe('Pagination configuration for external data'), -})); - -// Type exports -export type ExternalLookup = z.input; -/** Post-parse shape of {@link ExternalLookup} — defaults applied, transforms run (ADR-0122). */ -export type ExternalLookupParsed = z.infer; -export type ExternalDataSource = z.input; -export type ExternalFieldMapping = z.input; -/** Post-parse shape of {@link ExternalFieldMapping} — defaults applied, transforms run (ADR-0122). */ -export type ExternalFieldMappingParsed = z.infer; diff --git a/packages/spec/src/data/index.ts b/packages/spec/src/data/index.ts index b677027040..36c8792d6e 100644 --- a/packages/spec/src/data/index.ts +++ b/packages/spec/src/data/index.ts @@ -105,8 +105,27 @@ export * from './seed-loader.zod'; // Document Management Protocol export * from './document.zod'; -// External Lookup Protocol -export * from './external-lookup.zod'; +// external-lookup.zod (ExternalDataSourceSchema / ExternalFieldMappingSchema / +// ExternalLookupSchema + every type alias) was REMOVED per ADR-0049 +// enforce-or-remove (#8075). The module declared a real-time external-data +// lookup protocol — a per-field external data source with an +// `authentication.config` record whose own docblock example wrote an inline +// `clientSecret` — and nothing anywhere consumed it: no metadata-type binding, +// no stack collection, no object/field embedding (`object.external` binds +// `ObjectExternalBindingSchema`, which carries no credentials), and zero +// imports outside this package in objectstack. An exported schema with no +// consumer reads as a capability to whoever finds it (#3950) — here it read as +// an invitation to put OAuth client secrets in cleartext metadata. +// +// The live mechanism external data actually goes through: `object.external` +// (`ObjectExternalBindingSchema` in object.zod.ts, ADR-0015/0062) names a +// datasource by reference, and connection credentials live in the datasource +// config (`datasource.zod.ts` / `driver/`), never inline in object metadata. +// `external-catalog.zod.ts` below is that federated path's catalog surface and +// is NOT part of this retirement. Real-time per-field lookup, if ever built, +// returns via the enforce route of ADR-0049 through a new ADR — the executor +// first, the vocabulary second. See the D3 record +// `external-lookup-message-queue-families-retired`. export * from './datasource.zod'; // Per-driver `datasource.config` contracts (#4410) — the enforcement half of diff --git a/packages/spec/src/integration/connector.test.ts b/packages/spec/src/integration/connector.test.ts index 7daef24fae..c1a67f84bc 100644 --- a/packages/spec/src/integration/connector.test.ts +++ b/packages/spec/src/integration/connector.test.ts @@ -907,13 +907,14 @@ describe('[#4911] `./integration` no longer publishes an outbound rate-limit sha // CONCEPT: the column mapping of a CSV/table import. // // The first two are base-and-superset, so "converge them" is a tempting read. -// It is wrong in both directions: widening the base to 7 keys pushes connector -// sync semantics onto `data/external-lookup.zod.ts` which also extends it -// (`automation/sync.zod.ts` embedded the base too until its retirement in -// #4738), and narrowing the connector side to 4 is a retirement of +// It is wrong in both directions: widening the base to 7 keys pushed connector +// sync semantics onto every other extender of the base +// (`data/external-lookup.zod.ts` extended it until its #8075 retirement; +// `automation/sync.zod.ts` embedded it too until its retirement in #4738), and +// narrowing the connector side to 4 is a retirement of // three live keys, not a naming fix. ADR-0112 D9a's prefix remedy applies, and -// the file next door already demonstrates it: `data/ExternalFieldMappingSchema` -// extends the same base and, purely because it carries a prefix, never entered +// the file next door already demonstrated it: `data/ExternalFieldMappingSchema` +// extended the same base and, purely because it carried a prefix, never entered // the dual-source baseline at all. // // The `./data` side is not a spelling variant of anything. The tests below pin diff --git a/packages/spec/src/integration/connector.zod.ts b/packages/spec/src/integration/connector.zod.ts index 8543de3930..e11744a862 100644 --- a/packages/spec/src/integration/connector.zod.ts +++ b/packages/spec/src/integration/connector.zod.ts @@ -153,9 +153,10 @@ import { retiredKey } from '../shared/retired-key'; * base name for the base, matching `ConnectorErrorCategory` and * `ConnectorRetryStrategy` in this same file (`ConnectorRateLimitConfig`, * #4684, was the fourth until its whole shape was retired in #4911), and - * `ExternalFieldMappingSchema` in `data/external-lookup.zod.ts` — which extends - * the same base and, precisely because it carries a domain prefix, never - * entered the dual-source baseline. + * `ExternalFieldMappingSchema` in `data/external-lookup.zod.ts` — which + * extended the same base and, precisely because it carried a domain prefix, + * never entered the dual-source baseline (the external-lookup family was + * itself retired whole in #8075 — ADR-0049, zero consumers). */ import { lazySchema } from '../shared/lazy-schema'; export const ConnectorFieldMappingSchema = lazySchema(() => BaseFieldMappingSchema.extend({ diff --git a/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalDataSource.ts b/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalDataSource.ts new file mode 100644 index 0000000000..22031d64f5 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalDataSource.ts @@ -0,0 +1,14 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — data/external-lookup.zod.ts, retired whole (ADR-0049 +// enforce-or-remove; fork (b) of the #8075 census, accepted 2026-08-12). The +// external-data-source shape whose `authentication.config` record accepted +// inline OAuth client secrets and API keys — its own docblock example wrote +// `"clientSecret": "..."`. Zero consumers: no metadata-type binding, no stack +// collection, no object/field embedding (`object.external` binds +// `ObjectExternalBindingSchema`, which routes credentials through datasource +// config per ADR-0015/0062), and its only in-module consumer +// (`ExternalLookupSchema.dataSource`) was itself consumed by nothing. Route 3: +// no tombstone, no D2 conversion — this table plus the D3 +// `external-lookup-message-queue-families-retired` are the declaration. +export const entry = 'data/ExternalDataSource'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalFieldMapping.ts b/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalFieldMapping.ts new file mode 100644 index 0000000000..6db5a64089 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalFieldMapping.ts @@ -0,0 +1,12 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — data/external-lookup.zod.ts, retired whole (ADR-0049). Extended +// `shared/FieldMapping` and was embedded only by `ExternalLookupSchema`, which +// nothing consumed. Its `transform` key's #5552 `retiredKey()` tombstone — and +// the `data/ExternalFieldMapping:transform` RETIRED_KEYS entry that registered +// it — are SUBSUMED here, the WidgetManifest.performance way: they go with the +// shape that carried them, which is strictly stronger, because there is no +// longer a mapping shape to author the key INTO. The base tombstone on +// `shared/FieldMapping` and the `integration/ConnectorFieldMapping` spelling +// are untouched and still reject the key with the #5552 prescription. +export const entry = 'data/ExternalFieldMapping'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalLookup.ts b/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalLookup.ts new file mode 100644 index 0000000000..6b2c74e8e1 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.data__ExternalLookup.ts @@ -0,0 +1,11 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — data/external-lookup.zod.ts, retired whole (ADR-0049). The root of +// the module's dependency shape: it embedded `ExternalDataSource` (the inline +// credential sink) and `ExternalFieldMapping`, and was itself referenced by no +// stack collection, metadata type or import outside packages/spec — the +// #5552 conversion's docblock had already recorded that no external-lookup +// document exists for the conversion walker to visit. Real-time external +// lookup returns via the enforce route of ADR-0049: the executor first, the +// vocabulary second. +export const entry = 'data/ExternalLookup'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.system__ConsumerConfig.ts b/packages/spec/src/migrations/entries/retired-defs/17.system__ConsumerConfig.ts new file mode 100644 index 0000000000..f0ca17f799 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.system__ConsumerConfig.ts @@ -0,0 +1,7 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). Consumer +// group config (groupId / offset reset / auto-commit / poll size), embedded +// only by `system/MessageQueueConfig` (retired in the same change) and +// consumed by nothing. +export const entry = 'system/ConsumerConfig'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.system__DeadLetterQueue.ts b/packages/spec/src/migrations/entries/retired-defs/17.system__DeadLetterQueue.ts new file mode 100644 index 0000000000..1c4c4b9473 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.system__DeadLetterQueue.ts @@ -0,0 +1,9 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). DLQ config, +// embedded only by `system/MessageQueueConfig` (retired in the same change) +// and consumed by nothing. ⚠️ `kernel/DeadLetterQueueEntry` is a different +// declaration (the event bus's per-event DLQ record) and is untouched — +// name adjacency is not evidence, the `system/ServerRateLimitConfig` note +// applies verbatim. +export const entry = 'system/DeadLetterQueue'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.system__MessageQueueConfig.ts b/packages/spec/src/migrations/entries/retired-defs/17.system__MessageQueueConfig.ts new file mode 100644 index 0000000000..6e428a2ace --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.system__MessageQueueConfig.ts @@ -0,0 +1,14 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — system/message-queue.zod.ts, retired whole (ADR-0049 +// enforce-or-remove; fork (b) of the #8075 census, accepted 2026-08-12). The +// top-level broker-administration config, carrying a required inline +// `sasl.password` credential when `sasl` was present. Zero consumers outside +// packages/spec repo-wide; no `message_queue` metadata type; the connector +// 'message_queue' ConnectorType enum value never referenced this schema. The +// consumed near-namesake `kernel/EventMessageQueueConfig` +// (`EventBusConfigSchema.messageQueue`) deliberately carries NO credential +// field and is untouched. Route 3: no tombstone, no D2 conversion — this +// table plus the D3 `external-lookup-message-queue-families-retired` are the +// declaration. +export const entry = 'system/MessageQueueConfig'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.system__MessageQueueProvider.ts b/packages/spec/src/migrations/entries/retired-defs/17.system__MessageQueueProvider.ts new file mode 100644 index 0000000000..36736ba159 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.system__MessageQueueProvider.ts @@ -0,0 +1,9 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). The +// six-value broker provider enum, embedded only by +// `system/MessageQueueConfig` (retired in the same change). Not the same +// declaration as `kernel/EventMessageQueueConfig.provider` — that is its own +// INLINE `z.enum` (same six values, never a reference to this def) and is +// untouched. +export const entry = 'system/MessageQueueProvider'; diff --git a/packages/spec/src/migrations/entries/retired-defs/17.system__TopicConfig.ts b/packages/spec/src/migrations/entries/retired-defs/17.system__TopicConfig.ts new file mode 100644 index 0000000000..42e88b4fe0 --- /dev/null +++ b/packages/spec/src/migrations/entries/retired-defs/17.system__TopicConfig.ts @@ -0,0 +1,8 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +// #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). Topic +// partitions/replication/retention/compression config, embedded only by +// `system/MessageQueueConfig` (retired in the same change) and consumed by +// nothing — leaving it behind would strand an exported schema with no +// consumer (#3950). +export const entry = 'system/TopicConfig'; diff --git a/packages/spec/src/migrations/entries/retired-keys/17.data__ExternalFieldMapping__transform.ts b/packages/spec/src/migrations/entries/retired-keys/17.data__ExternalFieldMapping__transform.ts deleted file mode 100644 index 0ce8abcc57..0000000000 --- a/packages/spec/src/migrations/entries/retired-keys/17.data__ExternalFieldMapping__transform.ts +++ /dev/null @@ -1,3 +0,0 @@ -// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. - -export const entry = 'data/ExternalFieldMapping:transform'; diff --git a/packages/spec/src/migrations/entries/semantic/17.external-lookup-message-queue-families-retired.ts b/packages/spec/src/migrations/entries/semantic/17.external-lookup-message-queue-families-retired.ts new file mode 100644 index 0000000000..58b0058a3b --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/17.external-lookup-message-queue-families-retired.ts @@ -0,0 +1,73 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +export const entry: SemanticMigration = { + id: 'external-lookup-message-queue-families-retired', + surface: + 'data.externalLookup / data.externalDataSource / data.externalFieldMapping ' + + '(the whole of data/external-lookup.zod.ts — 3 defs, 8 exported names) and ' + + 'system.messageQueue (the whole of system/message-queue.zod.ts — ' + + 'MessageQueueConfig, MessageQueueProvider, TopicConfig, ConsumerConfig, ' + + 'DeadLetterQueue — 5 defs, 14 exported names)', + replacement: + '(removed — there is no replacement key, because there was never a key: neither ' + + 'family was reachable from any metadata-type binding, stack collection or /meta ' + + 'door, so no document could carry either. For external data: `object.external` ' + + '(`ObjectExternalBindingSchema`, ADR-0015/0062) names a datasource by reference ' + + 'and connection credentials live in the datasource config — never inline in ' + + 'object metadata; `data/external-catalog.zod.ts` is that federated path\'s ' + + 'catalog surface and is untouched. For message queues: the LIVE surface is ' + + '`kernel/events/integrations.zod.ts`\'s `EventMessageQueueConfig` ' + + '(`EventBusConfig.messageQueue`), which deliberately carries NO credential ' + + 'field — broker connection and SASL credentials are runtime deployment ' + + 'configuration, not authorable metadata. Either capability returns via the ' + + 'ENFORCE route of ADR-0049 through a new ADR — the executor / broker admin ' + + 'service first, the vocabulary second)', + reason: + 'Both families are the #8075 census verdict (fork (b), accepted 2026-08-12): ' + + 'security-shaped declared surface with inline-credential sinks and ZERO ' + + 'consumers. `ExternalDataSourceSchema.authentication.config` is a record of ' + + 'unknown whose own docblock example wrote `"clientSecret": "..."` inline, and ' + + '`MessageQueueConfigSchema.sasl.password` was a required inline broker ' + + 'credential — the #7990 class (cleartext-at-rest credential sinks), except ' + + 'that unlike #7990\'s two measured surfaces nothing ever persisted these: no ' + + 'metadata-type binding (kernel/metadata-type-schemas.ts imports neither ' + + 'module), no stack collection, no object/field embedding (`object.external` ' + + 'binds `ObjectExternalBindingSchema` — remoteName/remoteSchema/writable/' + + 'columnMap, no authentication), and zero imports outside packages/spec ' + + 'repo-wide, with the corpus-reach control (`DatasourceSchema` under identical ' + + 'exclusions) returning hits in the same run. The consumed MQ near-namesake ' + + '`kernel/EventMessageQueueConfig` deliberately has no credential key, so the ' + + 'consumed shape had no credential and the credential-bearing shape had no ' + + 'consumer. A dead schema minus one field is still a dead schema, so the whole ' + + 'declarations go, not just the credential faces (#3950: an exported schema ' + + 'with no consumer reads as a capability to whoever finds it — here it read as ' + + 'an invitation to author secrets in cleartext). With no carrier key there is ' + + 'nothing to tombstone and no source or `sys_metadata` row for a D2 conversion ' + + 'to rewrite: route 3, the #4834 / #4988 / #5055 / #6486 shape — ' + + 'RETIRED_DEFS_BY_MAJOR plus this entry ARE the declaration. ' + + '⚠️ The #5552 `data/ExternalFieldMapping:transform` tombstone (one of that ' + + 'retirement\'s three spellings) is SUBSUMED by the def retirement, the ' + + 'WidgetManifest.performance way: it goes with the shape that carried it. The ' + + 'base `shared/FieldMapping` tombstone and the `integration/' + + 'ConnectorFieldMapping` spelling are untouched and still reject `transform` ' + + 'with the #5552 prescription. ' + + '⚠️ The #7990 Option-B reopen trigger ("a third measured artefact-type ' + + 'surface") is NOT met by this census — that ruling\'s parked class-level ' + + 'write-boundary guard stays parked; this is the ADR-0049 leg of the fork the ' + + 'triage pre-agreed.', + acceptanceCriteria: + 'No code imports `ExternalLookup(Schema|Parsed)`, `ExternalDataSource(Schema)`, ' + + '`ExternalFieldMapping(Schema|Parsed)`, `MessageQueueConfig(Schema|Parsed)`, ' + + '`MessageQueueProvider(Schema)`, `TopicConfig(Schema|Parsed)`, ' + + '`ConsumerConfig(Schema|Parsed)` or `DeadLetterQueue(Schema|Parsed)` from ' + + '`@objectstack/spec`, `@objectstack/spec/data` or `@objectstack/spec/system` — ' + + 'every one is TS2305 after upgrade, on every public entry (pinned by resolved ' + + 'symbol identity in `data/external-lookup-retirement.test.ts` and ' + + '`system/message-queue-retirement.test.ts`). No metadata document needs ' + + 'editing, because none could ever carry one of these shapes. ' + + '`kernel/EventMessageQueueConfig` (with its inline provider enum and no ' + + 'credential key), `data/external-catalog.zod.ts`, `object.external` and ' + + '`kernel/DeadLetterQueueEntry` survive unchanged.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 147a0f5498..e6c5309f4d 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -2834,6 +2834,75 @@ const step17: MigrationStep = { + "engine's `applyFieldDefaults` gate, so read `fields[name].defaultValue` and apply " + 'that same `!= null` test yourself.', }, + { + id: 'external-lookup-message-queue-families-retired', + surface: + 'data.externalLookup / data.externalDataSource / data.externalFieldMapping ' + + '(the whole of data/external-lookup.zod.ts — 3 defs, 8 exported names) and ' + + 'system.messageQueue (the whole of system/message-queue.zod.ts — ' + + 'MessageQueueConfig, MessageQueueProvider, TopicConfig, ConsumerConfig, ' + + 'DeadLetterQueue — 5 defs, 14 exported names)', + replacement: + '(removed — there is no replacement key, because there was never a key: neither ' + + 'family was reachable from any metadata-type binding, stack collection or /meta ' + + 'door, so no document could carry either. For external data: `object.external` ' + + '(`ObjectExternalBindingSchema`, ADR-0015/0062) names a datasource by reference ' + + 'and connection credentials live in the datasource config — never inline in ' + + 'object metadata; `data/external-catalog.zod.ts` is that federated path\'s ' + + 'catalog surface and is untouched. For message queues: the LIVE surface is ' + + '`kernel/events/integrations.zod.ts`\'s `EventMessageQueueConfig` ' + + '(`EventBusConfig.messageQueue`), which deliberately carries NO credential ' + + 'field — broker connection and SASL credentials are runtime deployment ' + + 'configuration, not authorable metadata. Either capability returns via the ' + + 'ENFORCE route of ADR-0049 through a new ADR — the executor / broker admin ' + + 'service first, the vocabulary second)', + reason: + 'Both families are the #8075 census verdict (fork (b), accepted 2026-08-12): ' + + 'security-shaped declared surface with inline-credential sinks and ZERO ' + + 'consumers. `ExternalDataSourceSchema.authentication.config` is a record of ' + + 'unknown whose own docblock example wrote `"clientSecret": "..."` inline, and ' + + '`MessageQueueConfigSchema.sasl.password` was a required inline broker ' + + 'credential — the #7990 class (cleartext-at-rest credential sinks), except ' + + 'that unlike #7990\'s two measured surfaces nothing ever persisted these: no ' + + 'metadata-type binding (kernel/metadata-type-schemas.ts imports neither ' + + 'module), no stack collection, no object/field embedding (`object.external` ' + + 'binds `ObjectExternalBindingSchema` — remoteName/remoteSchema/writable/' + + 'columnMap, no authentication), and zero imports outside packages/spec ' + + 'repo-wide, with the corpus-reach control (`DatasourceSchema` under identical ' + + 'exclusions) returning hits in the same run. The consumed MQ near-namesake ' + + '`kernel/EventMessageQueueConfig` deliberately has no credential key, so the ' + + 'consumed shape had no credential and the credential-bearing shape had no ' + + 'consumer. A dead schema minus one field is still a dead schema, so the whole ' + + 'declarations go, not just the credential faces (#3950: an exported schema ' + + 'with no consumer reads as a capability to whoever finds it — here it read as ' + + 'an invitation to author secrets in cleartext). With no carrier key there is ' + + 'nothing to tombstone and no source or `sys_metadata` row for a D2 conversion ' + + 'to rewrite: route 3, the #4834 / #4988 / #5055 / #6486 shape — ' + + 'RETIRED_DEFS_BY_MAJOR plus this entry ARE the declaration. ' + + '⚠️ The #5552 `data/ExternalFieldMapping:transform` tombstone (one of that ' + + 'retirement\'s three spellings) is SUBSUMED by the def retirement, the ' + + 'WidgetManifest.performance way: it goes with the shape that carried it. The ' + + 'base `shared/FieldMapping` tombstone and the `integration/' + + 'ConnectorFieldMapping` spelling are untouched and still reject `transform` ' + + 'with the #5552 prescription. ' + + '⚠️ The #7990 Option-B reopen trigger ("a third measured artefact-type ' + + 'surface") is NOT met by this census — that ruling\'s parked class-level ' + + 'write-boundary guard stays parked; this is the ADR-0049 leg of the fork the ' + + 'triage pre-agreed.', + acceptanceCriteria: + 'No code imports `ExternalLookup(Schema|Parsed)`, `ExternalDataSource(Schema)`, ' + + '`ExternalFieldMapping(Schema|Parsed)`, `MessageQueueConfig(Schema|Parsed)`, ' + + '`MessageQueueProvider(Schema)`, `TopicConfig(Schema|Parsed)`, ' + + '`ConsumerConfig(Schema|Parsed)` or `DeadLetterQueue(Schema|Parsed)` from ' + + '`@objectstack/spec`, `@objectstack/spec/data` or `@objectstack/spec/system` — ' + + 'every one is TS2305 after upgrade, on every public entry (pinned by resolved ' + + 'symbol identity in `data/external-lookup-retirement.test.ts` and ' + + '`system/message-queue-retirement.test.ts`). No metadata document needs ' + + 'editing, because none could ever carry one of these shapes. ' + + '`kernel/EventMessageQueueConfig` (with its inline provider enum and no ' + + 'credential key), `data/external-catalog.zod.ts`, `object.external` and ' + + '`kernel/DeadLetterQueueEntry` survive unchanged.', + }, { id: 'filter-regex-options-retired', // No backticks in `surface` — build-upgrade-guide.ts renders it inside a @@ -4451,11 +4520,14 @@ export const MIGRATION_MAJORS: readonly number[] = Object.keys(MIGRATIONS_BY_MAJ */ export const RETIRED_KEYS_BY_MAJOR: Readonly> = { // The first entries since #4659 built this table (#5552). ONE tombstone - // produces THREE keys: `transform` is declared on `shared/FieldMapping` and - // `integration/ConnectorFieldMapping` / `data/ExternalFieldMapping` are - // `.extend()`s of it, so the retired property is copied into all three walked - // shapes and `authorable-surface.json` marks each `[RETIRED]` separately. - // Registered per key, as the gate reads them — nothing radiates from the base. + // produced THREE keys: `transform` is declared on `shared/FieldMapping`, and + // every `.extend()` of the base copies the retired property into its own + // walked shape, which `authorable-surface/` marks `[RETIRED]` separately. + // Registered per key, as the gate reads them — nothing radiates from the + // base. Two of the three remain: `data/ExternalFieldMapping:transform` left + // when #8075 retired the whole external-lookup family (the def retirement + // subsumes the key entry, the WidgetManifest.performance way — see + // RETIRED_DEFS_BY_MAJOR below). 17: [ // One file per entry under `entries/retired-keys/`, concatenated here sorted by // entry id by `gen:migration-registry` (#7297). Add an entry by adding a @@ -4540,7 +4612,6 @@ export const RETIRED_KEYS_BY_MAJOR: Readonly> // takes (`query-joins-retired` / `query-cursor-retired` / // `query-distinct-retired` / `query-window-functions-retired`, #4286). 'data/AggregationNode:distinct', - 'data/ExternalFieldMapping:transform', // #7990 — sibling of `data/PostgresConfig:password`, same ruling, same // disposition: tombstoned inline credential; the secret binder / // `external.credentialsRef` is the mechanism. See that entry for the reasoning @@ -4763,6 +4834,37 @@ export const RETIRED_DEFS_BY_MAJOR: Readonly> 'automation/ETLSyncMode', 'automation/ETLTransformation', 'automation/ETLTransformationType', + // #8075 — data/external-lookup.zod.ts, retired whole (ADR-0049 + // enforce-or-remove; fork (b) of the #8075 census, accepted 2026-08-12). The + // external-data-source shape whose `authentication.config` record accepted + // inline OAuth client secrets and API keys — its own docblock example wrote + // `"clientSecret": "..."`. Zero consumers: no metadata-type binding, no stack + // collection, no object/field embedding (`object.external` binds + // `ObjectExternalBindingSchema`, which routes credentials through datasource + // config per ADR-0015/0062), and its only in-module consumer + // (`ExternalLookupSchema.dataSource`) was itself consumed by nothing. Route 3: + // no tombstone, no D2 conversion — this table plus the D3 + // `external-lookup-message-queue-families-retired` are the declaration. + 'data/ExternalDataSource', + // #8075 — data/external-lookup.zod.ts, retired whole (ADR-0049). Extended + // `shared/FieldMapping` and was embedded only by `ExternalLookupSchema`, which + // nothing consumed. Its `transform` key's #5552 `retiredKey()` tombstone — and + // the `data/ExternalFieldMapping:transform` RETIRED_KEYS entry that registered + // it — are SUBSUMED here, the WidgetManifest.performance way: they go with the + // shape that carried them, which is strictly stronger, because there is no + // longer a mapping shape to author the key INTO. The base tombstone on + // `shared/FieldMapping` and the `integration/ConnectorFieldMapping` spelling + // are untouched and still reject the key with the #5552 prescription. + 'data/ExternalFieldMapping', + // #8075 — data/external-lookup.zod.ts, retired whole (ADR-0049). The root of + // the module's dependency shape: it embedded `ExternalDataSource` (the inline + // credential sink) and `ExternalFieldMapping`, and was itself referenced by no + // stack collection, metadata type or import outside packages/spec — the + // #5552 conversion's docblock had already recorded that no external-lookup + // document exists for the conversion walker to visit. Real-time external + // lookup returns via the enforce route of ADR-0049: the executor first, the + // vocabulary second. + 'data/ExternalLookup', // #4914 — the plugin manifest's `loading` block (ADR-0049 enforce-or-remove, // maintainer ruling 2026-08-04). `PluginLoadingConfig` was reachable from // authored metadata ONLY through `Manifest.loading`, and the ten members @@ -4788,11 +4890,48 @@ export const RETIRED_DEFS_BY_MAJOR: Readonly> 'kernel/PluginPreloadConfig', 'kernel/PluginSandboxing', 'shared/FieldMappingTransform', + // #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). Consumer + // group config (groupId / offset reset / auto-commit / poll size), embedded + // only by `system/MessageQueueConfig` (retired in the same change) and + // consumed by nothing. + 'system/ConsumerConfig', + // #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). DLQ config, + // embedded only by `system/MessageQueueConfig` (retired in the same change) + // and consumed by nothing. ⚠️ `kernel/DeadLetterQueueEntry` is a different + // declaration (the event bus's per-event DLQ record) and is untouched — + // name adjacency is not evidence, the `system/ServerRateLimitConfig` note + // applies verbatim. + 'system/DeadLetterQueue', + // #8075 — system/message-queue.zod.ts, retired whole (ADR-0049 + // enforce-or-remove; fork (b) of the #8075 census, accepted 2026-08-12). The + // top-level broker-administration config, carrying a required inline + // `sasl.password` credential when `sasl` was present. Zero consumers outside + // packages/spec repo-wide; no `message_queue` metadata type; the connector + // 'message_queue' ConnectorType enum value never referenced this schema. The + // consumed near-namesake `kernel/EventMessageQueueConfig` + // (`EventBusConfigSchema.messageQueue`) deliberately carries NO credential + // field and is untouched. Route 3: no tombstone, no D2 conversion — this + // table plus the D3 `external-lookup-message-queue-families-retired` are the + // declaration. + 'system/MessageQueueConfig', + // #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). The + // six-value broker provider enum, embedded only by + // `system/MessageQueueConfig` (retired in the same change). Not the same + // declaration as `kernel/EventMessageQueueConfig.provider` — that is its own + // INLINE `z.enum` (same six values, never a reference to this def) and is + // untouched. + 'system/MessageQueueProvider', // #5295 — system/http-server.zod.ts runtime vocabulary 'system/ServerCapabilities', 'system/ServerEvent', 'system/ServerEventType', 'system/ServerStatus', + // #8075 — system/message-queue.zod.ts, retired whole (ADR-0049). Topic + // partitions/replication/retention/compression config, embedded only by + // `system/MessageQueueConfig` (retired in the same change) and consumed by + // nothing — leaving it behind would strand an exported schema with no + // consumer (#3950). + 'system/TopicConfig', 'ui/DateFormat', 'ui/I18nObject', 'ui/LocaleConfig', diff --git a/packages/spec/src/shared/mapping.test.ts b/packages/spec/src/shared/mapping.test.ts index 2a59fb9320..69d2cb1d22 100644 --- a/packages/spec/src/shared/mapping.test.ts +++ b/packages/spec/src/shared/mapping.test.ts @@ -1,7 +1,6 @@ import { describe, it, expect } from 'vitest'; import { FieldMappingSchema } from './mapping.zod'; import { ConnectorFieldMappingSchema } from '../integration/connector.zod'; -import { ExternalFieldMappingSchema } from '../data/external-lookup.zod'; describe('FieldMappingSchema', () => { it('should accept minimal valid mapping', () => { @@ -95,14 +94,16 @@ describe('[#5552] FieldMapping.transform is retired, and says so', () => { expect(result.error!.issues[0]!.message).toMatch(/os migrate meta --from 16/s); }); - it('both extenders inherit the tombstone — one retirement, three authorable spellings', () => { - // `ConnectorFieldMappingSchema` and `ExternalFieldMappingSchema` are - // `.extend()`s of the base, so the retired property is copied into their - // shapes. This is why `RETIRED_KEYS_BY_MAJOR` registers three keys for one - // tombstone; if `.extend()` ever stopped copying it, this goes red. + it('the surviving extender inherits the tombstone — one retirement, two authorable spellings', () => { + // `ConnectorFieldMappingSchema` is an `.extend()` of the base, so the + // retired property is copied into its shape. This is why + // `RETIRED_KEYS_BY_MAJOR` registers a key per walked shape — nothing + // radiates from the base; if `.extend()` ever stopped copying it, this + // goes red. (`ExternalFieldMappingSchema` was the third spelling until the + // whole external-lookup family left in #8075; its retired-keys entry was + // subsumed by the def retirement, the WidgetManifest.performance way.) for (const [name, schema] of [ ['ConnectorFieldMapping', ConnectorFieldMappingSchema], - ['ExternalFieldMapping', ExternalFieldMappingSchema], ] as const) { const result = schema.safeParse(RETIRED); expect(result.success, `${name} must reject the retired key`).toBe(false); diff --git a/packages/spec/src/shared/mapping.zod.ts b/packages/spec/src/shared/mapping.zod.ts index 073225c3ac..adb6df247f 100644 --- a/packages/spec/src/shared/mapping.zod.ts +++ b/packages/spec/src/shared/mapping.zod.ts @@ -6,14 +6,16 @@ import { retiredKey } from './retired-key'; /** * Base Field Mapping Protocol * - * Shared by: Connector, External Lookup + * Shared by: Connector * * This module provides the canonical field mapping schema used across * ObjectStack for data synchronization. * * **Use Cases:** * - Integration connectors (integration/connector.zod.ts) - * - External lookups (data/external-lookup.zod.ts) + * - (`data/external-lookup.zod.ts`'s `ExternalFieldMappingSchema` extended + * this base too, until the whole external-lookup family was retired in + * #8075 — ADR-0049, zero consumers.) * * @example Basic field mapping * ```typescript diff --git a/packages/spec/src/system/index.ts b/packages/spec/src/system/index.ts index 7f6cc704e9..f6d9f1ff95 100644 --- a/packages/spec/src/system/index.ts +++ b/packages/spec/src/system/index.ts @@ -13,7 +13,22 @@ // Infrastructure Services export * from './cache.zod'; export * from './disaster-recovery.zod'; -export * from './message-queue.zod'; +// message-queue.zod (MessageQueueProviderSchema / TopicConfigSchema / +// ConsumerConfigSchema / DeadLetterQueueSchema / MessageQueueConfigSchema +// + every type alias) was REMOVED per ADR-0049 enforce-or-remove (#8075). The +// module declared broker-administration config for six providers — partitions, +// consumer groups, DLQs — including a required inline `sasl.password` +// credential, and no runtime ever parsed or read any of it: zero consumers +// outside this package repo-wide, no `message_queue` metadata type, and the +// connector 'message_queue' ConnectorType value never referenced these shapes. +// The consumed near-namesake is `kernel/events/integrations.zod.ts`'s +// `EventMessageQueueConfigSchema` (`EventBusConfigSchema.messageQueue`), which +// deliberately carries NO credential field — broker connection/credentials are +// deployment configuration for the runtime, not authorable metadata. Queue +// administration as authorable protocol metadata returns via the enforce route +// of ADR-0049 through a new ADR — the broker admin service first, the +// vocabulary second. See the D3 record +// `external-lookup-message-queue-families-retired`. export * from './object-storage.zod'; export * from './search-engine.zod'; export * from './http-server.zod'; diff --git a/packages/spec/src/system/message-queue-retirement.test.ts b/packages/spec/src/system/message-queue-retirement.test.ts new file mode 100644 index 0000000000..b7e1baa022 --- /dev/null +++ b/packages/spec/src/system/message-queue-retirement.test.ts @@ -0,0 +1,169 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; + +import { + EXPORT_ENTRY_POINTS, + exportNamesOf, + holdersOf, +} from '../../scripts/lib/export-origins-testkit'; + +// ─── [#8075] the message-queue config family is RETIRED ───────────────────── +// +// ADR-0049 enforce-or-remove, fork (b) of the #8075 census (accepted +// 2026-08-12): `system/message-queue.zod.ts` is deleted whole — 5 emitted defs +// (`system/MessageQueueConfig`, `system/MessageQueueProvider`, +// `system/TopicConfig`, `system/ConsumerConfig`, `system/DeadLetterQueue`), +// 14 exported names, reference docs with them. +// +// The measurement that decided it (issue #8075, report comment; spot-verified +// on the merged tree before this removal, control passing in the SAME run): +// +// 1. STATIC — zero consumers outside `packages/spec` repo-wide (definition + +// own tests + generated artifacts only), while the corpus-reach control +// (`DatasourceSchema` under identical exclusions) returns hits. The +// kernel "hits" for the name are the SUBSTRING inside +// `EventMessageQueueConfigSchema` — a different declaration. +// 2. DOORS — `DEFAULT_METADATA_TYPE_REGISTRY` has no `message_queue` type; +// `integration/connector.zod.ts`'s `'message_queue'` is only a +// `ConnectorType` enum VALUE (#7990's already-measured surface) and never +// referenced these shapes. +// 3. The security face: `MessageQueueConfigSchema.sasl` required an inline +// `password` whenever present — a broker credential in authorable +// metadata — while the CONSUMED near-namesake +// (`kernel/EventMessageQueueConfig`, `EventBusConfig.messageQueue`) +// deliberately carries no credential field. The consumed MQ shape has no +// credential key; the credential-bearing MQ shape had no consumer. +// +// ## Why route 3, and why there is nothing to tombstone +// +// With no carrier key there is no shape on which a `retiredKey()` tombstone +// could sit, and no author document for an ADR-0087 D2 conversion to rewrite — +// a prescription nobody can receive is noise. The declared record is the D3 +// `SemanticMigration` `external-lookup-message-queue-families-retired` plus +// the `RETIRED_DEFS_BY_MAJOR[17]` entries the manifest-deletion gate reads. +// +// Form follows #4988 / #5055: resolved symbol identity over every public entry +// via the build-time `export-origins/` artifact, plus the file-deletion probe +// in the #4988 direction (whole-file retirement, no surviving occupant). +describe('[#8075] system/ message-queue config family retirement', () => { + /** The 14 names the five retired defs exported (5 schema consts + 9 types). */ + const RETIRED_NAMES = [ + 'MessageQueueProviderSchema', 'MessageQueueProvider', + 'TopicConfigSchema', 'TopicConfig', 'TopicConfigParsed', + 'ConsumerConfigSchema', 'ConsumerConfig', 'ConsumerConfigParsed', + 'DeadLetterQueueSchema', 'DeadLetterQueue', 'DeadLetterQueueParsed', + 'MessageQueueConfigSchema', 'MessageQueueConfig', 'MessageQueueConfigParsed', + ] as const; + + /** + * The near-namesakes a "finish everything message-queue" sweep would + * plausibly take, each a DIFFERENT declaration with a live consumer: + * the event bus's MQ integration (no credential field, inline provider + * enum) and its per-event DLQ record. + */ + const MUST_SURVIVE_KERNEL = [ + 'EventMessageQueueConfigSchema', 'EventMessageQueueConfig', 'EventMessageQueueConfigParsed', + 'EventBusConfigSchema', + 'DeadLetterQueueEntrySchema', 'DeadLetterQueueEntry', 'DeadLetterQueueEntryParsed', + ] as const; + + /** System-entry neighbours that stay. */ + const MUST_SURVIVE_SYSTEM = [ + 'CacheConfigSchema', + 'CacheTierSchema', + ] as const; + + it('every retired name has ZERO holders on any public entry; the survivors still stand', () => { + // Anti-vacuity: the baseline must cover the real surface. + for (const needed of ['.', './system', './kernel', './data']) { + expect(EXPORT_ENTRY_POINTS, `exports map must include ${needed}`).toContain(needed); + } + expect(exportNamesOf('./system').length, './system must export a non-trivial surface').toBeGreaterThan(100); + + // ── ABSENCE (every entry — exact names, so the near-namesakes cannot + // satisfy these by substring) ──────────────────────────────────────── + for (const name of RETIRED_NAMES) { + expect(holdersOf(name), `${name} must have zero holders after #8075`).toEqual([]); + } + + // ── SURVIVAL ────────────────────────────────────────────────────────── + const kernelNames = exportNamesOf('./kernel'); + for (const name of MUST_SURVIVE_KERNEL) { + expect(kernelNames, `${name} must SURVIVE this retirement`).toContain(name); + } + const systemNames = exportNamesOf('./system'); + for (const name of MUST_SURVIVE_SYSTEM) { + expect(systemNames, `${name} must SURVIVE this retirement`).toContain(name); + } + }); + + it('the module is gone from disk, and nothing imports it any more', async () => { + const fs = await import('node:fs'); + const path = await import('node:path'); + const { fileURLToPath } = await import('node:url'); + const srcRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); + + for (const f of ['message-queue.zod.ts', 'message-queue.test.ts']) { + expect(fs.existsSync(path.join(srcRoot, 'system', f)), `system/${f} must be deleted`).toBe(false); + } + // Anti-vacuity: a kept sibling proves the probe looks in the right place. + expect(fs.existsSync(path.join(srcRoot, 'system', 'cache.zod.ts'))).toBe(true); + + const importers: string[] = []; + const walk = (dir: string) => { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) walk(full); + else if (entry.name.endsWith('.ts')) { + const src = fs.readFileSync(full, 'utf-8'); + if (/(?:import|export)[^;]*['"][^'"]*\/message-queue\.zod(?:\.js)?['"]/.test(src)) { + importers.push(path.relative(srcRoot, full)); + } + } + } + }; + walk(srcRoot); + expect(importers, 'a resurrected import means the retirement is being undone — re-read #8075').toEqual([]); + }); + + it('runtime namespace agrees with the compiler view', async () => { + const system = await import('./index'); + for (const name of RETIRED_NAMES) { + expect(name in system, `system must not export ${name}`).toBe(false); + } + for (const name of MUST_SURVIVE_SYSTEM) { + expect(name in system, `${name} must SURVIVE at runtime`).toBe(true); + } + const kernel = await import('../kernel/index'); + for (const name of MUST_SURVIVE_KERNEL.filter((n) => n.endsWith('Schema'))) { + expect(name in kernel, `${name} must SURVIVE at runtime`).toBe(true); + } + }); + + it('the consumed MQ shape still carries NO credential face', async () => { + // The retirement's argument in one assertion: the LIVE message-queue + // surface (`EventBusConfig.messageQueue`) parses real configs and has no + // sasl / password / username slot. If someone re-adds a credential face + // here, this pin asks for the #7990 / #8075 analysis to be re-run, not + // for a quiet green. + const { EventMessageQueueConfigSchema } = await import('../kernel/events/integrations.zod'); + const parsed = EventMessageQueueConfigSchema.parse({ + provider: 'kafka', + topic: 'objectstack_events', + }); + expect(parsed.provider).toBe('kafka'); + // Defaults applied — the shape is genuinely parsed, not passed through. + expect(parsed.format).toBe('json'); + + // A config smuggling SASL credentials is not accepted-with-secrets: the + // schema is not `.strict()`, so zod STRIPS the unknown key — nothing + // credential-shaped survives into the parsed value. + const smuggled = EventMessageQueueConfigSchema.parse({ + provider: 'kafka', + topic: 'objectstack_events', + sasl: { mechanism: 'plain', username: 'u', password: 'p' }, + }); + expect(smuggled).not.toHaveProperty('sasl'); + }); +}); diff --git a/packages/spec/src/system/message-queue.test.ts b/packages/spec/src/system/message-queue.test.ts deleted file mode 100644 index 1d856f5c87..0000000000 --- a/packages/spec/src/system/message-queue.test.ts +++ /dev/null @@ -1,202 +0,0 @@ -import { describe, it, expect } from 'vitest'; -import { - MessageQueueProviderSchema, - TopicConfigSchema, - ConsumerConfigSchema, - DeadLetterQueueSchema, - MessageQueueConfigSchema, -} from './message-queue.zod'; - -describe('MessageQueueProviderSchema', () => { - it('should accept valid providers', () => { - const providers = ['kafka', 'rabbitmq', 'aws-sqs', 'redis-pubsub', 'google-pubsub', 'azure-service-bus']; - - providers.forEach((provider) => { - expect(() => MessageQueueProviderSchema.parse(provider)).not.toThrow(); - }); - }); - - it('should reject invalid providers', () => { - expect(() => MessageQueueProviderSchema.parse('invalid')).toThrow(); - expect(() => MessageQueueProviderSchema.parse('nats')).toThrow(); - }); -}); - -describe('TopicConfigSchema', () => { - it('should accept valid topic with defaults', () => { - const topic = TopicConfigSchema.parse({ - name: 'user_events', - }); - - expect(topic.name).toBe('user_events'); - expect(topic.partitions).toBe(1); - expect(topic.replicationFactor).toBe(1); - expect(topic.compressionType).toBe('none'); - }); - - it('should accept full topic configuration', () => { - const topic = TopicConfigSchema.parse({ - name: 'order_events', - partitions: 12, - replicationFactor: 3, - retentionMs: 604800000, - compressionType: 'snappy', - }); - - expect(topic.partitions).toBe(12); - expect(topic.replicationFactor).toBe(3); - expect(topic.retentionMs).toBe(604800000); - expect(topic.compressionType).toBe('snappy'); - }); - - it('should accept all compression types', () => { - const types = ['none', 'gzip', 'snappy', 'lz4']; - - types.forEach((type) => { - expect(() => TopicConfigSchema.parse({ name: 'test', compressionType: type })).not.toThrow(); - }); - }); - - it('should reject missing required name', () => { - expect(() => TopicConfigSchema.parse({})).toThrow(); - }); - - it('should reject invalid compression type', () => { - expect(() => TopicConfigSchema.parse({ name: 'test', compressionType: 'zstd' })).toThrow(); - }); -}); - -describe('ConsumerConfigSchema', () => { - it('should accept valid consumer with defaults', () => { - const consumer = ConsumerConfigSchema.parse({ - groupId: 'order_processor', - }); - - expect(consumer.groupId).toBe('order_processor'); - expect(consumer.autoOffsetReset).toBe('latest'); - expect(consumer.enableAutoCommit).toBe(true); - expect(consumer.maxPollRecords).toBe(500); - }); - - it('should accept full consumer configuration', () => { - const consumer = ConsumerConfigSchema.parse({ - groupId: 'analytics_consumer', - autoOffsetReset: 'earliest', - enableAutoCommit: false, - maxPollRecords: 1000, - }); - - expect(consumer.autoOffsetReset).toBe('earliest'); - expect(consumer.enableAutoCommit).toBe(false); - expect(consumer.maxPollRecords).toBe(1000); - }); - - it('should accept all autoOffsetReset values', () => { - const values = ['earliest', 'latest']; - - values.forEach((value) => { - expect(() => ConsumerConfigSchema.parse({ groupId: 'test', autoOffsetReset: value })).not.toThrow(); - }); - }); - - it('should reject missing required groupId', () => { - expect(() => ConsumerConfigSchema.parse({})).toThrow(); - }); -}); - -describe('DeadLetterQueueSchema', () => { - it('should accept valid DLQ with defaults', () => { - const dlq = DeadLetterQueueSchema.parse({ - queueName: 'failed_messages', - }); - - expect(dlq.queueName).toBe('failed_messages'); - expect(dlq.enabled).toBe(false); - expect(dlq.maxRetries).toBe(3); - }); - - it('should accept full DLQ configuration', () => { - const dlq = DeadLetterQueueSchema.parse({ - enabled: true, - maxRetries: 5, - queueName: 'order_dlq', - }); - - expect(dlq.enabled).toBe(true); - expect(dlq.maxRetries).toBe(5); - expect(dlq.queueName).toBe('order_dlq'); - }); - - it('should reject missing required queueName', () => { - expect(() => DeadLetterQueueSchema.parse({})).toThrow(); - expect(() => DeadLetterQueueSchema.parse({ enabled: true })).toThrow(); - }); -}); - -describe('MessageQueueConfigSchema', () => { - it('should accept minimal config with defaults', () => { - const config = MessageQueueConfigSchema.parse({ - provider: 'kafka', - topics: [{ name: 'events' }], - }); - - expect(config.provider).toBe('kafka'); - expect(config.topics).toHaveLength(1); - expect(config.ssl).toBe(false); - }); - - it('should accept full configuration', () => { - const config = MessageQueueConfigSchema.parse({ - provider: 'kafka', - topics: [ - { name: 'user_events', partitions: 6, replicationFactor: 3 }, - { name: 'order_events', partitions: 12, compressionType: 'gzip' }, - ], - consumers: [ - { groupId: 'user_processor' }, - { groupId: 'order_processor', autoOffsetReset: 'earliest' }, - ], - deadLetterQueue: { - enabled: true, - maxRetries: 5, - queueName: 'dlq', - }, - ssl: true, - sasl: { - mechanism: 'scram-sha-256', - username: 'admin', - password: 'secret', - }, - }); - - expect(config.topics).toHaveLength(2); - expect(config.consumers).toHaveLength(2); - expect(config.deadLetterQueue?.enabled).toBe(true); - expect(config.ssl).toBe(true); - expect(config.sasl?.mechanism).toBe('scram-sha-256'); - }); - - it('should accept all SASL mechanisms', () => { - const mechanisms = ['plain', 'scram-sha-256', 'scram-sha-512']; - - mechanisms.forEach((mechanism) => { - expect(() => MessageQueueConfigSchema.parse({ - provider: 'kafka', - topics: [{ name: 'test' }], - sasl: { mechanism, username: 'user', password: 'pass' }, - })).not.toThrow(); - }); - }); - - it('should reject missing required fields', () => { - expect(() => MessageQueueConfigSchema.parse({})).toThrow(); - expect(() => MessageQueueConfigSchema.parse({ provider: 'kafka' })).toThrow(); - }); - - it('should reject invalid provider', () => { - expect(() => MessageQueueConfigSchema.parse({ - provider: 'invalid', - topics: [{ name: 'test' }], - })).toThrow(); - }); -}); diff --git a/packages/spec/src/system/message-queue.zod.ts b/packages/spec/src/system/message-queue.zod.ts deleted file mode 100644 index 22043ac6cc..0000000000 --- a/packages/spec/src/system/message-queue.zod.ts +++ /dev/null @@ -1,69 +0,0 @@ -// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. - -import { z } from 'zod'; - -/** - * Message queue protocol for async communication - * Supports Kafka, RabbitMQ, AWS SQS, Redis Pub/Sub - */ -import { lazySchema } from '../shared/lazy-schema'; -export const MessageQueueProviderSchema = lazySchema(() => z.enum([ - 'kafka', - 'rabbitmq', - 'aws-sqs', - 'redis-pubsub', - 'google-pubsub', - 'azure-service-bus', -]).describe('Supported message queue backend provider')); - -export type MessageQueueProvider = z.input; - -export const TopicConfigSchema = lazySchema(() => z.object({ - name: z.string().describe('Topic name identifier'), - partitions: z.number().default(1).describe('Number of partitions for parallel consumption'), - replicationFactor: z.number().default(1).describe('Number of replicas for fault tolerance'), - retentionMs: z.number().optional().describe('Message retention period in milliseconds'), - compressionType: z.enum(['none', 'gzip', 'snappy', 'lz4']).default('none').describe('Message compression algorithm'), -}).describe('Configuration for a message queue topic')); - -export type TopicConfig = z.input; -/** Post-parse shape of {@link TopicConfig} — defaults applied, transforms run (ADR-0122). */ -export type TopicConfigParsed = z.infer; - -export const ConsumerConfigSchema = lazySchema(() => z.object({ - groupId: z.string().describe('Consumer group identifier'), - autoOffsetReset: z.enum(['earliest', 'latest']).default('latest').describe('Where to start reading when no offset exists'), - enableAutoCommit: z.boolean().default(true).describe('Automatically commit consumed offsets'), - maxPollRecords: z.number().default(500).describe('Maximum records returned per poll'), -}).describe('Consumer group configuration for topic consumption')); - -export type ConsumerConfig = z.input; -/** Post-parse shape of {@link ConsumerConfig} — defaults applied, transforms run (ADR-0122). */ -export type ConsumerConfigParsed = z.infer; - -export const DeadLetterQueueSchema = lazySchema(() => z.object({ - enabled: z.boolean().default(false).describe('Enable dead letter queue for failed messages'), - maxRetries: z.number().default(3).describe('Maximum delivery attempts before sending to DLQ'), - queueName: z.string().describe('Name of the dead letter queue'), -}).describe('Dead letter queue configuration for unprocessable messages')); - -export type DeadLetterQueue = z.input; -/** Post-parse shape of {@link DeadLetterQueue} — defaults applied, transforms run (ADR-0122). */ -export type DeadLetterQueueParsed = z.infer; - -export const MessageQueueConfigSchema = lazySchema(() => z.object({ - provider: MessageQueueProviderSchema.describe('Message queue backend provider'), - topics: z.array(TopicConfigSchema).describe('List of topic configurations'), - consumers: z.array(ConsumerConfigSchema).optional().describe('Consumer group configurations'), - deadLetterQueue: DeadLetterQueueSchema.optional().describe('Dead letter queue for failed messages'), - ssl: z.boolean().default(false).describe('Enable SSL/TLS for broker connections'), - sasl: z.object({ - mechanism: z.enum(['plain', 'scram-sha-256', 'scram-sha-512']).describe('SASL authentication mechanism'), - username: z.string().describe('SASL username'), - password: z.string().describe('SASL password'), - }).optional().describe('SASL authentication configuration'), -}).describe('Top-level message queue configuration')); - -export type MessageQueueConfig = z.input; -/** Post-parse shape of {@link MessageQueueConfig} — defaults applied, transforms run (ADR-0122). */ -export type MessageQueueConfigParsed = z.infer; diff --git a/packages/spec/src/type-alias-convention.pin.test.ts b/packages/spec/src/type-alias-convention.pin.test.ts index afe3ee8595..a7560dc395 100644 --- a/packages/spec/src/type-alias-convention.pin.test.ts +++ b/packages/spec/src/type-alias-convention.pin.test.ts @@ -139,7 +139,6 @@ import type * as M61 from './data/driver/common.zod.js'; import type * as M62 from './data/driver/memory.zod.js'; import type * as M63 from './data/driver/sqlite.zod.js'; import type * as M181 from './data/driver/turso.zod.js'; -import type * as M64 from './data/external-lookup.zod.js'; import type * as M65 from './data/feed.zod.js'; import type * as M66 from './data/field.zod.js'; import type * as M67 from './data/filter.zod.js'; @@ -226,7 +225,6 @@ import type * as M133 from './system/incident-response.zod.js'; import type * as M134 from './system/job.zod.js'; import type * as M135 from './system/license.zod.js'; import type * as M136 from './system/logging.zod.js'; -import type * as M137 from './system/message-queue.zod.js'; import type * as M138 from './system/metadata-persistence.zod.js'; import type * as M139 from './system/metrics.zod.js'; import type * as M140 from './system/migration.zod.js'; @@ -265,7 +263,7 @@ import type * as M167 from './ui/view.zod.js'; import type * as M170 from './ui/component.zod.js'; // --------------------------------------------------------------------------- -// 826 isomorphic aliases: `z.input` === `z.infer`, so no `XParsed` is declared. +// 824 isomorphic aliases: `z.input` === `z.infer`, so no `XParsed` is declared. // // That number is machine-checked, not hand-kept. The runtime companion at the // bottom of this file recomputes the pin count from the source and asserts that @@ -743,8 +741,7 @@ export type Iso334 = Assert, z.infer< typeof M181.TursoTransportModeSchema > >>; -// data/external-lookup.zod.ts -export type Iso335 = Assert, z.infer< typeof M64.ExternalDataSourceSchema > >>; +// data/external-lookup.zod.ts — retired whole (#8075, ADR-0049); its pin left with it. // data/feed.zod.ts export type Iso336 = Assert, z.infer< typeof M65.FeedItemType > >>; @@ -1153,8 +1150,7 @@ export type Iso583 = Assert, z.inf export type Iso584 = Assert, z.infer< typeof M136.ExternalServiceDestinationConfigSchema > >>; export type Iso585 = Assert, z.infer< typeof M136.StructuredLogEntrySchema > >>; -// system/message-queue.zod.ts -export type Iso586 = Assert, z.infer< typeof M137.MessageQueueProviderSchema > >>; +// system/message-queue.zod.ts — retired whole (#8075, ADR-0049); its pin left with it. // system/metadata-persistence.zod.ts export type Iso587 = Assert, z.infer< typeof M138.MetadataScopeSchema > >>; @@ -1625,7 +1621,7 @@ describe('ADR-0122 type-alias convention', () => { // this title and the section header above the pin list — are now asserted // against the recomputed count below, so neither can go stale without a red // test naming it. - it('still declares all 826 isomorphic pins', () => { + it('still declares all 824 isomorphic pins', () => { // The truth of each pin is proved by tsc, not here — an `Assert>` // that stops holding is a compile error with the alias named. What tsc // cannot notice is a pin that was DELETED: removing the assertion removes @@ -1812,9 +1808,17 @@ describe('ADR-0122 type-alias convention', () => { // widens input and output identically, so `Iso136` still holds and the // count moves by exactly the one new schema. Its id is `Iso837`, the next // free one — the ids are claims about pins, not positions. + // + // 826 -> 824 is #8075's ADR-0049 retirement of two whole modules: + // `data/external-lookup.zod.ts` (`Iso335`, `ExternalDataSourceSchema`) and + // `system/message-queue.zod.ts` (`Iso586`, `MessageQueueProviderSchema`). + // A pin leaves when its schema leaves — the schemas are deleted, so the + // pins are deleted with them, not re-pointed. The ids `Iso335`/`Iso586` + // are retired with their subjects and are NOT free for reuse: the ids are + // claims about pins, not positions. const self = readFileSync(fileURLToPath(import.meta.url), 'utf8'); const pins = self.match(/^export type Iso\d+ = Assert