Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [feature](lance) Lance index admission, job inspection SQL and catalog DDL guard - #67630
Draft
u70b3 wants to merge 19 commits into
Draft
branch-4.1: [feature](lance) Lance index admission, job inspection SQL and catalog DDL guard#67630u70b3 wants to merge 19 commits into
u70b3 wants to merge 19 commits into
Conversation
Extend the top-level CREATE INDEX grammar with OR REPLACE and the BTREE/BITMAP index-type keywords, and carry orReplace plus the Lance-only type name on IndexDefinition without touching the persisted internal IndexDef.IndexType enum. The indexDef rule used by CREATE TABLE and ALTER TABLE ADD INDEX is unchanged, so previously parseable internal SQL behaves byte-identically; the new validate() guards only fire for SQL that was a syntax error before.
Add LanceIndexMutationValidator with the FE static bounds of the index lifecycle design: ANN/IVF_PQ property matrix, BTREE/BITMAP column-type sets, single non-null column, bounded index name, case-insensitive property keys with duplicate detection, and a fixed REST-catalog rejection. AlterTableCommand routes top-level CREATE/DROP INDEX on Lance tables through the validator and then rejects with a typed message before any Env id allocation; ALTER TABLE ADD/DROP INDEX keeps the existing generic rejection and internal tables never enter the branch. Arrow-level revalidation stays with the isolated worker, and property normalization for admission stays with the job layer.
Parser tests for the new grammar shapes and the OR REPLACE/IF NOT EXISTS exclusion, validator tests for the full static matrix, and command-level tests proving typed rejections after validation, REST messages, alter-clause fall-through, privilege-before-rejection, and that no Env id is allocated on any rejected path.
Pipeline-only suite mirroring test_lance_show_index: typed rejections and static-matrix errors on the filesystem catalog, REST-specific messages, uniform DROP INDEX IF EXISTS rejection, the generic ALTER TABLE ADD INDEX rejection, and ALTER privilege denial preceding the typed rejection. Requires the shared external docker env.
### What problem does this PR solve? Issue Number: apache#66497 Related PR: apache#67201 Problem Summary: Lance REST index DDL resolved databases and tables before returning the fixed unsupported-operation error, which could trigger remote metadata requests or expose connection errors first. Reject top-level CREATE, CREATE OR REPLACE, and DROP INDEX immediately after catalog lookup, preserve ALTER TABLE ADD/DROP behavior, and assign stable error codes to Lance index validation and unsupported-operation failures. ### Release note None ### Check List (For Author) - Test: Unit Test - LanceIndexMutationValidatorTest - AlterTableCommandLanceIndexTest - AlterTableCommandTest - IndexDefinitionTest - Standard FE build - Behavior changed: Yes. Lance REST index DDL now fails before database or table metadata resolution and Lance index errors expose stable client-visible codes. - Does this need documentation: No
### What problem does this PR solve? Issue Number: apache#66497 Related PR: apache#67201 Problem Summary: Nereids accepts an empty backquoted identifier, so CREATE INDEX `` and DROP INDEX `` reach Lance validation with an empty index name. validateCreateIndex only bounded names over 64 UTF-8 bytes, and the Directory DROP path returned the typed unsupported error before DropIndexOp.validate() could run its empty-name check, masking malformed SQL as an unsupported operation; if admission later reuses this validator, an empty logical name could reach the durable job and same-name fence. Add a shared Lance index-name validator rejecting null/empty names and names over 64 UTF-8 bytes, invoke it for Directory CREATE and DROP, and add command-level coverage for empty quoted names. The REST path retains its fail-fast unsupported response before database/table metadata resolution. ### Release note None ### Check List (For Author) - Test: Unit Test - LanceIndexMutationValidatorTest - AlterTableCommandLanceIndexTest - CreateIndexParserTest - AlterTableCommandTest - IndexDefinitionTest - Regression suite test_lance_index_ddl (external docker env, runs in CI) - Behavior changed: Yes. Blank Lance index names now fail with "index name cannot be empty" instead of the unsupported-operation error. - Does this need documentation: No
hello-stephen
commented
Sep 8, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Second sub-PR (PR3B) of slice 3 of the Lance index lifecycle design (apache#66497, v5.1 contract): the durable job model behind the one-shot mutation lifecycle. No user-visible entry point; admission, dispatch, and FORCE land in follow-up PRs. Model pieces (design sections in parentheses): - LanceIndexJob: the minimal durable job record (7.2) - identity, creator, revision, bounded timestamps, persisted target identity and same-name fence key material, mutation intent, admitted dataset version with the ordered schema-contract-v1 representation (4.2), independent mutation/refresh states, typed result with bounded sanitized message, dispatch identity (backend id, BE process epoch, immutable invocation id, deadline), possible-live ownership with termination proof, and the FORCE audit fields (populated by PR3E). No credentials, no unbounded values (4.3/8). - Dual state machines: PENDING -> RUNNING -> COMMITTED|NOT_COMMITTED| UNKNOWN (6.1, UNKNOWN terminal with no outgoing transition) and the independent NOT_REQUIRED|REQUIRED|RUNNING|DONE|FAILED refresh state (6.2). - LanceIndexJobResultCode: the provider-result classification table (6.3) as typed codes plus one pure classify(); IF_CONDITION_NOOP only for DROP IF EXISTS + LANCE_ERR_NOT_FOUND. - Normalization v1 (4.1): index names via toLowerCase(Locale.ROOT); dataset locators via trim, lowercased scheme, trailing-slash strip, and rejection of credential-bearing or identity-less forms. - LanceIndexFenceKey: (catalog id, DIRECTORY provider, normalized locator, normalized index name); display name is persisted on the job, never in the key. toString hides the locator.
…and quota PR3B part 2: the master-owned job/fence manager (Appendix B seam) plus edit-log and image wiring. It deliberately reuses neither the generic scheduling JobManager nor internal IndexChangeJob: the external one-shot CAS, no-redispatch rule, same-name fence, and possible-live ownership required by the design are not provided by either. - LanceIndexJobManager: every durable transition shares one write-path shape - validate under the write lock (state legality, revision CAS, callback identity), append one upsert record, then apply the same record locally - so master and followers run identical apply logic. Fence and unresolved quota (table-locator/catalog/global, 5.4) live and die together per 6.4: held by PENDING/RUNNING, by terminal jobs until their required refresh is DONE, and by UNKNOWN until a durable FORCE_RELEASE; rejection precedes any durable write, leaving no job, no fence, and no record. - Replay per 7.3: replayUpsertJob is a verbatim replace with a monotonic-revision guard and performs no state transformation, so a follower tailing a live master keeps a fresh RUNNING record RUNNING. RUNNING without a complete terminal result becomes UNKNOWN only in the master-election sweep (Env.transferToMaster, after metadata replay and before master daemons start, mirroring the insertOverwriteManager.allTaskFail precedent) through the same identity-checked channel; refresh RUNNING is downgraded to REQUIRED so the idempotent external-table refresh can resume. Replay never redispatches and never calls lance-c again. - Wiring: OP_LANCE_INDEX_JOB_UPSERT = 500 (verified unique), JournalEntity/EditLog dispatch, a lanceIndexJobManager image module appended to PersistMetaModules (no FeMetaVersion bump; old images never invoke the load method and Env pre-initializes an empty manager).
PR3B unit tests (105 cases, pure UT, no FE service): - Normalization v1 incl. the Turkish dotted-I corner; locator forms and rejections (userinfo, empty scheme, relative path, no identity). - Section 6.3 classification matrix independently restated per cell; IF_CONDITION_NOOP confined to DROP IF EXISTS + NOT_FOUND. - State machine legality: UNKNOWN has no outgoing transitions, refresh transitions stay independent, revision CAS, blank invocation ids rejected at the dispatch boundary. - Fence/quota co-release timing (immediate on NOT_REQUIRED, on refresh DONE, never for FAILED/UNKNOWN), three-level quota boundaries, and rebuild equivalence after image load. - Section 7.3 replay matrix: PENDING re-dispatchable once; RUNNING swept to UNKNOWN at master transfer and never redispatchable; terminal jobs resume only refresh (REQUIRED and FAILED stay visible to the refresh driver); UNKNOWN rebuilds fence/quota/possible-live; force-released UNKNOWN frees the name; stale callbacks rejected on revision/invocation/epoch mismatch; replay idempotent with a monotonic revision guard; identity-less corrupt records tolerated without throwing, including follow-up upserts for the same job id. - Manager image write/read round-trip rebuilds derived fence/quota; JournalEntity round-trip covers the new op-500 dispatch; over-bounds text fields rejected at construction.
…ved records A replayed unresolved job record that lacks fence identity (corrupt journal/image metadata: provider, locator, or index name missing) was stored and queryable but excluded from the fence and quota books, so a new job for the same real target could pass admission and be dispatched while the old mutation's outcome is still ambiguous — breaking the retain-the-fence-on-ambiguity invariant. Keep admission fail-closed instead: a derived corruptUnresolvedJobIds set tracks such records (maintained in applyToMemory, rebuilt on image load), and createJob rejects every admission under the write lock while the set is non-empty, before the fence CAS, with a bounded message that discloses no target identity. Only admission is blocked; in-flight lifecycle steps of healthy jobs keep proceeding. The blockade lifts when a later durable record settles the job, or when the force-release transition of a follow-up PR releases it by id without needing its fence key. Also rewrite the isUnresolved() javadoc: holding the fence requires the fence identity to survive, which the old wording got wrong for identity-less records.
… guard Add the PR3C admission prerequisites: the enable_lance_index_mutation gate (mutable, masterOnly, EXPERIMENTAL, default off), three positive unresolved-job quotas (per-table/catalog/global, 8/64/256) and two positive static bounds (num_partitions 4096, num_sub_vectors 256) with assigning validators that reject non-positive values; block ALTER CATALOG identity-key value changes and DROP CATALOG on a Lance catalog with unresolved index jobs while leaving credentials, same-value rewrites, renames and replay unguarded; extend LanceIndexJobManager with getAllJobsSnapshot and hasUnresolvedJobsForCatalog.
Rewire the Lance branch of AlterTableCommand behind the enable_lance_index_mutation gate (default off): validate() keeps the static validation, rejects with the new 5102 error while the gate stays off (no metadata read, no id allocation), and otherwise collects the top-level CREATE/DROP INDEX op; run() admits it through the new LanceIndexAdmission and answers a single-column JobId result set — one row when admitted, zero rows on an IF no-op — bypassing the generic op loop and alterTable entirely (for ANN this also stops an internal index id from being burned on the rejected path). Admission works off the pinned admission snapshot only: reserved __lance_ names and ambiguous case-only collisions fail closed, the IF preflight compares the requested algorithm, the corroborating physical family, the single normalized column, and each whitelist property the snapshot actually exposes (metric case-folded, numeric properties by parsed value, num_partitions never), the schema contract is built from the stored column name, REPLACE/DROP persist the stored display name of a unique case-insensitive match, and exactly one id is allocated after every preflight passes, immediately before the durable createJob transfer whose fence/quota rejections pass through verbatim. A non-positive unresolved-job quota that slipped in through fe.conf (callbacks only fire on ADMIN SET) is asserted with 5102 before any allocation. The static layer gains the matching pieces: rejectMutationDisabled (5102), the reserved __lance_ prefix rejection shared with admission so CREATE/CREATE OR REPLACE/DROP all reject it in either gate mode, and the configured static upper bounds for num_partitions and num_sub_vectors (design section 2.4). New error codes: ERR_LANCE_INDEX_MUTATION_DISABLED (5102) and ERR_LANCE_INDEX_JOB_NOT_FOUND (5103).
Add the job inspection surface of the Lance index lifecycle: SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "tbl" [AND State = "PENDING"]] lists the durable job records held by the master, and SHOW LANCE INDEX JOB <jobId> shows the single-job detail with the dispatch and FORCE audit fields. Job rows are authorized per persisted target: rows whose catalog or db/table no longer resolves (orphan and half-orphan) are visible to global ADMIN only and are omitted entirely for everyone else, leaking neither existence nor count; the detail command answers a missing job and an unauthorized job with the same fixed ERR_LANCE_INDEX_JOB_NOT_FOUND (5103) response naming only the job id. Locator, provider, normalized names, propertiesJson and the schema contract are never shown. Result and dispatch fields, which are all null before a worker exists, render as empty strings. The WHERE clause is deliberately narrowed to EqualTo predicates combined with AND over TableName/State (no Like), State values are validated against LanceIndexJobMutationState, and both commands run as FORWARD_NO_SYNC like the other master-replayed-state SHOW commands.
u70b3force-pushed
the
pr3c-lance-index-admission
branch
from
September 8, 2026 02:20
6b1bbd6 to
602d1dfCompare…merics The whitelist property comparison treated an exposed but non-primitive numeric value (compression.num_sub_vectors / num_bits as a JSON object or array) and a compression block that is not an object as "no stable value exposed" and skipped the comparison, while the metric comparison and the pinned property semantics reject the same payloads as malformed data (design section 3.4). A CREATE IF NOT EXISTS against such a snapshot could no-op as a false definition match. Make the numeric path fail closed symmetrically with the metric path and cover both payload shapes in LanceIndexAdmissionTest.
u70b3
commented
Sep 8, 2026
ContributorAuthor
run buildall |
u70b3
commented
Sep 8, 2026
ContributorAuthor
/review |
Capture and revalidate catalog target identity around the remote snapshot read, then create the durable job under the catalog guard lock. A local target generation rejects ABA changes and tentative ALTER rollbacks; IF no-ops also revalidate, and replay accepts null old properties. Use strict parsed-long equality for IF property comparisons to reject fractional and overflowing provider values. Isolate the admission regression suite and restore its original master configuration values. Validation: FE compile; 187 tests across 13 focused FE classes; Maven validate/Checkstyle; regression cleanup harness and Groovy compilation. External MinIO/REST regression and BE build were not run locally.
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue: #66497 (design v5.1, slice 3). Target:
branch-4.1.This is PR3C, stacked on #67201 (PR3A, DDL surface) and #67235 (PR3B, durable jobs). It adds the FE admission path for top-level Lance CREATE, CREATE OR REPLACE, and DROP INDEX, returns durable job IDs, and exposes job inspection SQL.
enable_lance_index_mutationremains false by default. This slice does not execute index mutations: admitted jobs remain PENDING until dispatch/worker support and FORCE_RELEASE arrive in later slices. Enabling it now leaves unresolved jobs that prevent target-changing ALTER CATALOG and DROP CATALOG.Behavior
countRows()orgetIndexStatistics(). Schema contract v1 preserves Arrow field identity, nullability, vector dimensions/element types, and relevant type parameters for later worker revalidation.num_partitionsis not compared.JobIdrow after the job and fence are durable; an IF no-op returns the same column with zero rows. The response works for direct and forwarded connections.SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "t" [AND State = "PENDING"]]andSHOW LANCE INDEX JOB <id>authorize each persisted target. Orphans require global ADMIN; unauthorized rows are omitted, and unauthorized/missing job IDs share error 5103. Locators, credentials, properties JSON, and schema-contract contents are not exposed.Concurrency and configuration
Catalog identity is captured under the CatalogMgr lock before the remote snapshot read. After the read, admission rechecks catalog existence, identity properties, and a local identity-change generation under the same lock used by catalog DDL, then allocates the ID and creates the job. This closes both the guard-check/create race and A → B → A identity changes. IF no-ops also revalidate the target. Identity changes invalidate in-flight reads even when a tentative ALTER is rolled back; same-value rewrites with unchanged key spelling and credential-only changes remain allowed.
Metadata I/O runs outside the DDL lock. Final admission uses the lock order CatalogMgr → job manager → journal. DROP and changes to
lance.catalog.type,warehouse, or namespace parent/delimiter/root_database reject unresolved jobs. Renames and replay are not blocked by the unresolved-job guard.The gate and unresolved-job quotas (table/catalog/global defaults 8/64/256) are mutable and masterOnly. ADMIN SET changes runtime values on the master; it does not persist them to the custom configuration file or automatically synchronize other FEs. Restart loads configured file values/defaults, while promotion uses the promoted FE's own configuration. The two static bounds (
num_partitions4096,num_sub_vectors256) are not masterOnly; use ADMIN SET ALL FRONTENDS CONFIG to update all FEs.ADMIN SET callbacks validate positive quotas/bounds. File-loaded quotas are checked again before ID allocation; the job manager independently rejects non-positive limits. Gate-off admission uses error 5102.
Job SHOW commands use FORWARD_NO_SYNC: follower requests execute on the master, without waiting for follower journal replay. This does not make SHOW a follower-local stale read.
Validation
git diff --checkpassed.The external regression suite runs as
nonConcurrent, captures the original master gate/quota values, and restores them even on failure. Cleanup failures are reported. Admitted PENDING jobs/catalogs remain durable in this slice; per-run names avoid same-name collisions, but repeated runs still consume the global unresolved-job quota.Deferred
Dispatch, BE selection, worker execution, possible-live slot configuration, and the local/file operator assertion belong to subsequent dispatch/worker slices. FORCE_RELEASE/RESOLVE and retention GC belong to PR3E. The gate default remains unchanged. REST catalog mutations, ALTER TABLE ADD/DROP INDEX, and BUILD INDEX remain unsupported.
Release note
Experimental, default-disabled FE admission and job inspection for Lance index mutations; actual index execution remains deferred.