[feature](lance) Durable Lance index job infrastructure with fence, quota and replay

[u70b3]

What problem does this PR solve?

Issue Number: #66497

Related PR: #66637 (merged), #67201 (open — independent; this PR shares no files with it), #66671 (open — independent)

Problem Summary:

This is the second sub-PR of delivery slice 3 of the v5.1 design (final 4.2 contract, scope confirmed in this review): the durable Lance index job infrastructure — job records, the compact dual-state lifecycle, the same-name fence, three-level unresolved quotas, typed results, and replay semantics — behind a master-owned manager. This PR has no user-visible entry point: no SQL, no admission, no dispatcher, no worker, no enablement gate. It is the persistence and state-machine foundation that the admission and dispatch follow-ups build on.

What this PR adds:

• The minimal durable job record of Section 7.2 (LanceIndexJob): job identity/creator/revision/bounded timestamps, persisted target identity and same-name fence key material (catalog id, DIRECTORY provider tag, normalized dataset locator, display + normalized index name), mutation intent, admitted dataset version and the ordered schema-contract-v1 representation (Section 4.2), the independent mutation/refresh states, the typed result with bounded sanitized message, dispatch identity (backend id, BE process epoch, immutable invocation id, deadline), possible-live ownership with termination proof, and the FORCE audit fields (populated by a follow-up PR; only replay semantics land here). The record carries no credentials and no unbounded values (Section 4.3/8).
• The Section 6.1/6.2 dual state machines and the Section 6.3 provider-result classification table as data plus one pure classify function (13 typed result codes × CREATE/REPLACE/DROP × IF-flags × external-advancement), including IF_CONDITION_NOOP only for DROP IF EXISTS + LANCE_ERR_NOT_FOUND. Normalization v1 (Section 4.1) for index names (toLowerCase(Locale.ROOT)) and dataset locators (scheme case, trailing slashes, credential-bearing URL rejection); the fence key and the three-level (table-locator/catalog/global) unresolved-quota counters of Section 5.4.
• LanceIndexJobManager (Appendix B seam: a master-owned minimal job/fence manager; it deliberately reuses neither the generic scheduling JobManager nor internal IndexChangeJob, neither of which provides external one-shot CAS, no-redispatch, same-name fence, or possible-live semantics). All durable transitions share one write-path shape — validate under the write lock (state legality, revision CAS, callback identity), append one upsert record to the edit log, then apply the same record locally — so master and followers run identical apply logic. Fence and unresolved quota live and die together exactly as Sections 5.4/6.4 require: held by PENDING/RUNNING, by terminal jobs until their required refresh is DONE, and by UNKNOWN until a durable FORCE_RELEASE; a quota or fence rejection precedes any durable write, leaving no job, no fence, and no record. Admission is fail-closed on corrupt metadata: a replayed unresolved record that lacks fence identity (whose fence key cannot be reconstructed) stays queryable but out of the fence/quota books, and instead blocks every new admission — with a bounded message that discloses no target identity — until a later durable record resolves it or the force-release transition of a follow-up PR releases it by job id.
• Replay per Section 7.3. replayUpsertJob is a verbatim replace with a monotonic-revision guard and performs no state transformation — a follower tailing a live master must keep a fresh RUNNING record RUNNING. The RUNNING→UNKNOWN transition happens only in the master-election sweep (Env.transferToMaster, after metadata replay and before any master daemon starts, mirroring the insertOverwriteManager.allTaskFail() precedent), which writes UNKNOWN upserts through the same identity-checked channel so followers converge and stale callbacks (revision/invocation/epoch mismatch) are rejected. Refresh RUNNING is downgraded to REQUIRED at the sweep so the idempotent external-table refresh can resume; replay never calls lance-c again.
• Persistence wiring per Doris convention: one new edit-log op (OP_LANCE_INDEX_JOB_UPSERT = 500), JournalEntity/EditLog dispatch, a new lanceIndexJobManager image module appended to PersistMetaModules.MODULE_NAMES (no FeMetaVersion bump; old images simply never invoke the load method, and Env pre-initializes an empty manager).

Explicitly not in this PR: SQL/admission/IF preflight and job SQL (the admission follow-up, including the Section 9.7 gating configuration — quota limits here are parameters, not Config); BE selection, dispatch, possible-live slot reservation on BEs, and worker invocation (the dispatch follow-up); the FORCE_RELEASE transition and its auth protocol (a follow-up PR — only the durable fields and the replay row exist); Arrow-schema→contract construction (the admission follow-up); job-record retention/GC (follows the bounded-retention policy with the force-release follow-up). PR1's SHOW INDEX and PR2's inspection surface are untouched; this branch shares no files with #67201.

Release note

None (internal infrastructure only; no user-visible behavior change).

Check List (For Author)

• Test

  • Regression test
    • None: this PR adds no SQL surface and no external behavior; regression suites for the DDL surface belong to [feature](lance) Add statically validated Lance index DDL surface #67201 and the end-to-end lifecycle to later slices.
  • Unit Test
    • New suites under org.apache.doris.datasource.lance.job (126 cases, all green with checkstyle enabled): normalization v1 (incl. the Turkish-İ corner), locator normalization and rejection forms, the full Section 6.3 classification matrix (independently restated per cell), state-machine legality (UNKNOWN has no outgoing transitions; refresh independence; revision CAS), fence/quota co-release timing, quota three-level boundaries, the Section 7.3 replay matrix (PENDING re-dispatchable once; RUNNING swept to UNKNOWN and never redispatchable; terminal jobs resume only refresh; UNKNOWN rebuilds fence/quota/possible-live; force-released UNKNOWN frees the name), stale-callback rejection, replay idempotence and monotonic revision, corrupt-record tolerance, manager image write/read round-trip with derived-state rebuild, and JournalEntity op-500 round-trip.
    • Corrupt-record fail-closed admission: replayed unresolved records that lack fence identity block all new admissions until resolved, the blockade survives the master-election sweep and image write/read round-trips, an identity-full upsert heals the record into the books, healthy jobs proceed during the blockade, and the rejection message stays bounded and non-disclosing.
    • Scoped regression green (524 cases): org.apache.doris.persist.**, org.apache.doris.journal.**, org.apache.doris.dictionary.**, org.apache.doris.indexpolicy.**, org.apache.doris.job.**, org.apache.doris.datasource.lance.** (cd fe && mvn test -pl fe-common,fe-core -am -DfailIfNoTests=false -Dtest='...', no -Dcheckstyle.skip).
  • Manual test
    • mvn compile -pl fe-common,fe-core -am green with checkstyle (validate phase); the new image module binding resolves at PersistMetaModules static init; the edit-log op code 500 verified unique repo-wide.

• Behavior changed:

  • No. New code paths are unreachable from any SQL or RPC surface in this PR; existing edit-log ops, image modules, and manager behaviors are unchanged.

• Does this need documentation?

  • No. User documentation tracks the broader lifecycle work in [Feature](lance) Add vector and scalar index lifecycle management #66497 once mutation is enabled.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[u70b3]

run buildall

[u70b3]

/review

[Gabriel39]

Could we keep admission fail-closed when an unresolved replayed record lacks fence identity? This branch stores a PENDING/RUNNING/UNKNOWN job but omits it from both the fence/quota books and getUnresolvedJobs(). After replay, a new job for the same real target can therefore pass admission and be dispatched even though the old mutation may still be live or may already have committed. That conflicts with the core invariant that ambiguous outcomes retain the fence. Since the key cannot be reconstructed, please introduce a global corrupt-unresolved admission blocker (or fail replay/startup) instead of silently excluding this record from the books.

[u70b3]

Good catch — fixed in fb70bf2.

createJob now rejects every admission while any replayed unresolved record lacks fence identity: a derived corruptUnresolvedJobIds set is maintained in applyToMemory, rebuilt on image load, and checked under the write lock before the fence CAS. The rejection is bounded (count + smallest job id) and discloses no target identity; in-flight transitions of already-admitted jobs are untouched.

I didn't make replay/startup fail: one corrupt record would then crash-loop every FE, which is what the replay-never-throws convention is for.

The blockade lifts when a later durable record resolves the job, and the force-release follow-up will release such jobs by id, without needing the fence key. Covered by new replay/persist tests; 126 lance.job + 524 scoped cases green.

[u70b3]

run buildall

[u70b3]

/review

[hello-stephen]

FE UT Coverage Report

Increment line coverage 86.84% (726/836) 🎉
Increment coverage report
Complete coverage report

[u70b3]

run buildall

[u70b3]

run buildall

[u70b3]

@Gabriel39 The unresolved-record fence blocker you raised is fixed in fb70bf2 (details in the reply above). FE UT and coverage are green; COMPILE is still running. Could you take another look / approve when you have a moment?

cc @zhangstar333 for a second approval. Thanks!