Skip to content

Complete hotspot split M2 migration lifecycle - #1096

Merged
bootjp merged 11 commits into
design/hotspot-split-m2-promotion-completefrom
design/hotspot-split-m2-runner
Jul 19, 2026
Merged

Complete hotspot split M2 migration lifecycle#1096
bootjp merged 11 commits into
design/hotspot-split-m2-promotion-completefrom
design/hotspot-split-m2-runner

Conversation

@bootjp

@bootjpbootjp commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Summary

  • Complete the Milestone 2 split migration lifecycle from PLANNED through BACKFILL, FENCE, DELTA_COPY, CUTOVER, CLEANUP, DONE, and pre-cutover ABANDON.
  • Require durable per-voter source/target barriers, including membership changes, source-issued snapshot and fence timestamps, cutover read fencing, cleanup proofs, and bounded job history.
  • Add bounded physical cleanup for source and abandoned target data, plus metadata cleanup and promotion completion proofs.
  • Add split CLI job inspection/abandon operations and a Jepsen split workload with concurrent left/right register traffic, migration polling, faults, and FENCE abandonment coverage.

Safety

  • The source write tracker and no-prune retention pin are applied and acknowledged by every current source voter before the snapshot timestamp is selected.
  • Snapshot and fence timestamps are issued by the source group after a linearizable leader check and are strictly greater than its last committed timestamp.
  • CUTOVER waits for target readiness and source read-fence acknowledgements from every current voter; membership or address changes reopen the barrier.
  • CLEANUP waits for descriptor visibility, read-drain grace, bounded physical deletion, and per-voter metadata absence before marking the job complete.

Validation

Go Test Results

@coderabbitai

coderabbitaiBot commented Jul 17, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f2fba8a2-2691-48d1-b01c-0c22bdd7fce4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a background split job runner to the DistributionServer that periodically identifies split jobs in the cleanup phase, promotes target-local staged rows, and completes the promotion via the coordinator. It also adds comprehensive unit tests and wires up the runner and its gRPC connection cache in main.go. Feedback on these changes suggests suppressing context.Canceled errors in the runner loop to avoid noisy logs during shutdown, and adding a check for empty addresses returned by RaftLeaderForKey in the client factory to fail fast with a descriptive error.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment threadadapter/distribution_server.go
Comment threadmain.go Outdated

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:c065ad98fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadmain.go Outdated
Comment threadadapter/distribution_server.go Outdated
Comment threadmain.go
Comment threadmain.go Outdated
@bootjpChatGPT Codex Connector

Copy link
Copy Markdown
OwnerAuthor

@codex review

@bootjpChatGPT Codex Connector

Copy link
Copy Markdown
OwnerAuthor

@codex review

Addressed current review findings:

  • resolve promotion target by TargetGroupID instead of split-key route lookup
  • start the split runner after startup rotation and mutator gate readiness
  • keep capability closed while migration import/promote opcodes are disabled
  • bound promotion RPC attempts with a timeout and suppress shutdown context noise

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Current head 252bd64 now has all GitHub checks green and the previously reported findings were addressed in the latest commit. Requesting a fresh current-head review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit:252bd64d6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Addressed the current-head split runner findings in 938a284:

  • split job runner shutdown suppression now recognizes wrapped gRPC Canceled/DeadlineExceeded statuses, avoiding noisy shutdown tick logs.
  • split migration capability now checks local import/promote opcode readiness before probing peers, so StartSplitMigration stays fail-closed while local migration opcodes are disabled.

Caller audit:

  • StartSplitMigration -> requireMigrationCapability now fails closed on local opcode readiness before peer capability probes.
  • GetSplitMigrationCapability already uses WithSplitJobRunnerReadinessGate(splitMigrationLocalReadinessGate), so advertisement remains closed while local opcodes are disabled.
  • RunSplitJobRunner is the runtime caller for RunSplitJobRunnerOnce; only shutdown/deadline status classification changed, and non-shutdown errors still warn.

Validation:

  • go test ./adapter -run 'TestSplitJobRunnerContextDoneRecognizesWrappedGRPCStatus|TestDistributionServerRunSplitJobRunnerOnce|TestDistributionServerGetSplitMigrationCapability' -count=1 -timeout=180s
  • go test . -run 'TestSplitMigrationCapabilityGate|TestSplitMigrationLocalReadinessGate' -count=1 -timeout=180s
  • go test ./adapter . -count=1 -timeout=300s
  • git diff --check
  • git verify-commit HEAD

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Current head 938a284 now has all GitHub checks green. The two current-head inline findings have code changes, regression tests, and inline replies. Requesting a fresh current-head review.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:938a2842ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadadapter/distribution_server.go
Comment threadadapter/distribution_server.go Outdated
Comment threadadapter/distribution_server.go Outdated
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (1c7ea5b):

  • distribution/migration_promotion_complete.go
  • distribution/migration_promotion_complete_test.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Addressed current-head promotion cleanup findings in 1c7ea5b:

  • cleanup jobs are now terminalized to SplitJobPhaseDone after target promotion, including previously promoted cleanup jobs that only had TargetPromotionDone set.
  • the split job runner applies the promoted catalog snapshot to the local distribution engine before returning.
  • promotion completion commit timestamps now go through kv.NextTimestampAfterThrough, so TSO-enabled deployments use the configured allocator instead of the local HLC path.

Caller audit:

  • RunSplitJobRunnerOnce is the only production caller of nextCleanupSplitJob and promoteSplitJobTargetAndComplete; it now handles both unpromoted cleanup jobs and already-promoted cleanup jobs.
  • CompleteTargetPromotionState is called by CatalogStore.CompleteSplitJobTargetPromotion and the DistributionServer promotion path; both now terminalize completed jobs and preserve an existing PromotionCompletedTS when only terminalizing old partial state.
  • promotion retry matchers now treat current Done + TargetPromotionDone as the idempotent completion form for an expected Cleanup job.

Validation:

  • go test ./distribution -run 'TestCompleteTargetPromotionState|TestCatalogStoreCompleteSplitJobTargetPromotion' -count=1 -timeout=180s
  • go test ./adapter -run 'TestDistributionServerCompleteSplitJobTargetPromotion|TestDistributionServerRunSplitJobRunnerOnce|TestPromoteSplitJobTargetRejectsNoCursorProgress|TestSplitJobRunnerContextDoneRecognizesWrappedGRPCStatus' -count=1 -timeout=180s
  • go test ./distribution -count=1 -timeout=180s
  • go test ./adapter -run 'TestDistributionServer' -count=1 -timeout=180s
  • go test . -run 'TestSplitMigrationCapabilityGate|TestSplitMigrationLocalReadinessGate' -count=1 -timeout=180s
  • golangci-lint --config=.golangci.yaml run ./adapter ./distribution --timeout=5m
  • git diff --check
  • git verify-commit HEAD

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:1c7ea5b343

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threaddistribution/migration_promotion_complete.go
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (7c1f0aa):

  • distribution/migration_promotion_complete.go
  • distribution/migration_promotion_complete_test.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Latest head 7c1f0aa addresses the promotion completion timestamp finding. Every transition to Done now records TerminalAtMs; already promoted jobs with a missing terminal time are normalized; both catalog and coordinator retry matchers ignore the completion-owned terminal timestamp when comparing the original expected job. Catalog, coordinator, and runner tests passed; full package compile passed; golangci-lint reported 0 issues; all production completion callers were audited.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:7c1f0aa431

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadmain.go
Comment threaddistribution/migration_promotion_complete.go Outdated
Comment threadadapter/distribution_server.go Outdated
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (0c74b60):

  • distribution/migration_promotion_complete.go
  • distribution/migration_promotion_complete_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Current head 0c74b60 implements the cross-group runner phases, durable source write tracking/fences, route-faithful lock drain, resumable bracket cursor imports, cutover guards, target promotion, and migration retention handling.

Validation:

  • go test ./adapter ./kv ./store ./distribution -run relevant migration suites
  • go test ./... -run ^$
  • golangci-lint run ./... --timeout=5m --allow-parallel-runners
  • git diff --check
  • signed commit verified

Please review the latest head; earlier review commits are stale.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:0c74b6043d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadadapter/split_job_runner.go Outdated
Comment threadkv/fsm.go
Comment threadadapter/split_job_runner.go
Comment threadadapter/split_job_runner.go Outdated
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (634a75e):

  • distribution/migration_promotion_complete.go
  • distribution/migration_promotion_complete_test.go
  • distribution/split_job_catalog.go
  • distribution/split_job_catalog_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjpbootjp changed the title Run split target promotionComplete hotspot split M2 migration lifecycleJul 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (1cbee6a):

  • distribution/migration_promotion_complete.go
  • distribution/migration_promotion_complete_test.go
  • distribution/split_job_catalog.go
  • distribution/split_job_catalog_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review\n\nCurrent head 1cbee6a addresses the current-head P1 for 2PC commits prepared before tracker arm and committed after BACKFILL opens.\n\nCaller audit:\n- handleTxnRequest routes COMMIT only through handleCommitRequest.\n- raw, DEL_PREFIX, PREPARE, and one-phase paths already record their admitted write footprint.\n- COMMIT now records after commit mutation construction succeeds and before store apply.\n\nValidation:\n- go test ./kv -run 'TestFSMMigrationWriteTracker|TestFSMRejectsPrepareOnWriteFencedRouteButAllowsAbort|TestFSMAbortCleanup' -count=1\n- go test ./... -run '^$' -count=1\n- golangci-lint run ./... --timeout=5m --allow-parallel-runners\n- git diff --check\n- git verify-commit HEAD

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (8264ea3):

  • distribution/migration_promotion_complete.go
  • distribution/migration_promotion_complete_test.go
  • distribution/split_job_catalog.go
  • distribution/split_job_catalog_test.go
  • kv/fsm.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review\n\nCurrent head 8264ea3 also addresses the two remaining current, non-outdated P2 threads:\n- migration capability and mutating migration RPCs now remain unavailable until the startup rotation/registration gate opens;\n- legacy DONE promotion retries persist TerminalAtMs normalization through the coordinator CAS.\n\nValidation:\n- go test . -run 'TestStartupPublicKVGate' -count=1\n- go test ./adapter -run 'TestDistributionServerCompleteSplitJobTargetPromotion' -count=1\n- go test ./distribution -run 'TestCompleteTargetPromotionState|TestCatalogStoreCompleteSplitJobTargetPromotion' -count=1\n- go test ./... -run '^$' -count=1\n- golangci-lint run ./... --timeout=5m --allow-parallel-runners\n- git verify-commit HEAD

@bootjp

Copy link
Copy Markdown
OwnerAuthor

@codex review

Current head 8264ea3 now has all GitHub checks green, no unresolved review threads, and no current-head root inline findings. The current-head P1/P2 findings have code changes, regression tests, inline replies, and resolved threads. Requesting final current-head review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit:8264ea3835

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp
bootjp merged commit 28449f9 into design/hotspot-split-m2-promotion-completeJul 19, 2026
12 checks passed
@bootjp
bootjp deleted the design/hotspot-split-m2-runner branch July 19, 2026 06:10
bootjp added a commit that referenced this pull request Jul 23, 2026
## Summary
- promote the hotspot split M2 migration design from partial to
implemented
- record the completed production runner, current-voter barriers, route
cutover, cleanup, DONE history, and deterministic Jepsen workload
- update the parent hotspot design and scaling roadmaps to the
implemented path and current status
## Evidence
- PR #1096 merged the final M2 runner/lifecycle slice into this stacked
base
- latest #1096 head review reported no major issues and all GitHub
checks passed
- targeted Go migration, voter barrier, cleanup, promotion, and route
tests passed
- `lein test elastickv.split-workload-test` passed (2 tests, 5
assertions)
- old partial filename references are absent
## Remaining scope
- M3 owns automatic hotspot detection, target selection, and split
scheduling
- M4 owns the broader route-shuffle nemesis matrix and production-scale
fault campaigns
- reverse migration after CUTOVER and concurrent migration jobs remain
future extensions
bootjp added a commit that referenced this pull request Aug 21, 2026
Review findings on the current HEAD:
- The hotspot split M2 and S3 blob offload rows cited the pre-rename
*_proposed_* filenames. Both docs were promoted to *_partial_* and the
pointers were left behind.
- The streaming transport row and the §5 gap table still called the
multi-group soak evidence outstanding. It landed with the design's §8:
cmd/elastickv-raft-stream-soak, the Jepsen soak script, and a
fail-closed verifier over docs/evidence/raft_streaming_multigroup_soak.json.
The reviewer also flagged that the cited PR states were a 2026-07-19
snapshot and needed a spot-check. Re-audited every cited PR against
GitHub: #1096, #1097, #1108, #1082, #1057, #1063, and #1059 have merged
since. Rows updated and the audit date moved to 2026-08-22.
Claude-Session: https://claude.ai/code/session_013rNHooj7NF3giihWVba8QE
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@bootjp