Skip to content

Fix duplicate scheduling in procedure execution - #17902

Merged
jt2594838 merged 6 commits into
masterfrom
fix-procedure-duplicate-scheduling
Jun 16, 2026
Merged

Fix duplicate scheduling in procedure execution#17902
jt2594838 merged 6 commits into
masterfrom
fix-procedure-duplicate-scheduling

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Prevent duplicate procedure execution

  • Add a per-procedure execution guard so a duplicated scheduler entry cannot run the same Procedure instance concurrently.
  • Release the guard, active executor count, and worker state in the worker finally path.
  • Release locks when a scheduled procedure is no longer RUNNABLE.

Avoid duplicate lock wakeups

  • De-duplicate LockQueue waiters by procedure id before adding a waiting procedure.
  • Keep PipeTaskCoordinatorLock waiting until a permit is actually acquired, even if the thread is already interrupted.

Tests

  • Add coverage for duplicate procedure scheduling, duplicate LockQueue wakeups, interrupted PipeTaskCoordinatorLock acquisition, and completed PipeHandleLeaderChangeProcedure equality.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.confignode.procedure.Procedure
  • org.apache.iotdb.confignode.procedure.ProcedureExecutor
  • org.apache.iotdb.confignode.procedure.scheduler.LockQueue
  • org.apache.iotdb.confignode.manager.pipe.coordinator.task.PipeTaskCoordinatorLock
Local verification
  • PASS: mvn -pl iotdb-core/confignode spotless:apply
  • FAIL: mvn -pl iotdb-core/confignode -Dtest=TestLockRegime,TestProcedureExecutor,PipeHandleLeaderChangeProcedureTest,PipeTaskCoordinatorLockTest test failed during compile before test execution because local generated/snapshot dependencies were missing classes such as TGetCommitProgressReq, TPullCommitProgressReq, CommitProgressKeeper, and RegionProgress.

@codecov

codecovBot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.50360% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.17%. Comparing base (8fad784) to head (bca866a).
⚠️ Report is 7 commits behind head on master.

Files with missing linesPatch %Lines
.../iotdb/confignode/procedure/ProcedureExecutor.java68.51%17 Missing ⚠️
...ode/procedure/impl/node/AbstractNodeProcedure.java0.00%6 Missing ⚠️
...dure/impl/pipe/AbstractOperatePipeProcedureV2.java0.00%6 Missing ⚠️
...db/confignode/procedure/TimeoutExecutorThread.java75.00%4 Missing ⚠️
...otdb/confignode/procedure/scheduler/LockQueue.java66.66%2 Missing ⚠️
...otdb/db/schemaengine/table/DataNodeTableCache.java50.00%2 Missing ⚠️
...g/apache/iotdb/confignode/procedure/Procedure.java80.00%1 Missing ⚠️
.../impl/schema/AlterTimeSeriesDataTypeProcedure.java0.00%1 Missing ⚠️
...ntime/SubscriptionHandleLeaderChangeProcedure.java0.00%1 Missing ⚠️
.../procedure/scheduler/SimpleProcedureScheduler.java94.44%1 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## master #17902 +/- ##
============================================
+ Coverage 41.15% 41.17% +0.02% 
Complexity 318 318 ============================================
Files 5258 5258 Lines 365613 365743 +130 Branches 47282 47300 +18 ============================================
+ Hits 150460 150596 +136 + Misses 215153 215147 -6 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +110 to +115
boolean duplicated = blockingProcedure.awaitExecution(3, TimeUnit.SECONDS);

blockingProcedure.releaseExecutions(duplicated ? 2 : 1);
ProcedureTestUtil.waitForProcedure(procExecutor, procId);

Assert.assertFalse(duplicated);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 112 is weird together with line 115.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate FailedQuality Gate failed

Failed conditions
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@jt2594838
jt2594838 merged commit 0a45a3b into masterJun 16, 2026
42 of 43 checks passed
@jt2594838
jt2594838 deleted the fix-procedure-duplicate-scheduling branch June 16, 2026 10:19
jt2594838 pushed a commit that referenced this pull request Jun 18, 2026
* Fix duplicate scheduling in procedure execution
* Fix delayed procedure deduplication and semaphore release
* Fix SQL parser error handler traversal
* Fix pipe procedure lock release race
* Fix procedure lock wait scheduling
(cherry picked from commit c25849a)
(cherry picked from commit 0a45a3b)
MileaRobertStefan pushed a commit to MileaRobertStefan/iotdb that referenced this pull request Jun 26, 2026
* Fix duplicate scheduling in procedure execution
* Fix delayed procedure deduplication and semaphore release
* Fix SQL parser error handler traversal
* Fix pipe procedure lock release race
* Fix procedure lock wait scheduling
(cherry picked from commit c25849a)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Caideyipi@jt2594838