Skip to content

branch-4.1: [fix](regression) handle E-2010 compaction wait#65290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yiguolei merged 2 commits into from
Jul 8, 2026
Merged

branch-4.1: [fix](regression) handle E-2010 compaction wait #65290

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
14b4e18
[fix](regression) handle cumulative delete-version compaction wait (#…
shuke987 Jul 1, 2026
f0971dc
[fix](regression) handle E-2010 compaction success wait (#65209)
shuke987 Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion regression-test/plugins/plugin_compaction.groovy
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -120,6 +120,16 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac

// 3. wait all compaction finished
def running = triggered_tablets.size() > 0
def toLongOrNull = { value ->
if (value == null) {
return null
}
try {
return value.toString().trim().toLong()
} catch (Throwable ignored) {
return null
}
}
Awaitility.await().atMost(timeout_seconds, TimeUnit.SECONDS).pollInterval(1, TimeUnit.SECONDS).until(() -> {
for (tablet in triggered_tablets) {
def be_host = backendId_to_backendIP["${tablet.BackendId}"]
Expand All@@ -138,9 +148,31 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
def tabletStatus = parseJson(stdout.trim())
def oldStatus = be_tablet_compaction_status.get("${be_host}-${tablet.TabletId}")
// last compaction success/failure time isn't updated, indicates compaction is not started(so we treat it as running and wait)
def handedOffToBaseCompactionAfterDeleteVersion = false
def completedByBaseCompactionAfterDeleteVersion = false
if (compaction_type == "cumulative") {
def oldCumulativePoint = toLongOrNull(oldStatus["cumulative point"])
def newCumulativePoint = toLongOrNull(tabletStatus["cumulative point"])
def lastCumulativeStatus = "${tabletStatus["last cumulative status"]}".toLowerCase()
def baseSuccessTimeChanged = oldStatus["last base success time"] != tabletStatus["last base success time"]
def cumulativeSuccessTimeChanged =
oldStatus["last cumulative success time"] != tabletStatus["last cumulative success time"]
// E-2010 advances the cumulative point and lets base compaction handle delete-version rowsets.
// In some timing windows, base success is already visible in the cached old status while
// cumulative success advances later, so accept either success signal but not failure time alone.
handedOffToBaseCompactionAfterDeleteVersion = lastCumulativeStatus.contains("e-2010") &&
oldCumulativePoint != null && newCumulativePoint != null &&
newCumulativePoint > oldCumulativePoint
completedByBaseCompactionAfterDeleteVersion =
handedOffToBaseCompactionAfterDeleteVersion &&
(baseSuccessTimeChanged || cumulativeSuccessTimeChanged)

@github-actionsgithub-actionsBotJul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This completion check can fire too early in the E-2010 handoff path. In Cloud, the handoff is committed as an EMPTY_CUMULATIVE job; meta-service explicitly treats that as a metadata-only update to the cumulative point/count, but it also sets last_cumu_compaction_time_ms, which BE exposes as last cumulative success time. With this || cumulativeSuccessTimeChanged, the helper can return right after the empty cumulative handoff, before the base compaction that actually merges the delete-version rowsets has run. The affected tests inspect rowsets immediately after this helper returns, so this reintroduces the flaky early-exit window. Please keep this fast path tied to a base-compaction completion signal, or add another rowset/base-completion check before accepting cumulative success time here.

}
def success_time_unchanged = (oldStatus["last ${compaction_type} success time"] == tabletStatus["last ${compaction_type} success time"])
def failure_time_unchanged = (oldStatus["last ${compaction_type} failure time"] == tabletStatus["last ${compaction_type} failure time"])
running = running || (success_time_unchanged && failure_time_unchanged)
def currentCompactionTimestampChanged = !success_time_unchanged || !failure_time_unchanged
def compactionFinished = completedByBaseCompactionAfterDeleteVersion ||
(!handedOffToBaseCompactionAfterDeleteVersion && currentCompactionTimestampChanged)
running = running || !compactionFinished
if (running) {
logger.info("compaction is still running, be host: ${be_host}, tablet id: ${tablet.TabletId}, run status: ${compactionStatus.run_status}, old status: ${oldStatus}, new status: ${tabletStatus}")
return false
Expand Down
Loading