Uh oh!
There was an error while loading. Please reload this page.
branch-4.0: pick #61696 #61621 #62272 - #62706
Merged
yiguolei merged 4 commits intoApr 27, 2026
Merged
Conversation
hello-stephen
commented
Apr 22, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Yukang-Lianforce-pushed
the
lianyukang/pick-4.0-61696-61621-62272
branch
from
April 22, 2026 09:38
207ecb3 to
453c0d6CompareYukang-Lian
commented
Apr 22, 2026
CollaboratorAuthor
run buildall |
hello-stephen
commented
Apr 22, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
Yukang-Lianforce-pushed
the
lianyukang/pick-4.0-61696-61621-62272
branch
from
April 22, 2026 16:27
a014c1c to
9e1684cCompareYukang-Lian
commented
Apr 22, 2026
CollaboratorAuthor
run buildall |
… HTTP API (apache#61696) Introduce `CompactionTaskTracker` to provide **full lifecycle observability** for base/cumulative/full compaction tasks. This PR adds two query channels: 1. **`information_schema.be_compaction_tasks` system table** (38 columns) — query via SQL across all BEs 2. **`GET /api/compaction/profile` HTTP API** — query via JSON on a single BE, with filtering support Currently, compaction execution metrics (input/output data sizes, row counts, merge latency, etc.) are lost once the compaction object is destructed. Operators have no way to inspect historical compaction performance — only the current running status is available. Users also cannot see pending or running compaction task details through SQL. This PR solves both problems with a unified `CompactionTaskTracker` mechanism that tracks compaction tasks across their full lifecycle: **PENDING → RUNNING → FINISHED/FAILED**. ``` ┌── Push ──┐ Compaction entry points ──→ CompactionTaskTracker (in-memory) ←── Push ── Compaction base class (register_task) │ _active_tasks (PENDING/RUNNING) (complete/fail) │ _completed_tasks (FINISHED/FAILED) │ ┌──── Pull ────┴──── Pull ────┐ ↓ ↓ SchemaCompactionTasksScanner CompactionProfileAction get_all_tasks() snapshot get_completed_tasks() filtered ↓ ↓ FE fan-out to all BEs → SQL JSON HTTP Response ``` - **Push-based collection**: Compaction entry points and execution layer push task info at lifecycle boundaries - **Pull-based querying**: System table scanner and HTTP action pull snapshots with read locks, never blocking compaction - **Pure in-memory storage**: Two containers, no persistence. Data clears on BE restart. --- | Column | Type | Description | |--------|------|-------------| | BACKEND_ID | BIGINT | BE node ID | | COMPACTION_ID | BIGINT | Unique task ID | | TABLE_ID | BIGINT | Table ID | | PARTITION_ID | BIGINT | Partition ID | | TABLET_ID | BIGINT | Tablet ID | | COMPACTION_TYPE | VARCHAR | `base` / `cumulative` / `full` | | STATUS | VARCHAR | `PENDING` / `RUNNING` / `FINISHED` / `FAILED` | | TRIGGER_METHOD | VARCHAR | `AUTO` / `MANUAL` / `LOAD_TRIGGERED` | | COMPACTION_SCORE | BIGINT | Tablet compaction score at registration time | | SCHEDULED_TIME | DATETIME | Task registration time | | START_TIME | DATETIME | Execution start time (NULL when PENDING) | | END_TIME | DATETIME | Execution end time (NULL when not completed) | | ELAPSED_TIME_MS | BIGINT | RUNNING: real-time `now - start_time`; FINISHED/FAILED: `end - start` | | INPUT_ROWSETS_COUNT | BIGINT | Number of input rowsets | | INPUT_ROW_NUM | BIGINT | Number of input rows | | INPUT_DATA_SIZE | BIGINT | Input data size (bytes) | | INPUT_INDEX_SIZE | BIGINT | Input index size (bytes) | | INPUT_TOTAL_SIZE | BIGINT | Input total size (data + index) | | INPUT_SEGMENTS_NUM | BIGINT | Number of input segments | | INPUT_VERSION_RANGE | VARCHAR | Input version range, e.g., `[0-5]` | | MERGED_ROWS | BIGINT | Merged row count | | FILTERED_ROWS | BIGINT | Filtered row count | | OUTPUT_ROWS | BIGINT | Merger output rows (0 for ordered compaction) | | OUTPUT_ROW_NUM | BIGINT | Output rowset row count | | OUTPUT_DATA_SIZE | BIGINT | Output data size (bytes) | | OUTPUT_INDEX_SIZE | BIGINT | Output index size (bytes) | | OUTPUT_TOTAL_SIZE | BIGINT | Output total size (data + index) | | OUTPUT_SEGMENTS_NUM | BIGINT | Number of output segments | | OUTPUT_VERSION | VARCHAR | Output version range | | MERGE_LATENCY_MS | BIGINT | Merge phase latency in ms (0 for ordered compaction) | | BYTES_READ_FROM_LOCAL | BIGINT | Bytes read from local storage | | BYTES_READ_FROM_REMOTE | BIGINT | Bytes read from remote storage | | PEAK_MEMORY_BYTES | BIGINT | Peak memory usage (bytes) | | IS_VERTICAL | BOOLEAN | Whether vertical compaction was used | | PERMITS | BIGINT | Compaction permits consumed | | VERTICAL_TOTAL_GROUPS | BIGINT | Vertical compaction total column groups (0 if horizontal) | | VERTICAL_COMPLETED_GROUPS | BIGINT | Completed column groups (real-time during RUNNING) | | STATUS_MSG | VARCHAR | Error message (empty on success) | **1. View running compaction tasks** ```sql SELECT BACKEND_ID, TABLET_ID, COMPACTION_TYPE, ELAPSED_TIME_MS, INPUT_DATA_SIZE FROM information_schema.be_compaction_tasks WHERE STATUS = 'RUNNING' ORDER BY ELAPSED_TIME_MS DESC; ``` ``` +------------+-----------+-----------------+-----------------+-----------------+ | BACKEND_ID | TABLET_ID | COMPACTION_TYPE | ELAPSED_TIME_MS | INPUT_DATA_SIZE | +------------+-----------+-----------------+-----------------+-----------------+ | 10001 | 123456 | base | 35210 | 524288000 | | 10002 | 789012 | cumulative | 1250 | 10485760 | +------------+-----------+-----------------+-----------------+-----------------+ ``` **2. Track vertical compaction progress** ```sql SELECT TABLET_ID, VERTICAL_COMPLETED_GROUPS AS done, VERTICAL_TOTAL_GROUPS AS total, CONCAT(ROUND(VERTICAL_COMPLETED_GROUPS * 100.0 / NULLIF(VERTICAL_TOTAL_GROUPS, 0), 1), '%') AS progress FROM information_schema.be_compaction_tasks WHERE STATUS = 'RUNNING' AND IS_VERTICAL = true; ``` ``` +-----------+------+-------+----------+ | TABLET_ID | done | total | progress | +-----------+------+-------+----------+ | 123456 | 2 | 6 | 33.3% | | 234567 | 5 | 8 | 62.5% | +-----------+------+-------+----------+ ``` **3. Find the slowest finished compactions** ```sql SELECT TABLET_ID, COMPACTION_TYPE, ELAPSED_TIME_MS, INPUT_DATA_SIZE, OUTPUT_DATA_SIZE FROM information_schema.be_compaction_tasks WHERE STATUS = 'FINISHED' ORDER BY ELAPSED_TIME_MS DESC LIMIT 3; ``` ``` +-----------+-----------------+-----------------+-----------------+------------------+ | TABLET_ID | COMPACTION_TYPE | ELAPSED_TIME_MS | INPUT_DATA_SIZE | OUTPUT_DATA_SIZE | +-----------+-----------------+-----------------+-----------------+------------------+ | 123456 | base | 42000 | 524288000 | 210000000 | | 345678 | cumulative | 8500 | 30000000 | 15000000 | | 456789 | full | 5200 | 20000000 | 12000000 | +-----------+-----------------+-----------------+-----------------+------------------+ ``` **4. Compaction count by trigger method** ```sql SELECT TRIGGER_METHOD, COUNT(*) AS cnt FROM information_schema.be_compaction_tasks WHERE STATUS = 'FINISHED' GROUP BY TRIGGER_METHOD; ``` ``` +----------------+-----+ | TRIGGER_METHOD | cnt | +----------------+-----+ | AUTO | 847 | | MANUAL | 12 | | LOAD_TRIGGERED | 53 | +----------------+-----+ ``` --- Queries completed compaction profiles on a **single BE** with optional filtering. All parameters can be combined (AND logic). | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `tablet_id` | int64 | No | Filter by tablet ID | | `top_n` | int64 | No | Return most recent N records | | `compact_type` | string | No | Filter by type: `base` / `cumulative` / `full` | | `success` | string | No | Filter by result: `true` / `false` | ```json { "status": "Success", "compaction_profiles": [ { "compaction_id": 487, "compaction_type": "cumulative", "tablet_id": 12345, "table_id": 67890, "partition_id": 11111, "trigger_method": "AUTO", "compaction_score": 10, "scheduled_time": "2025-07-15 14:02:30", "start_time": "2025-07-15 14:02:31", "end_time": "2025-07-15 14:02:31", "cost_time_ms": 236, "success": true, "input_rowsets_count": 5, "input_row_num": 52000, "input_data_size": 10706329, "input_index_size": 204800, "input_total_size": 10911129, "input_segments_num": 5, "input_version_range": "[12-16]", "merged_rows": 1200, "filtered_rows": 50, "output_rows": 50750, "output_row_num": 50750, "output_data_size": 5033164, "output_index_size": 102400, "output_total_size": 5135564, "output_segments_num": 1, "output_version": "[12-16]", "merge_latency_ms": 180, "bytes_read_from_local": 10706329, "bytes_read_from_remote": 0, "peak_memory_bytes": 33554432, "is_vertical": true, "permits": 10706329, "vertical_total_groups": 4, "vertical_completed_groups": 4 } ] } ``` Failed compactions additionally include `"status_msg": "error message"`. ```bash curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?top_n=10" curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?tablet_id=12345&top_n=5" curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?compact_type=base&success=false" curl "http://BE_HOST:BE_HTTP_PORT/api/compaction/profile?tablet_id=12345&top_n=3&compact_type=cumulative&success=false" ``` --- | Config | Type | Default | Description | |--------|------|---------|-------------| | `enable_compaction_task_tracker` | mutable bool | `true` | Master switch. When disabled, all push operations are no-op, queries return empty. | | `compaction_task_tracker_max_records` | mutable int32 | `10000` | Max completed task records kept in memory (~500 bytes/record, default ~5MB). | --- - **Tracked**: base / cumulative / full compaction (local + cloud mode) - **Not tracked**: single_replica, cold_data, index_change compaction (low frequency, limited diagnostic value) - **Trigger methods**: AUTO (background scheduling), MANUAL (HTTP API), LOAD_TRIGGERED (load-time cumulative) --- - [x] 15 unit tests (`CompactionTaskTrackerTest.*`) covering full lifecycle, failure paths, TRY_LOCK_FAILED cleanup, vertical progress, concurrent safety, config switch, filters - [x] Regression test: `test_be_compaction_tasks` (system table) - [x] Regression test: `test_compaction_profile_action` (HTTP API)
…dit log disabled (apache#61621) - When `enable_prepared_stmt_audit_log = false` (default), `auditAfterExec()` is entirely skipped for prepared statements. Since QPS metric counting lives inside `AuditLogHelper.logAuditLog()`, these metrics are also lost for prepared statement SELECT queries. - Add `AuditLogHelper.updateMetrics()` to count metrics independently of audit log writing, and call it in the `else` branch when audit log is disabled. - This decouples the metric counting from audit log writing without modifying the existing `logAuditLogImpl()` logic. - **AuditLogHelper.java**: Add `updateMetrics()` + `updateMetricsImpl()` methods for standalone metric counting - **MysqlConnectProcessor.java**: Add `else` branch in `handleExecute()` to call `updateMetrics()` when audit log is disabled. - **AuditLogHelperTest.java**: Unit tests covering OK query, ERR query, non-query (INSERT), and debug mode short-circuit scenarios. - [x] Unit tests pass (`AuditLogHelperTest` - 4 cases)
Yukang-Lianforce-pushed
the
lianyukang/pick-4.0-61696-61621-62272
branch
from
April 22, 2026 17:07
9e1684c to
e1d6f9aCompareYukang-Lian
commented
Apr 22, 2026
CollaboratorAuthor
run buildall |
hello-stephen
commented
Apr 22, 2026
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
…compaction failures (apache#62272) ## Summary - Add dedicated `SC_COMPACTION_CONFLICT(101)` TStatusCode for the cross-V1 compaction detection in cloud schema change - Include this error code in FE's schema change retry whitelist (`AlterJobV2.getRetryTimes()`) - Previously, this transient condition returned `INTERNAL_ERROR` which FE does not retry, causing the schema change job to be permanently cancelled ## Background The cloud SC compaction optimization allows new tablets to do compaction during schema change queue wait. A safety check detects when a compaction rowset crosses the alter_version (V1) boundary. This is a transient condition - on retry, V1 will be higher and the crossing rowset falls within `[2, V1']`. The bug was that FE treated `INTERNAL_ERROR` as a permanent failure (no retry), so this transient condition caused the entire schema change to be cancelled. ## Changes | File | Change | |------|--------| | `gensrc/thrift/Status.thrift` | Add `SC_COMPACTION_CONFLICT = 101` | | `be/src/common/status.h` | Register `TStatusError(SC_COMPACTION_CONFLICT, false)` | | `be/src/cloud/cloud_schema_change_job.cpp` | Cross-V1 check: `INTERNAL_ERROR` → `SC_COMPACTION_CONFLICT` | | `fe/.../alter/AlterJobV2.java` | Add `SC_COMPACTION_CONFLICT` to retry whitelist | | `fe/.../alter/AlterJobV2RetryTest.java` | Unit tests for `getRetryTimes()` (7 cases) | | `regression-test/.../test_sc_compaction_cross_v1_retry.groovy` | E2E test: cross-V1 failure → retry → success | ## Test plan - [x] FE unit test: `AlterJobV2RetryTest` - verifies retry whitelist for all error codes, null handling, and config toggle - [x] Regression test: `test_sc_compaction_cross_v1_retry` - uses debug points to trigger cross-V1, verifies SC stays RUNNING under override, then succeeds after override removal. Asserts data integrity and schema correctness.
Yukang-Lianforce-pushed
the
lianyukang/pick-4.0-61696-61621-62272
branch
from
April 23, 2026 02:27
e1d6f9a to
0367bffCompareYukang-Lian
commented
Apr 23, 2026
CollaboratorAuthor
run buildall |
hello-stephen
commented
Apr 23, 2026
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Apr 23, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Yukang-Lian
commented
Apr 23, 2026
CollaboratorAuthor
run feut |
Yukang-Lian
commented
Apr 23, 2026
CollaboratorAuthor
run p0 |
hello-stephen
commented
Apr 23, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Apr 23, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Apr 27, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick the following PRs to
branch-4.0:#61696 had conflicts due to the BE directory layout difference between master (
be/src/storage/...) and branch-4.0 (be/src/olap/...). New files were relocated to the branch-4.0 paths; include paths and FE schema/thrift entries were merged accordingly.