Uh oh!
There was an error while loading. Please reload this page.
[improve](metrics) Export file cache remote index byte metrics - #65398
Conversation
hello-stephen
commented
Jul 8, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hoshinojyunn
commented
Jul 8, 2026
run buildall |
hoshinojyunn
commented
Jul 8, 2026
run buildall |
hello-stephen
commented
Jul 8, 2026
TPC-H: Total hot run time: 29891 ms |
hello-stephen
commented
Jul 8, 2026
TPC-DS: Total hot run time: 181241 ms |
hello-stephen
commented
Jul 8, 2026
ClickBench: Total hot run time: 25.24 s |
Issue Number: None Related PR: None Problem Summary: File cache profile statistics already track `inverted_index_bytes_read_from_remote` and `segment_footer_index_bytes_read_from_remote`, but BE Doris metrics only exported aggregate cache, remote, and peer read bytes. That made these index-specific remote read costs visible in query profile only, not in the BE metrics endpoint. This change registers two new Doris metrics, extends the existing `FileCacheMetrics` aggregation path to accumulate them, and exports them through the normal metrics hook. The update also extends the existing file cache profile unit test helpers so segment footer index statistics are covered and adds a test for the new metric export path. Expose `inverted_index_bytes_read_from_remote` and `segment_footer_index_bytes_read_from_remote` in BE Doris metrics. - Test: Manual compile verification - Started `./run-be-ut.sh --run --filter=FileCacheProfileReporterTest.*`, but it triggered a large `doris_be_test` rebuild and was interrupted after the changed objects compiled successfully - Recompiled changed objects with `ninja -C be/ut_build_ASAN src/common/CMakeFiles/Common.dir/metrics/doris_metrics.cpp.o src/io/CMakeFiles/IO.dir/cache/block_file_cache_profile.cpp.o test/CMakeFiles/doris_be_test.dir/io/cache/block_file_cache_profile_reporter_test.cpp.o` - Behavior changed: Yes (adds two new BE metrics for file cache remote index bytes) - Does this need documentation: No
hoshinojyunn
commented
Jul 9, 2026
run buildall |
hello-stephen
commented
Jul 9, 2026
TPC-H: Total hot run time: 29644 ms |
hello-stephen
commented
Jul 9, 2026
TPC-DS: Total hot run time: 180696 ms |
hello-stephen
commented
Jul 9, 2026
ClickBench: Total hot run time: 25.15 s |
hello-stephen
commented
Jul 9, 2026
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Uh oh!
There was an error while loading. Please reload this page.
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
yiguolei
commented
Jul 10, 2026
/review |
There was a problem hiding this comment.
Automated review summary for PR #65398.
I reviewed the changed BE metrics, file-cache metric aggregation/export path, BE unit test, and cloud docker regression test. I found no additional blocking issue to submit as a new inline comment.
Duplicate suppression: the existing thread on be/src/io/cache/block_file_cache_profile.cpp:74 already raises the set_value vs add/increment concern, so I did not submit a duplicate. I also checked the surrounding exporter path: FileCacheMetrics::update() accumulates deltas into the internal cumulative statistics, and the existing aggregate file-cache metric hook exports those cumulative values with set_value.
Critical checkpoint conclusions:
- Goal and coverage: the PR exposes two file-cache remote index byte metrics and adds BE/regression coverage for the export path.
- Scope: the change is focused on metric registration, file-cache stat accumulation, and tests.
- Concurrency/lifecycle: no new lock-order or lifecycle issue found; the added fields follow the existing atomic
FileCacheMetricspattern. - Config/compatibility/persistence/transactions: no new config, protocol, storage format, edit-log, or transaction behavior is introduced.
- Parallel paths: both inverted-index and segment-footer read types feed the same file-cache statistics update path; no optimizer/rewrite impact found.
- Performance/observability: the hot path adds only the same style of atomic accumulation used by existing file-cache counters, and the new metrics improve observability.
- Testing standards: the regression test drops before use, orders query output where it inspects rows, and checks both exported metrics.
Validation: static review only in this runner. thirdparty/installed, thirdparty/installed/bin/protoc, and be/ut_build_ASAN are absent, so I did not rerun the claimed BE unit or docker regression tests locally.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions: optimizer-rewrite and tests-session-config both reported NO_NEW_VALUABLE_FINDINGS in the final convergence round for the current ledger and empty proposed final comment set. No subagent candidates were accepted, and my only local candidate was dismissed with evidence because BE-UT builds doris_be_test with -fno-access-control.
Uh oh!
There was an error while loading. Please reload this page.
Issue Number: None Related PR: None Problem Summary: File cache statistics already track the remote-read bytes for inverted index and segment footer index data, but BE Doris metrics only export the aggregate cache/local/remote/peer byte counters. As a result, these two index-specific remote read costs are visible in query profile internals but not exposed from the BE `/metrics` endpoint, which makes them harder to monitor and alert on in cloud/file-cache scenarios. This PR exposes the following two BE metrics: - `doris_be_inverted_index_bytes_read_from_remote` - `doris_be_segment_footer_index_bytes_read_from_remote` The change keeps the existing file cache metrics aggregation path and extends it in three places: 1. Register the two counters in `DorisMetrics` 2. Extend `FileCacheMetrics::AtomicStatistics` and the aggregation/update path to accumulate them 3. Export them from the existing metrics hook and cover the behavior with a BE unit test ### Release note Add two new BE metrics for file cache remote index read bytes: - `doris_be_inverted_index_bytes_read_from_remote` - `doris_be_segment_footer_index_bytes_read_from_remote` ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - `./run-regression-test.sh --run -d cloud_p0 -s test_file_cache_remote_index_byte_metrics -runMode=cloud` - [x] Unit Test - `./run-be-ut.sh --run --filter=DorisMetricsTest.*:FileCacheProfileReporterTest.*` - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
Issue Number: None
Related PR: None
Problem Summary:
File cache statistics already track the remote-read bytes for inverted index and segment footer
index data, but BE Doris metrics only export the aggregate cache/local/remote/peer byte counters.
As a result, these two index-specific remote read costs are visible in query profile internals but
not exposed from the BE
/metricsendpoint, which makes them harder to monitor and alert on incloud/file-cache scenarios.
This PR exposes the following two BE metrics:
doris_be_inverted_index_bytes_read_from_remotedoris_be_segment_footer_index_bytes_read_from_remoteThe change keeps the existing file cache metrics aggregation path and extends it in three places:
DorisMetricsFileCacheMetrics::AtomicStatisticsand the aggregation/update path to accumulate themRelease note
Add two new BE metrics for file cache remote index read bytes:
doris_be_inverted_index_bytes_read_from_remotedoris_be_segment_footer_index_bytes_read_from_remoteCheck List (For Author)
Test
./run-regression-test.sh --run -d cloud_p0 -s test_file_cache_remote_index_byte_metrics -runMode=cloud./run-be-ut.sh --run --filter=DorisMetricsTest.*:FileCacheProfileReporterTest.*Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)