Skip to content

[Enhancement](Recycler) Add bvar metrics for operation logs - #60520

Merged
w41ter merged 3 commits into
apache:masterfrom
Yukang-Lian:feature/recycler-stat
Feb 9, 2026
Merged

[Enhancement](Recycler) Add bvar metrics for operation logs#60520
w41ter merged 3 commits into
apache:masterfrom
Yukang-Lian:feature/recycler-stat

Conversation

@Yukang-Lian

@Yukang-LianYukang-Lian commented Feb 5, 2026•
edited by gavinchou
Loading

Copy link
Copy Markdown
Collaborator
  • Add dedicated bvar metrics for the recycle_operation_logs path, which previously had no observability through the monitoring system
  • Introduce OplogRecycleStats to track per-round recycling statistics including skipped-by-snapshot count, earliest recyclable version, failure count, and generated recycle key counts
    (partition/index/rowset/txn)
  • Add scan_and_statistics_operation_logs() to collect pre-recycling statistics via RecyclerMetricsContext, consistent with existing scan_and_statistics_* functions
  • Use a PendingCounts pattern in OperationLogRecycler to avoid inflating committed metrics on transaction failure — local counts are only merged into OplogRecycleStats atomics after successful
    commit()
  • All new statistics are gated behind the enable_recycler_stats_metrics config flag

@hello-stephen

hello-stephen commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian

Yukang-Lian commented Feb 5, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

gavinchou
gavinchou reviewed Feb 5, 2026
View reviewed changes
Comment threadcloud/src/recycler/recycler_operation_log.cpp Outdated
gavinchou
gavinchou reviewed Feb 5, 2026
View reviewed changes
Comment threadcloud/src/recycler/recycler_operation_log.cpp Outdated
@doris-robot

doris-robot commented Feb 5, 2026

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 86.09% (99/115) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.35% (1794/2261)
Line Coverage64.91% (32019/49332)
Region Coverage65.54% (15950/24337)
Branch Coverage56.16% (8496/15128)

w41ter
w41ter reviewed Feb 5, 2026
View reviewed changes
Comment threadcloud/src/recycler/recycler_operation_log.cpp Outdated
Comment threadcloud/src/recycler/recycler_operation_log.cpp Outdated
@Yukang-Lian

Yukang-Lian commented Feb 5, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

w41ter
w41ter previously approved these changes Feb 5, 2026
View reviewed changes
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 5, 2026
@github-actions

github-actionsBot commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@Yukang-Lian

Yukang-Lian commented Feb 5, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run cloudut

1 similar comment
@Yukang-Lian

Yukang-Lian commented Feb 6, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run cloudut

@Yukang-Lian

Yukang-Lian commented Feb 6, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Feb 6, 2026
@Yukang-Lian

Yukang-Lian commented Feb 6, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run cloudut

@hello-stephen

hello-stephen commented Feb 6, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 89.32% (92/103) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.35% (1795/2262)
Line Coverage64.86% (31957/49269)
Region Coverage65.52% (15937/24323)
Branch Coverage56.05% (8472/15114)

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2026
@github-actions

github-actionsBot commented Feb 7, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@w41ter
w41ter merged commit 6bf54a8 into apache:masterFeb 9, 2026
34 of 35 checks passed
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 30, 2026
@Yukang-Lian
[Enhancement](Recycler) Add bvar metrics for operation logs (apache#6…
e71d109
…0520)
- Add dedicated bvar metrics for the recycle_operation_logs path, which
previously had no observability through the monitoring system
- Introduce OplogRecycleStats to track per-round recycling statistics
including skipped-by-snapshot count, earliest recyclable version,
failure count, and generated recycle key counts
(partition/index/rowset/txn)
- Add scan_and_statistics_operation_logs() to collect pre-recycling
statistics via RecyclerMetricsContext, consistent with existing
scan_and_statistics_* functions
- Use a PendingCounts pattern in OperationLogRecycler to avoid inflating
committed metrics on transaction failure — local counts are only merged
into OplogRecycleStats atomics after successful
commit()
- All new statistics are gated behind the enable_recycler_stats_metrics
config flag
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 30, 2026
@Yukang-Lian
[Enhancement](Recycler) Add bvar metrics for operation logs (apache#6…
cd07ef2
…0520)
- Add dedicated bvar metrics for the recycle_operation_logs path, which
previously had no observability through the monitoring system
- Introduce OplogRecycleStats to track per-round recycling statistics
including skipped-by-snapshot count, earliest recyclable version,
failure count, and generated recycle key counts
(partition/index/rowset/txn)
- Add scan_and_statistics_operation_logs() to collect pre-recycling
statistics via RecyclerMetricsContext, consistent with existing
scan_and_statistics_* functions
- Use a PendingCounts pattern in OperationLogRecycler to avoid inflating
committed metrics on transaction failure — local counts are only merged
into OplogRecycleStats atomics after successful
commit()
- All new statistics are gated behind the enable_recycler_stats_metrics
config flag
This was referenced Mar 30, 2026
Closed
Closed
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 31, 2026
@Yukang-Lian
[Enhancement](Recycler) Add bvar metrics for operation logs (apache#6…
6870012
…0520)
- Add dedicated bvar metrics for the recycle_operation_logs path, which
previously had no observability through the monitoring system
- Introduce OplogRecycleStats to track per-round recycling statistics
including skipped-by-snapshot count, earliest recyclable version,
failure count, and generated recycle key counts
(partition/index/rowset/txn)
- Add scan_and_statistics_operation_logs() to collect pre-recycling
statistics via RecyclerMetricsContext, consistent with existing
scan_and_statistics_* functions
- Use a PendingCounts pattern in OperationLogRecycler to avoid inflating
committed metrics on transaction failure — local counts are only merged
into OplogRecycleStats atomics after successful
commit()
- All new statistics are gated behind the enable_recycler_stats_metrics
config flag
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Mar 31, 2026
@Yukang-Lian
[Enhancement](Recycler) Add bvar metrics for operation logs (apache#6…
1eb910d
…0520)
- Add dedicated bvar metrics for the recycle_operation_logs path, which
previously had no observability through the monitoring system
- Introduce OplogRecycleStats to track per-round recycling statistics
including skipped-by-snapshot count, earliest recyclable version,
failure count, and generated recycle key counts
(partition/index/rowset/txn)
- Add scan_and_statistics_operation_logs() to collect pre-recycling
statistics via RecyclerMetricsContext, consistent with existing
scan_and_statistics_* functions
- Use a PendingCounts pattern in OperationLogRecycler to avoid inflating
committed metrics on transaction failure — local counts are only merged
into OplogRecycleStats atomics after successful
commit()
- All new statistics are gated behind the enable_recycler_stats_metrics
config flag
This was referenced Mar 31, 2026
Merged
Merged
yiguolei pushed a commit that referenced this pull request Mar 31, 2026
@Yukang-Lian
branch-4.0: [pick] Cherry-pick#60520#60239#59331#60310(#61920)
e16a6a5
## Proposed changes
Cherry-pick the following PRs to branch-4.0:
- [Enhancement](Recycler) Add bvar metrics for operation logs #60520
- [Feat](Snapshot) require root privilege for cluster snapshot commands
#60239
- [Fix](Recycler) Fix recycler OOM by limiting queued delete tasks
#59331
- [feature](cloud) Support compaction read-write separation for cloud
mode #60310
yiguolei pushed a commit that referenced this pull request Apr 1, 2026
@Yukang-Lian
branch-4.1: [pick] Cherry-pick#60520#60239#60310(#61921)
b387fb9
## Proposed changes
Cherry-pick the following PRs to branch-4.1:
- [Enhancement](Recycler) Add bvar metrics for operation logs #60520
- [Feat](Snapshot) require root privilege for cluster snapshot commands
#60239
- [feature](cloud) Support compaction read-write separation for cloud
mode #60310
@morningmanmorningman mentioned this pull request Jun 3, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@w41terw41terw41ter approved these changes
@dataroaringdataroaringAwaiting requested review from dataroaring

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.clouddev/4.0.5-mergeddev/4.1.0-mergedreviewedsnapshot

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Yukang-Lian@hello-stephen@doris-robot@gavinchou@w41ter@yiguolei