Skip to content

[feature](cloud) Support query with freshness tolerance - #53540

Merged
dataroaring merged 35 commits into
apache:masterfrom
bobhan1:query-freshness-session-var
Sep 18, 2025
Merged

[feature](cloud) Support query with freshness tolerance#53540
dataroaring merged 35 commits into
apache:masterfrom
bobhan1:query-freshness-session-var

Conversation

@bobhan1

@bobhan1bobhan1 commented Jul 18, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

doc: apache/doris-website#2934

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jul 18, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
bobhan1force-pushed the query-freshness-session-var branch 3 times, most recently from 0919575 to 39a0eeaCompareJuly 25, 2025 02:48
dataroaring
dataroaring reviewed Jul 26, 2025
View reviewed changes
Comment threadbe/src/cloud/cloud_tablet.cpp Outdated
dataroaring
dataroaring reviewed Jul 26, 2025
View reviewed changes
Comment threadbe/src/cloud/cloud_tablet.cpp
@bobhan1
bobhan1force-pushed the query-freshness-session-var branch 10 times, most recently from 74d1d57 to 0791b6cCompareAugust 1, 2025 06:52
@bobhan1
bobhan1force-pushed the query-freshness-session-var branch 5 times, most recently from fa4aa5e to 963a65eCompareAugust 4, 2025 09:11
gavinchou
gavinchou reviewed Aug 5, 2025
View reviewed changes
Comment threadgensrc/proto/olap_file.proto Outdated
Comment threadcloud/src/meta-service/meta_service_job.cpp Outdated
Comment threadbe/src/olap/version_graph.h
Comment threadbe/src/olap/tablet_meta.h
Comment threadbe/src/olap/rowset/rowset_meta.h Outdated
Comment threadbe/src/cloud/cloud_tablet.h
Comment threadbe/src/cloud/cloud_tablet.cpp Outdated
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
Comment threadbe/src/pipeline/exec/olap_scan_operator.cpp Outdated
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
Comment threadbe/src/vec/exec/scan/olap_scanner.cpp Outdated
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
Comment threadcloud/src/meta-service/meta_service_txn.cpp Outdated
dataroaring
dataroaring reviewed Aug 7, 2025
View reviewed changes
Comment threadbe/src/olap/version_graph.h Outdated
@zhannngchenzhannngchen mentioned this pull request Aug 13, 2025
Merged
16 tasks
zhannngchen pushed a commit that referenced this pull request Aug 18, 2025
zhannngchen
[feat](warm up) introduce immediate warmup on read cluster (#54611)
64b57dc
### What problem does this PR solve?
Problem Summary:
introduce a config `enable_warmup_immediately_on_new_rowset`, if user
set it to `true`
warm up will be triggerd automatically and immediately on syncing new
rowsets
NOTE：
the method `get_rowset_warmup_state()` is used for #53540
@bobhan1
bobhan1force-pushed the query-freshness-session-var branch 2 times, most recently from 69ca66b to 60cc5f6CompareAugust 20, 2025 09:23
@dataroaring
dataroaring dismissed stale reviews from gavinchou and themself via 6bb6fb7September 18, 2025 04:02
@dataroaring

dataroaring commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 18, 2025
@hello-stephen

hello-stephen commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 58.14% (25/43) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage84.02% (1541/1834)
Line Coverage68.02% (27614/40594)
Region Coverage68.42% (13587/19859)
Branch Coverage58.73% (7257/12356)

@hello-stephen

hello-stephen commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 72.31% (47/65) 🎉
Increment coverage report
Complete coverage report

@doris-robot

doris-robot commented Sep 18, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 1517 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bb6fb750c97619ddc47d6584b1b7b7e3724bf08, data reload: false
------ Round 1 ----------------------------------
q1	17227	29	20	20
q2	2615	26	25	25
q3	10496	16	14	14
q4	11192	16	17	16
q5	8271	15	15	15
q6	236	14	13	13
q7	994	26	24	24
q8	9518	15	13	13
q9	9725	26	13	13
q10	7040	13	12	12
q11	715	21	20	20
q12	694	13	12	12
q13	18678	12	11	11
q14	250	12	11	11
q15	606	13	10	10
q16	1066	992	979	979
q17	565	13	11	11
q18	7686	13	11	11
q19	1697	13	12	12
q20	1940	372	243	243
q21	8956	21	21	21
q22	1093	13	11	11
Total cold run time: 121260 ms
Total hot run time: 1517 ms
----- Round 2, with runtime_filter_mode=off -----
q1	11	11	10	10
q2	19	21	20	20
q3	12	11	10	10
q4	13	11	10	10
q5	12	12	11	11
q6	11	11	10	10
q7	19	19	19	19
q8	11	11	11	11
q9	10	11	10	10
q10	10	10	11	10
q11	18	19	18	18
q12	11	11	11	11
q13	10	11	12	11
q14	11	12	12	12
q15	13	11	10	10
q16	1062	1099	1041	1041
q17	12	10	10	10
q18	10	10	10	10
q19	11	10	11	10
q20	1897	1935	1816	1816
q21	22	19	19	19
q22	10	10	10	10
Total cold run time: 3215 ms
Total hot run time: 3099 ms

@doris-robot

doris-robot commented Sep 18, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 2721 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bb6fb750c97619ddc47d6584b1b7b7e3724bf08, data reload: false
query1	1085	20	12	12
query2	7128	18	15	15
query3	7628	11	11	11
query4	27326	12	12	12
query5	4514	13	12	12
query6	368	12	10	10
query7	5443	12	10	10
query8	353	18	15	15
query9	9210	11	10	10
query10	735	11	10	10
query11	15929	11	10	10
query12	174	12	10	10
query13	1725	11	9	9
query14	10893	15	12	12
query15	235	10	9	9
query16	7170	11	10	10
query17	1528	12	35	12
query18	2774	10	9	9
query19	204	10	9	9
query20	133	11	9	9
query21	223	13	12	12
query22	4038	10	10	10
query23	33938	20	15	15
query24	9210	12	13	12
query25	722	9	9	9
query26	1065	10	9	9
query27	3405	10	8	8
query28	6295	10	9	9
query29	1196	10	8	8
query30	630	11	9	9
query31	1732	15	12	12
query32	109	11	9	9
query33	1071	12	11	11
query34	1589	819	512	512
query35	901	11	10	10
query36	993	11	10	10
query37	114	36	10	10
query38	3636	9	8	8
query39	1492	719	747	719
query40	227	9	11	9
query41	77	11	10	10
query42	147	10	9	9
query43	499	9	8	8
query44	1348	10	9	9
query45	195	10	9	9
query46	898	10	10	10
query47	1762	10	10	10
query48	410	10	9	9
query49	1132	10	9	9
query50	762	10	10	10
query51	3950	10	9	9
query52	119	10	9	9
query53	252	10	10	10
query54	722	9	10	9
query55	92	9	9	9
query56	324	11	10	10
query57	1174	10	9	9
query58	298	11	9	9
query59	2725	9	9	9
query60	371	10	9	9
query61	174	11	9	9
query62	836	10	9	9
query63	280	10	12	10
query64	4135	10	9	9
query65	4022	9	9	9
query66	1094	11	9	9
query67	16208	12	10	10
query68	3662	11	8	8
query69	727	10	9	9
query70	1399	10	9	9
query71	421	346	342	342
query72	7455	10	9	9
query73	527	9	9	9
query74	9517	11	10	10
query75	3358	10	11	10
query76	2527	10	11	10
query77	1013	11	10	10
query78	9798	11	10	10
query79	1058	15	12	12
query80	764	9	9	9
query81	729	10	9	9
query82	198	10	8	8
query83	290	11	10	10
query84	290	10	8	8
query85	918	9	10	9
query86	359	9	8	8
query87	3879	9	10	9
query88	2901	13	11	11
query89	406	11	11	11
query90	2007	10	9	9
query91	181	9	9	9
query92	76	9	8	8
query93	1070	10	9	9
query94	801	10	9	9
query95	527	10	10	10
query96	433	9	8	8
query97	3156	12	9	9
query98	233	228	223	223
query99	1487	10	11	10
Total cold run time: 283314 ms
Total hot run time: 2721 ms

@doris-robot

doris-robot commented Sep 18, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 0.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bb6fb750c97619ddc47d6584b1b7b7e3724bf08, data reload: false
query1	0.06	0.02	0.01
query2	0.12	0.00	0.01
query3	0.30	0.01	0.01
query4	1.71	0.01	0.00
query5	0.30	0.00	0.00
query6	1.64	0.00	0.01
query7	0.04	0.01	0.00
query8	0.08	0.00	0.00
query9	0.66	0.00	0.01
query10	0.62	0.00	0.00
query11	0.26	0.00	0.01
query12	0.26	0.00	0.01
query13	0.68	0.00	0.00
query14	1.10	0.00	0.00
query15	0.99	0.00	0.00
query16	0.42	0.00	0.00
query17	1.08	0.00	0.00
query18	0.24	0.01	0.00
query19	2.02	0.00	0.00
query20	0.03	0.01	0.00
query21	15.36	0.01	0.00
query22	5.71	0.00	0.00
query23	15.57	0.00	0.00
query24	1.81	0.00	0.00
query25	0.17	0.00	0.00
query26	0.20	0.00	0.01
query27	0.10	0.00	0.00
query28	1.57	0.00	0.00
query29	12.88	0.01	0.00
query30	0.34	0.00	0.01
query31	2.66	0.00	0.00
query32	6.28	0.00	0.00
query33	4.36	0.01	0.01
query34	8.50	0.00	0.01
query35	7.55	0.00	0.00
query36	0.68	0.01	0.00
query37	0.24	0.00	0.00
query38	0.20	0.00	0.00
query39	0.07	0.00	0.00
query40	0.23	0.00	0.00
query41	0.13	0.01	0.01
query42	0.10	0.00	0.00
query43	0.08	0.01	0.00
Total cold run time: 97.4 s
Total hot run time: 0.04 s

@hello-stephen

hello-stephen commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 80.08% (390/487) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.34% (17595/33614)
Line Coverage37.55% (159678/425288)
Region Coverage32.05% (121484/379020)
Branch Coverage33.43% (53349/159581)

@hello-stephen

hello-stephen commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 72.85% (432/593) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.03% (23334/32853)
Line Coverage57.37% (243483/424395)
Region Coverage52.76% (202719/384196)
Branch Coverage54.42% (87239/160318)

@hello-stephen

hello-stephen commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 78.46% (51/65) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Sep 18, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2025
@github-actions

github-actionsBot commented Sep 18, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

dataroaring
dataroaring approved these changes Sep 18, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 5e98aad into apache:masterSep 18, 2025
29 of 33 checks passed
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
zhannngchen@bobhan1
[feat](warm up) introduce immediate warmup on read cluster (apache#54611
df1efc9
)
Problem Summary:
introduce a config `enable_warmup_immediately_on_new_rowset`, if user
set it to `true`
warm up will be triggerd automatically and immediately on syncing new
rowsets
NOTE：
the method `get_rowset_warmup_state()` is used for apache#53540
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
43b9405
@bobhan1bobhan1 mentioned this pull request Sep 24, 2025
Merged
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
a37b02d
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 24, 2025
@bobhan1
[feature](cloud) Support query with freshness tolerance (apache#53540)
7d08978
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Sep 28, 2025
zhannngchen@bobhan1
[feat](warm up) introduce immediate warmup on read cluster (apache#54611
c783468
)
Problem Summary:
introduce a config `enable_warmup_immediately_on_new_rowset`, if user
set it to `true`
warm up will be triggerd automatically and immediately on syncing new
rowsets
NOTE：
the method `get_rowset_warmup_state()` is used for apache#53540
@bobhan1bobhan1 mentioned this pull request Sep 30, 2025
Merged
8 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@dataroaringdataroaringdataroaring approved these changes
@yiguoleiyiguoleiAwaiting requested review from yiguoleiyiguolei is a code owner
@w41terw41terAwaiting requested review from w41ter
+1 more reviewer
@pingchunzhangpingchunzhangpingchunzhang left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.1.2-mergeddev/4.0.x-conflictreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@bobhan1@Thearas@doris-robot@hello-stephen@dataroaring@gavinchou@pingchunzhang@morrySnow