Skip to content

[feat](cloud) commit/drop partition/index save the min read version - #55001

Merged
dataroaring merged 1 commit into
apache:masterfrom
w41ter:snapshot/versioned/operation_log_snapshot_version_index_partition
Aug 20, 2025
Merged

[feat](cloud) commit/drop partition/index save the min read version#55001
dataroaring merged 1 commit into
apache:masterfrom
w41ter:snapshot/versioned/operation_log_snapshot_version_index_partition

Conversation

@w41ter

@w41terw41ter commented Aug 19, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@w41ter
w41ter requested a review from gavinchou as a code ownerAugust 19, 2025 08:34
@Thearas

Thearas commented Aug 19, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter

w41ter commented Aug 19, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 19, 2025

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 65.61% (103/157) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage83.60% (1412/1689)
Line Coverage66.92% (24820/37087)
Region Coverage67.86% (12364/18220)
Branch Coverage57.64% (6527/11324)

@hello-stephen

hello-stephen commented Aug 19, 2025

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot

doris-robot commented Aug 19, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34107 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b2e601cac4f9e7e59bfb3bff46010ecc1a473a5, data reload: false
------ Round 1 ----------------------------------
q1	17108	5177	5100	5100
q2	1923	281	188	188
q3	10055	1302	729	729
q4	10226	1010	522	522
q5	7587	2377	2345	2345
q6	174	161	132	132
q7	948	730	642	642
q8	9300	1291	1127	1127
q9	7029	5122	5114	5114
q10	6951	2375	1977	1977
q11	470	303	289	289
q12	351	344	221	221
q13	17774	3659	3025	3025
q14	233	236	217	217
q15	556	475	474	474
q16	414	426	386	386
q17	604	858	356	356
q18	8079	7211	7097	7097
q19	1094	952	594	594
q20	345	328	225	225
q21	4007	3236	2351	2351
q22	1082	1046	996	996
Total cold run time: 106310 ms
Total hot run time: 34107 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5200	5098	5104	5098
q2	238	328	219	219
q3	2191	2655	2315	2315
q4	1358	1806	1368	1368
q5	4207	4449	4577	4449
q6	204	161	124	124
q7	2115	1996	1828	1828
q8	2707	2637	2627	2627
q9	7325	7316	7269	7269
q10	3063	3285	2918	2918
q11	725	572	495	495
q12	690	782	633	633
q13	3579	3939	3348	3348
q14	295	310	288	288
q15	532	479	479	479
q16	491	508	432	432
q17	1235	1737	1388	1388
q18	7844	7619	7638	7619
q19	878	840	900	840
q20	2012	2109	1836	1836
q21	4768	4422	4270	4270
q22	1101	1045	1003	1003
Total cold run time: 52758 ms
Total hot run time: 50846 ms

@doris-robot

doris-robot commented Aug 19, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 184805 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b2e601cac4f9e7e59bfb3bff46010ecc1a473a5, data reload: false
query1	991	388	403	388
query2	6528	1745	1776	1745
query3	6738	218	220	218
query4	26144	23385	23205	23205
query5	4361	652	491	491
query6	323	224	210	210
query7	4628	515	290	290
query8	261	229	212	212
query9	8621	2874	2805	2805
query10	467	326	283	283
query11	15670	15039	14835	14835
query12	163	118	120	118
query13	1660	556	416	416
query14	8555	5845	5734	5734
query15	210	201	177	177
query16	7150	674	454	454
query17	1217	732	592	592
query18	1991	434	331	331
query19	208	192	168	168
query20	137	125	118	118
query21	213	127	109	109
query22	4160	4360	3969	3969
query23	34318	33379	33331	33331
query24	8159	2421	2479	2421
query25	540	488	427	427
query26	1243	285	163	163
query27	2742	533	342	342
query28	4302	2222	2196	2196
query29	806	594	460	460
query30	290	223	186	186
query31	884	889	763	763
query32	81	77	77	77
query33	557	381	347	347
query34	779	898	520	520
query35	807	845	777	777
query36	989	1039	914	914
query37	120	111	92	92
query38	4032	4060	3842	3842
query39	1643	1427	1406	1406
query40	232	126	115	115
query41	58	58	55	55
query42	125	111	114	111
query43	500	520	450	450
query44	1362	845	849	845
query45	173	175	161	161
query46	822	1070	642	642
query47	1763	1801	1719	1719
query48	391	428	309	309
query49	742	506	388	388
query50	676	735	404	404
query51	4078	4191	4017	4017
query52	112	111	102	102
query53	237	272	204	204
query54	612	591	531	531
query55	91	87	86	86
query56	311	327	298	298
query57	1179	1202	1123	1123
query58	278	273	269	269
query59	2587	2659	2639	2639
query60	348	339	328	328
query61	125	122	121	121
query62	788	749	664	664
query63	240	189	194	189
query64	4291	1089	696	696
query65	4284	4219	4226	4219
query66	1157	422	324	324
query67	15467	15290	14956	14956
query68	8704	942	586	586
query69	488	333	288	288
query70	1249	1120	1138	1120
query71	449	327	319	319
query72	5397	4918	5094	4918
query73	849	723	361	361
query74	9218	9110	8640	8640
query75	4077	3092	2649	2649
query76	3812	1221	769	769
query77	777	455	344	344
query78	9585	9626	8821	8821
query79	3959	869	587	587
query80	646	559	470	470
query81	477	259	216	216
query82	473	140	102	102
query83	291	262	239	239
query84	296	103	93	93
query85	843	371	339	339
query86	342	317	276	276
query87	4241	4335	4127	4127
query88	3010	2208	2184	2184
query89	436	311	290	290
query90	1962	231	225	225
query91	136	144	109	109
query92	88	82	71	71
query93	2363	982	636	636
query94	692	419	308	308
query95	401	315	304	304
query96	481	640	274	274
query97	2644	2723	2573	2573
query98	250	223	208	208
query99	1416	1415	1262	1262
Total cold run time: 275769 ms
Total hot run time: 184805 ms

@doris-robot

doris-robot commented Aug 19, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 33.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b2e601cac4f9e7e59bfb3bff46010ecc1a473a5, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.06	0.06
query3	0.29	0.07	0.06
query4	1.61	0.09	0.09
query5	0.43	0.41	0.42
query6	1.16	0.64	0.64
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.65	0.53	0.53
query10	0.58	0.59	0.59
query11	0.26	0.13	0.13
query12	0.25	0.14	0.13
query13	0.64	0.63	0.65
query14	0.82	0.85	0.84
query15	0.98	0.88	0.88
query16	0.38	0.39	0.38
query17	1.04	1.06	1.05
query18	0.23	0.22	0.23
query19	2.02	1.78	1.95
query20	0.03	0.01	0.01
query21	15.38	1.04	0.73
query22	0.95	1.11	1.16
query23	14.67	1.59	0.79
query24	5.55	0.60	0.31
query25	0.17	0.10	0.08
query26	0.54	0.21	0.17
query27	0.10	0.09	0.10
query28	10.99	1.25	0.58
query29	12.57	4.06	3.35
query30	3.06	3.05	3.02
query31	2.81	0.63	0.42
query32	3.25	0.61	0.51
query33	3.12	3.18	3.08
query34	16.40	5.46	4.81
query35	4.86	4.86	4.87
query36	0.63	0.52	0.49
query37	0.22	0.19	0.18
query38	0.18	0.18	0.16
query39	0.05	0.05	0.05
query40	0.20	0.17	0.16
query41	0.10	0.05	0.05
query42	0.07	0.06	0.06
query43	0.06	0.05	0.05
Total cold run time: 107.55 s
Total hot run time: 33.64 s

@doris-robot

doris-robot commented Aug 19, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage59.57% (16769/28150)
Line Coverage48.51% (152550/314482)
Region Coverage37.49% (116017/309497)
Branch Coverage40.39% (51220/126806)

gavinchou
gavinchou approved these changes Aug 19, 2025
View reviewed changes

@gavinchougavinchou left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2025
@github-actions

github-actionsBot commented Aug 19, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 19, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

hello-stephen commented Aug 20, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage82.13% (22654/27582)
Line Coverage74.80% (235118/314314)
Region Coverage61.81% (194680/314961)
Branch Coverage66.08% (84430/127763)

dataroaring
dataroaring approved these changes Aug 20, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 4bedf08 into apache:masterAug 20, 2025
28 of 31 checks passed
@w41ter
w41ter deleted the snapshot/versioned/operation_log_snapshot_version_index_partition branch August 20, 2025 12:12
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.cloudreviewedsnapshot

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@w41ter@Thearas@doris-robot@hello-stephen@gavinchou@dataroaring