[Fix](Variant) Schema should be updated when update_tmp_rowset in cloud mode by eldenmoon · Pull Request #37557 · apache/doris

eldenmoon · 2024-07-09T11:56:45Z

In cloud mode the updated rowset meta after rowset->merge_rowset_meta in BaseTablet::update_delete_bitmap will be lost in update_tmp_rowset.So in order to keep the latest schema we should keep schema in update_tmp_rowset
cloud tablet should also do copy_without_variant_extracted_columns

Proposed changes

Issue Number: close #xxx

…ud mode 1. In cloud mode the updated rowset meta after `rowset->merge_rowset_meta` in `BaseTablet::update_delete_bitmap` will be lost in `update_tmp_rowset`.So in order to keep the latest schema we should keep schema in update_tmp_rowset 2. cloud tablet should also do copy_without_variant_extracted_columns

doris-robot · 2024-07-09T11:56:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

eldenmoon · 2024-07-09T11:57:11Z

run buildall

github-actions

clang-tidy made some suggestions

gavinchou

LGTM

github-actions · 2024-07-09T12:10:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-09T12:10:28Z

PR approved by anyone and no changes requested.

eldenmoon · 2024-07-10T09:44:59Z

run buildall

doris-robot · 2024-07-10T10:02:31Z

TPC-H: Total hot run time: 39972 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c04081e6e4d37297efcb8aba31b5631a56cf15ef, data reload: false
------ Round 1 ----------------------------------
q1	18080	4542	4351	4351
q2	2610	200	194	194
q3	11956	1199	1222	1199
q4	10856	771	801	771
q5	7511	2686	2674	2674
q6	222	140	143	140
q7	970	604	619	604
q8	9235	2081	2060	2060
q9	8808	6473	6463	6463
q10	8875	3749	3752	3749
q11	479	243	248	243
q12	407	233	233	233
q13	18832	2983	2997	2983
q14	269	229	220	220
q15	529	490	498	490
q16	504	385	378	378
q17	960	689	688	688
q18	7991	7387	7409	7387
q19	1819	1490	1367	1367
q20	685	311	323	311
q21	5005	3864	3131	3131
q22	414	336	339	336
Total cold run time: 117017 ms
Total hot run time: 39972 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4321	4251	4214	4214
q2	376	277	274	274
q3	2997	2671	2738	2671
q4	1877	1610	1572	1572
q5	5269	5324	5254	5254
q6	223	134	136	134
q7	2150	1798	1745	1745
q8	3211	3339	3297	3297
q9	8312	8329	8310	8310
q10	3894	3580	3666	3580
q11	593	488	495	488
q12	784	601	593	593
q13	16426	3015	2985	2985
q14	297	264	266	264
q15	515	484	474	474
q16	482	422	420	420
q17	1779	1470	1451	1451
q18	7654	7602	7460	7460
q19	3382	1750	1550	1550
q20	1990	1806	1796	1796
q21	4896	4695	4704	4695
q22	646	557	543	543
Total cold run time: 72074 ms
Total hot run time: 53770 ms

doris-robot · 2024-07-10T10:13:42Z

TPC-DS: Total hot run time: 172182 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c04081e6e4d37297efcb8aba31b5631a56cf15ef, data reload: false
query1	921	364	374	364
query2	6470	2408	2397	2397
query3	6650	206	218	206
query4	27603	17382	17345	17345
query5	4239	487	504	487
query6	287	175	185	175
query7	4608	290	286	286
query8	326	321	293	293
query9	8579	2380	2371	2371
query10	467	325	273	273
query11	11644	10053	9996	9996
query12	124	86	83	83
query13	1634	375	378	375
query14	10359	6889	7406	6889
query15	233	182	177	177
query16	7490	300	298	298
query17	1363	559	514	514
query18	1853	273	267	267
query19	194	147	143	143
query20	89	79	82	79
query21	209	129	124	124
query22	4210	3970	3885	3885
query23	33762	32796	32914	32796
query24	12346	2816	2786	2786
query25	703	360	368	360
query26	1805	156	152	152
query27	3079	265	273	265
query28	7640	2065	2055	2055
query29	1124	612	598	598
query30	281	149	144	144
query31	954	732	745	732
query32	100	52	55	52
query33	769	306	286	286
query34	996	503	491	491
query35	675	553	565	553
query36	1112	927	938	927
query37	273	80	113	80
query38	2825	2766	2742	2742
query39	830	795	806	795
query40	275	122	117	117
query41	52	52	52	52
query42	110	95	101	95
query43	605	580	562	562
query44	1225	757	746	746
query45	190	161	157	157
query46	1083	731	694	694
query47	1828	1762	1760	1760
query48	376	315	314	314
query49	1185	407	415	407
query50	776	400	399	399
query51	6909	6684	6780	6684
query52	103	93	92	92
query53	357	295	287	287
query54	962	455	446	446
query55	75	76	74	74
query56	288	266	299	266
query57	1138	1032	1072	1032
query58	263	245	241	241
query59	3547	3392	3134	3134
query60	293	280	286	280
query61	95	97	94	94
query62	836	652	651	651
query63	327	292	291	291
query64	10554	2175	1661	1661
query65	3204	3102	3116	3102
query66	1389	330	335	330
query67	15468	14901	15055	14901
query68	4516	537	542	537
query69	461	342	327	327
query70	1171	1135	1138	1135
query71	390	287	281	281
query72	7057	5871	5133	5133
query73	731	321	322	321
query74	6056	5438	5440	5438
query75	3353	2701	2676	2676
query76	2360	1022	987	987
query77	451	299	303	299
query78	9420	9236	8957	8957
query79	2860	519	514	514
query80	951	491	482	482
query81	567	216	224	216
query82	282	135	127	127
query83	192	162	163	162
query84	274	88	84	84
query85	1001	370	354	354
query86	368	337	323	323
query87	3285	3067	3121	3067
query88	4275	2476	2476	2476
query89	494	380	390	380
query90	1949	195	198	195
query91	143	117	124	117
query92	66	56	52	52
query93	4056	518	509	509
query94	1339	223	225	223
query95	422	326	331	326
query96	597	270	283	270
query97	3159	3079	3041	3041
query98	219	199	202	199
query99	1563	1298	1264	1264
Total cold run time: 286367 ms
Total hot run time: 172182 ms

doris-robot · 2024-07-10T10:19:13Z

ClickBench: Total hot run time: 31.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c04081e6e4d37297efcb8aba31b5631a56cf15ef, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.06
query5	0.49	0.47	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.50
query10	0.56	0.54	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.13
query13	0.59	0.59	0.58
query14	0.76	0.77	0.77
query15	0.85	0.83	0.82
query16	0.37	0.37	0.37
query17	1.04	0.95	0.98
query18	0.24	0.22	0.22
query19	1.76	1.69	1.72
query20	0.01	0.01	0.01
query21	15.41	0.78	0.66
query22	4.02	6.68	2.72
query23	18.32	1.32	1.29
query24	2.11	0.23	0.24
query25	0.15	0.09	0.09
query26	0.30	0.20	0.20
query27	0.45	0.23	0.22
query28	13.24	1.02	1.00
query29	12.69	3.31	3.31
query30	0.26	0.06	0.06
query31	2.85	0.39	0.39
query32	3.29	0.48	0.47
query33	2.90	2.94	2.92
query34	17.12	4.30	4.37
query35	4.40	4.37	4.44
query36	0.65	0.49	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.13	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 31.38 s

…ud mode (apache#37557) 1. In cloud mode the updated rowset meta after `rowset->merge_rowset_meta` in `BaseTablet::update_delete_bitmap` will be lost in `update_tmp_rowset`.So in order to keep the latest schema we should keep schema in update_tmp_rowset 2. cloud tablet should also do copy_without_variant_extracted_columns

#37674 #37839 #37883 #37857 #37794 #37640 #37557 #37217 #37121

…ud mode (apache#37557) 1. In cloud mode the updated rowset meta after `rowset->merge_rowset_meta` in `BaseTablet::update_delete_bitmap` will be lost in `update_tmp_rowset`.So in order to keep the latest schema we should keep schema in update_tmp_rowset 2. cloud tablet should also do copy_without_variant_extracted_columns

eldenmoon changed the title ~~[Fix](Variant) Schema should be updated when update_tmp_rowset in clo…~~[Fix](Variant) Schema should be updated when update_tmp_rowset in cloud modeJul 9, 2024

eldenmoon added the dev/3.0.x label Jul 9, 2024

github-actionsBot reviewed Jul 9, 2024
View reviewed changes

apache deleted a comment from github-actionsBotJul 9, 2024

gavinchou approved these changes Jul 9, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2024

github-actionsBot added the reviewed label Jul 9, 2024

Merge branch 'master' into var-fetch-opt
c04081e

qidaye approved these changes Jul 11, 2024
View reviewed changes

eldenmoon merged commit e76f34e into apache:masterJul 11, 2024

eldenmoon deleted the var-fetch-opt branch July 11, 2024 03:15

eldenmoon mentioned this pull request Jul 16, 2024
[Pick](Variant) pick some fix #37926
Merged

eldenmoon added dev/3.0.0-merged and removed dev/3.0.x labels Jul 16, 2024

xiaokang pushed a commit that referenced this pull request Jul 17, 2024
[Pick](Variant) pick some fix (#37926)
2994232
#37674 #37839 #37883 #37857 #37794 #37640 #37557 #37217 #37121

Navigation Menu

Uh oh!

[Fix](Variant) Schema should be updated when update_tmp_rowset in cloud mode - #37557

[Fix](Variant) Schema should be updated when update_tmp_rowset in cloud mode#37557
eldenmoon merged 2 commits into
apache:masterfrom
eldenmoon:var-fetch-opt

eldenmoon commented Jul 9, 2024•
edited
Loading

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

eldenmoon commented Jul 9, 2024

Uh oh!

github-actionsBot left a comment

Uh oh!

gavinchou left a comment

Uh oh!

github-actionsBot commented Jul 9, 2024

Uh oh!

github-actionsBot commented Jul 9, 2024

Uh oh!

eldenmoon commented Jul 10, 2024

Uh oh!

doris-robot commented Jul 10, 2024

Uh oh!

doris-robot commented Jul 10, 2024

Uh oh!

doris-robot commented Jul 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

eldenmoon commented Jul 9, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

eldenmoon commented Jul 9, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

gavinchou left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 9, 2024

Uh oh!

github-actionsBot commented Jul 9, 2024

Uh oh!

eldenmoon commented Jul 10, 2024

Uh oh!

doris-robot commented Jul 10, 2024

Uh oh!

doris-robot commented Jul 10, 2024

Uh oh!

doris-robot commented Jul 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eldenmoon commented Jul 9, 2024•
edited
Loading