[Feature](Row store) support column group with store row format for partial columns of table by eldenmoon · Pull Request #34089 · apache/doris

eldenmoon · 2024-04-24T14:11:26Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-24T14:11:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-24T14:18:11Z


+ int32_t rs_column_uid() const { return _rs_column_uid; }
+
+ const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; }


warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
Suggested change
conststd::unordered_set<int32_t>missing_col_uids() const { return_missing_col_uids; }
std::unordered_set<int32_t>missing_col_uids() const { return_missing_col_uids; }

github-actions · 2024-04-24T14:18:11Z

+
+ const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; }
+
+ const std::unordered_set<int32_t> include_col_uids() const { return _include_col_uids; }


warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
Suggested change
conststd::unordered_set<int32_t>include_col_uids() const { return_include_col_uids; }
std::unordered_set<int32_t>include_col_uids() const { return_include_col_uids; }

eldenmoon · 2024-04-25T03:05:48Z

run buildall

doris-robot · 2024-04-25T04:40:04Z

TeamCity be ut coverage result:
Function Coverage: 35.18% (8917/25345)
Line Coverage: 26.97% (73355/271978)
Region Coverage: 26.14% (37890/144923)
Branch Coverage: 22.97% (19303/84022)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7f805bb130b4017ce13046ab37857cd80e3a501b_7f805bb130b4017ce13046ab37857cd80e3a501b/report/index.html

eldenmoon · 2024-04-26T02:07:36Z

run buildall

doris-robot · 2024-04-26T03:12:56Z

TPC-H: Total hot run time: 40423 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1df17905dc6152130626833eb8936a8ef23cc91e, data reload: false
------ Round 1 ----------------------------------
q1	17615	4474	4235	4235
q2	2027	186	186	186
q3	10513	1224	1297	1224
q4	10190	762	815	762
q5	7552	2729	2625	2625
q6	213	131	131	131
q7	1016	604	611	604
q8	9217	2078	2050	2050
q9	9580	7185	7147	7147
q10	8770	3747	3725	3725
q11	459	239	238	238
q12	431	217	212	212
q13	17767	2955	2977	2955
q14	275	230	230	230
q15	509	468	470	468
q16	539	380	383	380
q17	977	654	671	654
q18	7998	7363	7466	7363
q19	2497	1502	1515	1502
q20	651	315	303	303
q21	5018	3163	3994	3163
q22	339	267	266	266
Total cold run time: 114153 ms
Total hot run time: 40423 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4359	4193	4194	4193
q2	365	280	266	266
q3	2988	2743	2755	2743
q4	1945	1613	1586	1586
q5	5305	5293	5285	5285
q6	208	119	120	119
q7	2272	1878	1904	1878
q8	3198	3353	3365	3353
q9	9234	9253	9269	9253
q10	3930	3715	3726	3715
q11	595	488	474	474
q12	795	582	599	582
q13	16379	2979	2946	2946
q14	301	290	272	272
q15	518	479	470	470
q16	465	416	447	416
q17	1759	1482	1466	1466
q18	7659	7429	7577	7429
q19	2340	1551	1539	1539
q20	1941	1752	1794	1752
q21	4909	4869	4853	4853
q22	567	470	477	470
Total cold run time: 72032 ms
Total hot run time: 55060 ms

doris-robot · 2024-04-26T03:24:58Z

TPC-DS: Total hot run time: 188110 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1df17905dc6152130626833eb8936a8ef23cc91e, data reload: false
query1	910	357	356	356
query2	6477	2454	2408	2408
query3	6661	204	207	204
query4	24481	21327	21340	21327
query5	4272	419	418	418
query6	275	179	170	170
query7	4572	302	283	283
query8	245	184	205	184
query9	8432	2377	2369	2369
query10	596	275	257	257
query11	15039	14400	14371	14371
query12	141	96	89	89
query13	1645	375	358	358
query14	8656	7866	8255	7866
query15	288	187	182	182
query16	8137	264	269	264
query17	1925	569	569	569
query18	2099	275	270	270
query19	230	144	149	144
query20	91	83	84	83
query21	200	131	134	131
query22	4972	4828	4842	4828
query23	33815	33098	33135	33098
query24	10825	3021	2980	2980
query25	626	365	386	365
query26	1534	156	151	151
query27	2978	318	321	318
query28	7273	2052	2017	2017
query29	948	609	596	596
query30	294	152	151	151
query31	973	756	745	745
query32	91	51	55	51
query33	745	248	239	239
query34	1055	483	484	483
query35	854	701	699	699
query36	1057	908	871	871
query37	147	67	66	66
query38	3345	3158	3185	3158
query39	1579	1548	1518	1518
query40	273	127	124	124
query41	45	39	40	39
query42	102	95	98	95
query43	574	534	547	534
query44	1238	722	737	722
query45	295	266	286	266
query46	1081	748	739	739
query47	1910	1874	1832	1832
query48	372	309	295	295
query49	1184	410	417	410
query50	768	393	406	393
query51	6872	6809	6664	6664
query52	103	88	97	88
query53	352	286	284	284
query54	306	237	251	237
query55	78	74	75	74
query56	261	253	252	252
query57	1233	1123	1121	1121
query58	225	201	207	201
query59	3312	3229	3166	3166
query60	267	252	266	252
query61	115	108	108	108
query62	642	438	455	438
query63	305	286	293	286
query64	9521	7330	7229	7229
query65	3131	3079	3072	3072
query66	1420	346	344	344
query67	15303	15139	14885	14885
query68	8442	555	546	546
query69	533	315	311	311
query70	1177	1128	1097	1097
query71	485	276	273	273
query72	8076	2753	2552	2552
query73	751	341	328	328
query74	6940	6430	6451	6430
query75	4016	2705	2667	2667
query76	4524	926	1036	926
query77	634	267	274	267
query78	11005	10266	10202	10202
query79	11197	538	524	524
query80	1831	480	586	480
query81	545	225	223	223
query82	342	98	94	94
query83	209	167	170	167
query84	268	80	86	80
query85	1004	268	267	267
query86	353	303	318	303
query87	3559	3309	3246	3246
query88	5320	2418	2422	2418
query89	508	383	383	383
query90	1985	187	186	186
query91	129	99	98	98
query92	68	48	49	48
query93	7398	532	515	515
query94	1360	185	183	183
query95	395	316	300	300
query96	639	266	265	265
query97	3122	2899	2960	2899
query98	242	220	215	215
query99	1229	860	830	830
Total cold run time: 308057 ms
Total hot run time: 188110 ms

doris-robot · 2024-04-26T03:42:14Z

TeamCity be ut coverage result:
Function Coverage: 35.18% (8919/25351)
Line Coverage: 26.97% (73369/272004)
Region Coverage: 26.15% (37902/144944)
Branch Coverage: 22.97% (19304/84052)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1df17905dc6152130626833eb8936a8ef23cc91e_1df17905dc6152130626833eb8936a8ef23cc91e/report/index.html

xiaokang

fe reviewed

xiaokang · 2024-05-13T03:33:20Z

 }
 }
+
+ if (properties != null) {


put it inside row store branch

xiaokang

be reviewed

xiaokang · 2024-05-13T03:38:51Z

+
+ // groups of this column belongs to
+ // contains column ids of group, and encode the group of columns into row store
+ std::vector<int32_t> _group_ids;


_row_store_group_ids

xiaokang · 2024-05-13T03:40:35Z

- void set_store_row_column(bool store_row_column) { _store_row_column = store_row_column; }
- bool store_row_column() const { return _store_row_column; }
+ // indicate if full row store column(all the columns encodes as row) exists
+ bool has_full_row_store_column() const { return _store_row_column; }


consistent name is better

xiaokang · 2024-05-13T03:55:14Z

 static void block_to_jsonb(const TabletSchema& schema, const Block& block, ColumnString& dst,
- int num_cols, const DataTypeSerDeSPtrs& serdes);
+ int num_cols, const DataTypeSerDeSPtrs& serdes,
+ const std::unordered_set<int32_t>& group_cids);


consistent name include_cids is better

xiaokang · 2024-05-13T04:03:38Z

 std::unordered_map<uint32_t, uint32_t> _col_uid_to_idx;
 std::vector<std::string> _col_default_values;
+ // picked rowstore(column group) column unique id
+ int32_t _rs_column_uid;


what's the meaning of name rs?

_row_store_coloumn_uid

xiaokang · 2024-05-13T04:10:29Z

+ const TabletColumn& target_rs_column = schema.column_by_uid(target_rs_column_id);
+ DCHECK(target_rs_column.is_row_store_column());
+ // The full column group is considered a full match, thus no missing cids
+ if (target_rs_column.group_ids().empty()) {


consider that if it's right after light schema chage

eldenmoon · 2024-06-05T16:20:20Z

run buildall

eldenmoon · 2024-06-05T16:21:12Z

run buildall

doris-robot · 2024-06-05T18:01:05Z

TeamCity be ut coverage result:
Function Coverage: 36.38% (8986/24703)
Line Coverage: 27.91% (73425/263116)
Region Coverage: 27.35% (38027/139051)
Branch Coverage: 23.94% (19302/80626)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1981d224f3b97bafdaa05c00e855364d5a356113_1981d224f3b97bafdaa05c00e855364d5a356113/report/index.html

eldenmoon · 2024-06-06T01:36:33Z

run buildall

doris-robot · 2024-06-06T02:05:35Z

TeamCity be ut coverage result:
Function Coverage: 36.38% (8985/24696)
Line Coverage: 27.91% (73400/262983)
Region Coverage: 27.34% (38008/139005)
Branch Coverage: 23.93% (19285/80574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d762d6bd2a3e26c2fcf4084cb3251a028b8796d_7d762d6bd2a3e26c2fcf4084cb3251a028b8796d/report/index.html

doris-robot · 2024-06-06T02:14:34Z

TPC-H: Total hot run time: 40900 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d762d6bd2a3e26c2fcf4084cb3251a028b8796d, data reload: false
------ Round 1 ----------------------------------
q1	17612	4275	4235	4235
q2	2021	194	203	194
q3	10448	1216	1124	1124
q4	10185	873	886	873
q5	7512	2688	2725	2688
q6	220	133	137	133
q7	944	601	617	601
q8	9209	2130	2113	2113
q9	9288	6642	6679	6642
q10	9340	3914	3877	3877
q11	431	246	241	241
q12	450	229	236	229
q13	17276	3206	3271	3206
q14	283	222	223	222
q15	533	462	472	462
q16	490	379	385	379
q17	969	601	543	543
q18	8347	7793	7795	7793
q19	8106	1459	1304	1304
q20	657	317	314	314
q21	5145	3398	4118	3398
q22	398	333	329	329
Total cold run time: 119864 ms
Total hot run time: 40900 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4612	4470	4423	4423
q2	367	273	270	270
q3	3119	2907	2930	2907
q4	1889	1610	1627	1610
q5	5498	5473	5520	5473
q6	222	126	135	126
q7	2189	1806	1788	1788
q8	3238	3395	3388	3388
q9	8593	8542	8685	8542
q10	4075	3866	3727	3727
q11	579	490	471	471
q12	788	580	594	580
q13	15935	3152	3135	3135
q14	283	275	265	265
q15	520	494	464	464
q16	476	432	421	421
q17	1779	1479	1501	1479
q18	8104	7708	7379	7379
q19	1718	1618	1547	1547
q20	3041	1770	1791	1770
q21	6112	4637	4656	4637
q22	620	536	541	536
Total cold run time: 73757 ms
Total hot run time: 54938 ms

doris-robot · 2024-06-06T02:25:25Z

TPC-DS: Total hot run time: 172450 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d762d6bd2a3e26c2fcf4084cb3251a028b8796d, data reload: false
query1	936	382	382	382
query2	6439	2615	2461	2461
query3	6639	205	211	205
query4	20385	17048	17176	17048
query5	4130	456	455	455
query6	248	163	145	145
query7	4603	299	299	299
query8	324	280	281	280
query9	8456	2341	2303	2303
query10	465	297	273	273
query11	10416	10040	9880	9880
query12	138	93	89	89
query13	1640	364	374	364
query14	10117	7732	7649	7649
query15	236	187	180	180
query16	7822	256	258	256
query17	1661	517	521	517
query18	1967	284	284	284
query19	208	180	153	153
query20	89	81	78	78
query21	208	139	126	126
query22	4426	4023	4054	4023
query23	33608	32960	32896	32896
query24	11031	2813	2771	2771
query25	601	373	353	353
query26	721	151	156	151
query27	2294	302	310	302
query28	5908	1987	2004	1987
query29	860	594	597	594
query30	285	149	150	149
query31	973	746	732	732
query32	94	52	53	52
query33	755	282	278	278
query34	972	470	475	470
query35	771	673	626	626
query36	1102	939	923	923
query37	138	67	75	67
query38	2850	2735	2713	2713
query39	886	818	805	805
query40	205	124	121	121
query41	48	50	48	48
query42	113	97	96	96
query43	583	568	525	525
query44	1208	733	741	733
query45	194	169	160	160
query46	1076	710	699	699
query47	1858	1772	1767	1767
query48	367	308	293	293
query49	989	400	403	400
query50	800	375	387	375
query51	6760	6663	6679	6663
query52	100	92	86	86
query53	364	290	295	290
query54	936	450	461	450
query55	74	71	76	71
query56	273	247	254	247
query57	1136	1040	1051	1040
query58	261	260	248	248
query59	3518	3363	3146	3146
query60	296	271	268	268
query61	90	98	89	89
query62	647	443	436	436
query63	317	298	289	289
query64	8840	2203	1700	1700
query65	3174	3089	3120	3089
query66	801	353	343	343
query67	15200	14791	14732	14732
query68	4637	549	542	542
query69	504	448	388	388
query70	1124	1057	1067	1057
query71	377	272	271	271
query72	7244	5698	5764	5698
query73	770	325	322	322
query74	5845	5417	5558	5417
query75	3397	2635	2654	2635
query76	2312	955	879	879
query77	519	302	301	301
query78	10253	9766	9631	9631
query79	2377	511	512	511
query80	1023	474	457	457
query81	586	227	220	220
query82	731	106	99	99
query83	234	166	168	166
query84	249	83	91	83
query85	1530	269	268	268
query86	510	310	307	307
query87	3290	3072	3089	3072
query88	4132	2425	2419	2419
query89	497	406	380	380
query90	1777	195	190	190
query91	126	97	99	97
query92	58	48	53	48
query93	1929	517	493	493
query94	1197	183	185	183
query95	399	308	306	306
query96	609	271	268	268
query97	3236	3017	2998	2998
query98	215	202	191	191
query99	1382	838	830	830
Total cold run time: 268699 ms
Total hot run time: 172450 ms

doris-robot · 2024-06-18T06:27:35Z

TeamCity be ut coverage result:
Function Coverage: 36.47% (9003/24688)
Line Coverage: 28.02% (73771/263315)
Region Coverage: 27.50% (38321/139371)
Branch Coverage: 24.19% (19533/80758)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d6a4a2dfbb7fcf2c6eb239814d64b46c03a175f0_d6a4a2dfbb7fcf2c6eb239814d64b46c03a175f0/report/index.html

eldenmoon · 2024-06-18T09:07:45Z

run buildall

doris-robot · 2024-06-18T09:35:49Z

TeamCity be ut coverage result:
Function Coverage: 36.47% (9004/24688)
Line Coverage: 28.02% (73777/263323)
Region Coverage: 27.50% (38332/139388)
Branch Coverage: 24.19% (19538/80760)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1921a29b81f6fce923a5cd6bee61d93f1e45d2de_1921a29b81f6fce923a5cd6bee61d93f1e45d2de/report/index.html

doris-robot · 2024-06-18T10:46:18Z

TPC-H: Total hot run time: 40387 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1921a29b81f6fce923a5cd6bee61d93f1e45d2de, data reload: false
------ Round 1 ----------------------------------
q1	17634	4321	4244	4244
q2	2030	194	191	191
q3	10462	1211	1123	1123
q4	10198	746	834	746
q5	7453	2687	2643	2643
q6	217	133	133	133
q7	953	605	602	602
q8	9244	2052	2084	2052
q9	8900	6503	6501	6501
q10	8892	3687	3763	3687
q11	441	235	236	235
q12	454	224	231	224
q13	18707	2988	2978	2978
q14	265	211	220	211
q15	517	485	473	473
q16	538	382	381	381
q17	959	675	680	675
q18	8278	8110	7833	7833
q19	8009	1540	1490	1490
q20	682	343	346	343
q21	5069	4078	3274	3274
q22	408	356	348	348
Total cold run time: 120310 ms
Total hot run time: 40387 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4579	4404	4414	4404
q2	375	279	274	274
q3	3043	2828	2877	2828
q4	1940	1650	1671	1650
q5	5515	5499	5561	5499
q6	224	129	126	126
q7	2212	1845	1863	1845
q8	3285	3397	3417	3397
q9	8696	8714	8716	8714
q10	4049	3839	3902	3839
q11	601	513	487	487
q12	819	648	706	648
q13	17231	3013	3182	3013
q14	282	260	264	260
q15	541	474	477	474
q16	498	448	429	429
q17	1846	1505	1478	1478
q18	7915	7899	7650	7650
q19	1778	1513	1716	1513
q20	3091	1878	1861	1861
q21	8481	4768	4987	4768
q22	723	573	546	546
Total cold run time: 77724 ms
Total hot run time: 55703 ms

doris-robot · 2024-06-18T10:57:12Z

TPC-DS: Total hot run time: 173431 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1921a29b81f6fce923a5cd6bee61d93f1e45d2de, data reload: false
query1	797	396	379	379
query2	6411	2438	2424	2424
query3	6416	210	215	210
query4	20767	17445	17251	17251
query5	3771	474	485	474
query6	249	158	155	155
query7	4470	298	289	289
query8	329	286	282	282
query9	8619	2415	2387	2387
query10	593	307	281	281
query11	10456	9988	10001	9988
query12	132	89	83	83
query13	1569	366	361	361
query14	10056	7227	7707	7227
query15	232	194	191	191
query16	7143	269	257	257
query17	1403	528	523	523
query18	1868	270	272	270
query19	193	149	152	149
query20	91	83	82	82
query21	212	138	130	130
query22	4366	4042	3904	3904
query23	33842	33790	33553	33553
query24	11131	2750	2766	2750
query25	584	366	351	351
query26	724	156	148	148
query27	2240	321	318	318
query28	5980	2040	2053	2040
query29	851	624	619	619
query30	254	150	151	150
query31	949	734	764	734
query32	86	56	53	53
query33	716	275	301	275
query34	869	487	465	465
query35	762	692	602	602
query36	1131	952	917	917
query37	137	71	69	69
query38	2865	2735	2728	2728
query39	844	813	774	774
query40	213	135	122	122
query41	53	46	49	46
query42	124	99	101	99
query43	599	565	543	543
query44	1163	752	734	734
query45	195	162	167	162
query46	1064	707	732	707
query47	1849	1790	1779	1779
query48	380	292	288	288
query49	938	407	409	407
query50	768	387	390	387
query51	6815	6710	6741	6710
query52	107	89	97	89
query53	357	291	291	291
query54	879	452	433	433
query55	104	73	81	73
query56	289	258	269	258
query57	1158	1055	1053	1053
query58	259	241	262	241
query59	3584	3353	3484	3353
query60	284	275	279	275
query61	93	94	92	92
query62	616	455	441	441
query63	319	303	300	300
query64	8949	2273	1778	1778
query65	3177	3126	3102	3102
query66	902	331	331	331
query67	15196	14792	14795	14792
query68	4450	548	537	537
query69	462	316	321	316
query70	1151	1093	1138	1093
query71	362	281	280	280
query72	7075	5676	5266	5266
query73	738	323	329	323
query74	5895	5594	5580	5580
query75	3376	2666	2658	2658
query76	2244	937	907	907
query77	496	307	313	307
query78	10307	9876	9716	9716
query79	2668	513	515	513
query80	2313	468	469	468
query81	562	219	226	219
query82	823	111	107	107
query83	309	172	167	167
query84	263	87	90	87
query85	1853	292	279	279
query86	476	316	311	311
query87	3256	3087	3093	3087
query88	3938	2340	2333	2333
query89	472	393	386	386
query90	1799	190	187	187
query91	141	112	113	112
query92	63	51	55	51
query93	2356	511	509	509
query94	1223	204	198	198
query95	428	331	326	326
query96	589	279	272	272
query97	3225	3068	3054	3054
query98	229	209	203	203
query99	1414	850	856	850
Total cold run time: 269498 ms
Total hot run time: 173431 ms

doris-robot · 2024-06-18T11:02:43Z

ClickBench: Total hot run time: 30.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1921a29b81f6fce923a5cd6bee61d93f1e45d2de, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.08	0.09
query5	0.50	0.49	0.50
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.54	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.79	0.78	0.78
query15	0.83	0.80	0.82
query16	0.38	0.38	0.37
query17	1.02	1.05	1.03
query18	0.20	0.27	0.25
query19	1.83	1.69	1.71
query20	0.01	0.00	0.01
query21	15.42	0.66	0.66
query22	4.22	7.33	1.99
query23	18.33	1.44	1.28
query24	2.12	0.23	0.23
query25	0.15	0.10	0.08
query26	0.26	0.18	0.18
query27	0.08	0.09	0.08
query28	13.30	1.04	1.01
query29	12.87	3.30	3.27
query30	0.25	0.06	0.06
query31	2.85	0.38	0.38
query32	3.28	0.48	0.47
query33	2.88	2.94	2.95
query34	17.05	4.42	4.39
query35	4.47	4.54	4.47
query36	0.65	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.17	0.14
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.66 s
Total hot run time: 30.77 s

xiaokang

LGTM

github-actions · 2024-06-18T14:19:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-18T14:19:04Z

PR approved by anyone and no changes requested.

…artial columns of table (#34089)

…36658) previous PR: #34089

### What problem does this PR solve? Problem Summary: 1. Because defer materialization in BE do not support project on scan node, so we use a little trick to do projection on last fragment's OutputExprs. 2. fix core when table use partial column row store, introduced by #34089 thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start

### What problem does this PR solve? Problem Summary: 1. Because defer materialization in BE do not support project on scan node, so we use a little trick to do projection on last fragment's OutputExprs. 2. fix core when table use partial column row store, introduced by apache#34089 thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start

…artial columns of table (apache#34089)

…pache#36658) previous PR: apache#34089

eldenmoon marked this pull request as draft April 24, 2024 14:11

github-actionsBot added the meta-change label Apr 24, 2024

github-actionsBot reviewed Apr 24, 2024
View reviewed changes

eldenmoon changed the title ~~Rs group~~[Feature](Row store) support column group which store row format for partial columns of tableApr 25, 2024

eldenmoon force-pushed the rs-group branch from e484058 to 7f805bbCompare April 25, 2024 03:04

eldenmoon marked this pull request as ready for review April 25, 2024 03:05

eldenmoon changed the title ~~[Feature](Row store) support column group which store row format for partial columns of table~~[Feature](Row store) support column group with store row format for partial columns of tableApr 26, 2024

eldenmoon force-pushed the rs-group branch from 7f805bb to 1df1790Compare April 26, 2024 02:07

xiaokang reviewed May 13, 2024
View reviewed changes

eldenmoon force-pushed the rs-group branch from 1df1790 to 33b72e1Compare June 4, 2024 03:58

eldenmoon marked this pull request as draft June 4, 2024 03:58

eldenmoon force-pushed the rs-group branch 4 times, most recently from 6f0854c to 1981d22Compare June 5, 2024 16:20

eldenmoon force-pushed the rs-group branch from d6a4a2d to cb80835Compare June 18, 2024 09:04

fix4
1921a29

eldenmoon force-pushed the rs-group branch from cb80835 to 1921a29Compare June 18, 2024 09:07

xiaokang approved these changes Jun 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2024

github-actionsBot added the reviewed label Jun 18, 2024

lxr599 approved these changes Jun 19, 2024
View reviewed changes

eldenmoon merged commit 53a555f into apache:masterJun 19, 2024

xiaokang added the dev/2.1.x label Jun 19, 2024

eldenmoon deleted the rs-group branch June 19, 2024 07:27

eldenmoon mentioned this pull request Jun 21, 2024
[Feature](Row Store) support partial columns row store in cloud mode #36658
Merged

eldenmoon removed the dev/2.1.x label Jun 21, 2024

dataroaring pushed a commit that referenced this pull request Jun 21, 2024
[Feature](Row store) support column group with store row format for p…
e6bd347
…artial columns of table (#34089)

eldenmoon added a commit that referenced this pull request Jun 25, 2024
[Feature](Row Store) support partial columns row store in cloud mode (#…
6551bf6
…36658) previous PR: #34089

dataroaring pushed a commit that referenced this pull request Jun 26, 2024
[Feature](Row Store) support partial columns row store in cloud mode (#…
3dc13c8
…36658) previous PR: #34089

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

morrySnow mentioned this pull request Feb 11, 2025
[opt](Nereids) support defer materialization with project #47661
Merged
16 tasks

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Feature](Row store) support column group with store row format for p…
be570a0
…artial columns of table (apache#34089)

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Feature](Row Store) support partial columns row store in cloud mode (a…
7f30aee
…pache#36658) previous PR: apache#34089


		int32_t rs_column_uid() const { return _rs_column_uid; }

		const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; }

	conststd::unordered_set<int32_t>missing_col_uids() const { return_missing_col_uids; }
	std::unordered_set<int32_t>missing_col_uids() const { return_missing_col_uids; }


		const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; }

		const std::unordered_set<int32_t> include_col_uids() const { return _include_col_uids; }

	conststd::unordered_set<int32_t>include_col_uids() const { return_include_col_uids; }
	std::unordered_set<int32_t>include_col_uids() const { return_include_col_uids; }

Uh oh!

Conversation

eldenmoon commented Apr 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 24, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 24, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 24, 2024

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented Apr 25, 2024

Uh oh!

doris-robot commented Apr 25, 2024

Uh oh!

eldenmoon commented Apr 26, 2024

Uh oh!

doris-robot commented Apr 26, 2024

Uh oh!

doris-robot commented Apr 26, 2024

Uh oh!

doris-robot commented Apr 26, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xiaokangMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

xiaokangMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokangMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xiaokangMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokangMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokangJun 11, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokangMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Jun 5, 2024

Uh oh!

eldenmoon commented Jun 5, 2024

Uh oh!

doris-robot commented Jun 5, 2024

Uh oh!

eldenmoon commented Jun 6, 2024

Uh oh!

doris-robot commented Jun 6, 2024

Uh oh!

doris-robot commented Jun 6, 2024

Uh oh!

eldenmoon commented Apr 24, 2024•
edited
Loading