[Fix](partial update) Fix rowset not found error when doing partial update #34112 by Yukang-Lian · Pull Request #34357 · apache/doris

Yukang-Lian · 2024-04-30T09:21:16Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…pdate (apache#34112) Cause: In the logic of partial column updates, the existing data columns are read first, and then the data is supplemented and written back. During the reading process, initialization involves initially fetching rowset IDs, and the actual rowset object is fetched only when needed later. However, between fetching the rowset IDs and the rowset object, compaction may occur, turning the old rowset into a stale rowset. If too much time passes, the stale rowset might be directly deleted. Thus, when the rowset object is needed for an update, it cannot be found. Although the update operation with partial column logic should be able to read all keys and should not encounter new keys, if the rowset disappears, the Backend (BE) will consider these keys as missing. Consequently, it will check whether other columns have default values or are nullable. If this check fails, the aforementioned error is thrown. Solution: To avoid such issues during partial column updates, the initialization step should involve fetching both the rowset IDs and the shared pointer to the rowset object simultaneously. This ensures that the rowset can always be found during data retrieval.

doris-robot · 2024-04-30T09:21:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Yukang-Lian · 2024-04-30T09:21:26Z

run buildall

github-actions · 2024-04-30T09:28:38Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-30T10:19:11Z

TPC-H: Total hot run time: 49680 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb0a21af5677fb6da3636c08a7ada358705ed22b, data reload: false
------ Round 1 ----------------------------------
q1	18155	4408	4353	4353
q2	2026	155	139	139
q3	10284	1890	1937	1890
q4	10098	1228	1311	1228
q5	8546	4209	3927	3927
q6	231	122	121	121
q7	2044	1603	1570	1570
q8	9269	2719	2726	2719
q9	10522	10434	10261	10261
q10	8671	3505	3568	3505
q11	411	235	243	235
q12	475	299	301	299
q13	18348	3983	4037	3983
q14	357	320	325	320
q15	509	454	451	451
q16	682	568	590	568
q17	1126	981	991	981
q18	7329	6955	6912	6912
q19	1725	1601	1491	1491
q20	515	320	328	320
q21	4409	4141	4032	4032
q22	486	375	400	375
Total cold run time: 116218 ms
Total hot run time: 49680 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4357	4295	4304	4295
q2	323	226	223	223
q3	4180	4140	4150	4140
q4	2742	2756	2720	2720
q5	7227	7093	7126	7093
q6	239	119	117	117
q7	3246	2823	2826	2823
q8	4360	4492	4487	4487
q9	16984	16806	16825	16806
q10	4239	4273	4276	4273
q11	755	697	657	657
q12	1021	869	842	842
q13	6622	3722	3772	3722
q14	472	427	427	427
q15	505	457	454	454
q16	727	683	676	676
q17	3844	3817	3843	3817
q18	8749	8792	8877	8792
q19	1730	1689	1682	1682
q20	2394	2162	2094	2094
q21	8542	8427	8405	8405
q22	993	943	969	943
Total cold run time: 84251 ms
Total hot run time: 79488 ms

doris-robot · 2024-04-30T10:25:48Z

TeamCity be ut coverage result:
Function Coverage: 37.79% (8069/21351)
Line Coverage: 29.45% (65904/223750)
Region Coverage: 28.93% (33927/117283)
Branch Coverage: 24.79% (17416/70242)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb0a21af5677fb6da3636c08a7ada358705ed22b_bb0a21af5677fb6da3636c08a7ada358705ed22b/report/index.html

doris-robot · 2024-04-30T10:31:48Z

TPC-DS: Total hot run time: 202791 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb0a21af5677fb6da3636c08a7ada358705ed22b, data reload: false
query1	923	390	381	381
query2	6540	2783	2618	2618
query3	6928	197	208	197
query4	19975	17915	17948	17915
query5	19712	6515	6488	6488
query6	273	218	232	218
query7	4155	301	290	290
query8	267	257	221	221
query9	3128	2729	2628	2628
query10	401	300	316	300
query11	11408	10655	10694	10655
query12	120	75	69	69
query13	5575	698	672	672
query14	17915	13215	13243	13215
query15	359	236	230	230
query16	6477	267	257	257
query17	1727	1454	864	864
query18	2335	416	394	394
query19	203	140	144	140
query20	76	76	74	74
query21	197	99	87	87
query22	5416	5224	5155	5155
query23	32519	31981	32006	31981
query24	6906	6485	6463	6463
query25	517	433	414	414
query26	524	162	159	159
query27	1870	295	290	290
query28	6222	2372	2335	2335
query29	2992	2702	2609	2609
query30	243	164	161	161
query31	905	723	721	721
query32	70	62	56	56
query33	386	237	256	237
query34	852	465	484	465
query35	1104	864	905	864
query36	1183	1129	1263	1129
query37	89	69	59	59
query38	3085	2975	2934	2934
query39	1382	1317	1317	1317
query40	194	96	103	96
query41	41	37	35	35
query42	96	92	78	78
query43	804	639	654	639
query44	1140	723	719	719
query45	238	230	227	227
query46	1212	942	934	934
query47	1892	1845	1787	1787
query48	997	710	693	693
query49	612	358	396	358
query50	856	609	597	597
query51	4736	4635	4636	4635
query52	86	85	73	73
query53	443	318	316	316
query54	2625	2446	2445	2445
query55	83	81	87	81
query56	217	188	197	188
query57	1149	1120	1044	1044
query58	215	204	191	191
query59	4302	4079	4187	4079
query60	207	186	213	186
query61	86	81	83	81
query62	821	485	441	441
query63	470	349	337	337
query64	2545	1462	1488	1462
query65	3623	3532	3568	3532
query66	765	378	378	378
query67	17112	15617	15320	15320
query68	8784	650	680	650
query69	565	337	352	337
query70	1623	1397	1370	1370
query71	397	299	315	299
query72	6469	3435	3408	3408
query73	743	318	329	318
query74	6326	5815	5840	5815
query75	5061	3608	3637	3608
query76	5119	1138	1190	1138
query77	852	246	243	243
query78	12577	11731	11884	11731
query79	8049	635	630	630
query80	1092	398	376	376
query81	493	235	224	224
query82	1537	94	94	94
query83	165	139	129	129
query84	254	68	68	68
query85	954	294	287	287
query86	351	291	295	291
query87	3259	3021	3022	3021
query88	5122	2322	2324	2322
query89	379	294	292	292
query90	1882	203	206	203
query91	172	132	146	132
query92	57	57	49	49
query93	5077	557	596	557
query94	728	199	194	194
query95	1102	1056	1061	1056
query96	641	327	328	327
query97	6461	6418	6456	6418
query98	196	173	174	173
query99	2911	1006	817	817
Total cold run time: 311924 ms
Total hot run time: 202791 ms

doris-robot · 2024-04-30T10:37:15Z

ClickBench: Total hot run time: 31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb0a21af5677fb6da3636c08a7ada358705ed22b, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.24	0.05	0.05
query4	1.78	0.07	0.08
query5	0.52	0.52	0.53
query6	1.29	0.62	0.64
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.48	0.49
query10	0.55	0.53	0.53
query11	0.11	0.08	0.09
query12	0.12	0.09	0.10
query13	0.61	0.61	0.61
query14	0.78	0.78	0.78
query15	0.77	0.76	0.76
query16	0.39	0.36	0.36
query17	1.02	1.00	1.00
query18	0.23	0.27	0.23
query19	1.92	1.85	1.83
query20	0.02	0.01	0.01
query21	15.46	0.59	0.56
query22	1.99	2.44	1.73
query23	17.30	0.98	0.95
query24	4.68	1.40	1.16
query25	0.36	0.17	0.04
query26	0.51	0.16	0.16
query27	0.05	0.03	0.04
query28	7.88	0.75	0.76
query29	12.57	2.18	2.31
query30	0.57	0.53	0.51
query31	2.80	0.40	0.38
query32	3.36	0.50	0.50
query33	3.12	3.11	3.06
query34	15.25	4.78	4.77
query35	4.87	4.87	4.85
query36	1.07	1.03	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.25 s
Total hot run time: 31 s

doris-robot · 2024-04-30T10:40:28Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bb0a21af5677fb6da3636c08a7ada358705ed22b with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s

xiaokang · 2024-04-30T12:03:31Z

#34112

…pdate apache#34112 (apache#34357)

Yukang-Lian added 2 commits April 30, 2024 15:33

2
bb0a21a

github-actionsBot added the kind/test label Apr 30, 2024

xiaokang changed the title ~~[Cherry-pick](branch-2.0) Pick #34112~~[Fix](partial update) Fix rowset not found error when doing partial update #34112Apr 30, 2024

xiaokang merged commit 74ed6e8 into apache:branch-2.0Apr 30, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[Fix](partial update) Fix rowset not found error when doing partial u…
5e5a077
…pdate apache#34112 (apache#34357)

Navigation Menu

Uh oh!

[Fix](partial update) Fix rowset not found error when doing partial update #34112 - #34357

[Fix](partial update) Fix rowset not found error when doing partial update #34112#34357
xiaokang merged 2 commits into
apache:branch-2.0from
Yukang-Lian:Pick_34112

Yukang-Lian commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

Yukang-Lian commented Apr 30, 2024

Uh oh!

github-actionsBot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

xiaokang commented Apr 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Yukang-Lian commented Apr 30, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

Yukang-Lian commented Apr 30, 2024

Uh oh!

github-actionsBot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

doris-robot commented Apr 30, 2024

Uh oh!

xiaokang commented Apr 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants