[fix](sequence column) fix update fail on nereids planner by zhannngchen · Pull Request #28031 · apache/doris

zhannngchen · 2023-12-05T14:05:25Z

Proposed changes

Issue Number: close #xxx

if we set enable_fallback_to_original_planner to false, the UPDATE sql in regression case test_unique_table_sequence will fail due to：
```
Table test_uniq_sequence has sequence column, need to specify the sequence column
```
The bug is introduced by [fix](sequence column) insert into should require sequence column in all scenario #27780
fix insert fail in transaction mode, which is introduced by [refactor](Nereids) let insert into compatible with legacy planner #27947
add cases with session variable enable_fallback_to_original_planner false

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

zhannngchen · 2023-12-05T14:38:38Z

run buildall

doris-robot · 2023-12-05T15:22:52Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 028ba1bdb8bc7f0812f613a43694b431b09fae5c, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4707	4447	4437	4437
q2	358	139	144	139
q3	1462	1235	1193	1193
q4	1119	923	866	866
q5	3152	3177	3207	3177
q6	251	129	130	129
q7	990	494	498	494
q8	2242	2241	2222	2222
q9	7613	6789	6648	6648
q10	3226	3268	3281	3268
q11	320	201	210	201
q12	353	219	208	208
q13	4682	3799	3823	3799
q14	243	216	210	210
q15	566	531	534	531
q16	428	374	378	374
q17	1013	595	613	595
q18	7558	7954	7013	7013
q19	1521	1375	1457	1375
q20	548	332	313	313
q21	3116	2653	2726	2653
q22	360	288	291	288
Total cold run time: 45828 ms
Total hot run time: 40133 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4413	4420	4430	4420
q2	273	162	178	162
q3	3534	3518	3521	3518
q4	2397	2382	2379	2379
q5	5742	5755	5737	5737
q6	239	122	119	119
q7	2396	1867	1874	1867
q8	3525	3518	3532	3518
q9	9064	9033	9062	9033
q10	3915	4002	4001	4001
q11	504	376	365	365
q12	768	585	590	585
q13	4336	3555	3599	3555
q14	290	241	263	241
q15	570	522	516	516
q16	500	443	450	443
q17	1864	1816	1873	1816
q18	8605	8175	8163	8163
q19	1737	1751	1746	1746
q20	2265	1945	1944	1944
q21	6520	6159	6142	6142
q22	516	430	417	417
Total cold run time: 63973 ms
Total hot run time: 60687 ms

doris-robot · 2023-12-05T15:25:18Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.85 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17166812892 Bytes

morrySnow · 2023-12-06T02:56:24Z

- if (!haveInputSeqCol && !isPartialUpdate) {
+ if (!haveInputSeqCol && !isPartialUpdate && (
+ boundSink.getDmlCommandType() != DMLCommandType.UPDATE
+ && boundSink.getDmlCommandType() != DMLCommandType.DELETE)) {


could we add a help function in DMLCommandType for easy use?

sounds not necessary? only 4 types

zhannngchen · 2023-12-13T03:55:18Z

run buildall

zhannngchen · 2023-12-13T04:03:02Z

run buildall

github-actions · 2023-12-13T05:17:04Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-12-13T05:17:07Z

PR approved by anyone and no changes requested.

doris-robot · 2023-12-13T06:04:39Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 10e66c33cb70873d2787f6e56bf87fd9c4336380, data reload: false
run tpch-sf100 query with default conf and session variables
q1	51523	31824	31854	31824
q2	383	144	143	143
q3	1502	1251	1194	1194
q4	1103	927	901	901
q5	3183	3247	3297	3247
q6	264	140	140	140
q7	966	495	476	476
q8	2151	2191	2179	2179
q9	7081	6916	6873	6873
q10	3281	3320	3284	3284
q11	336	202	218	202
q12	360	214	209	209
q13	4596	3830	3791	3791
q14	424	326	322	322
q15	769	1310	1329	1310
q16	449	394	390	390
q17	1028	588	555	555
q18	7715	7221	7112	7112
q19	1514	1370	1490	1370
q20	589	294	320	294
q21	3105	2650	2719	2650
q22	350	285	289	285
Total cold run time: 92672 ms
Total hot run time: 68751 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	30793	31862	31807	31807
q2	270	167	176	167
q3	3692	3621	3606	3606
q4	2425	2395	2382	2382
q5	5798	5797	5850	5797
q6	242	132	135	132
q7	3054	3402	3362	3362
q8	3532	3555	3555	3555
q9	9217	9127	9203	9127
q10	3950	4049	4050	4049
q11	802	915	1365	915
q12	780	596	614	596
q13	4263	3618	3589	3589
q14	507	443	434	434
q15	1104	1356	1318	1318
q16	510	449	467	449
q17	1896	1873	1869	1869
q18	8859	10091	8548	8548
q19	1752	1792	1753	1753
q20	2903	2577	2660	2577
q21	6588	6183	6158	6158
q22	490	419	424	419
Total cold run time: 93427 ms
Total hot run time: 92609 ms

zhannngchen · 2023-12-13T08:52:08Z

run clickbench-new

doris-robot · 2023-12-13T09:06:57Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 10e66c33cb70873d2787f6e56bf87fd9c4336380, data reload: false
run tpch-sf100 query with default conf and session variables
q1	50152	31845	31781	31781
q2	389	141	137	137
q3	1566	1291	1182	1182
q4	1120	935	897	897
q5	3185	3463	3172	3172
q6	267	142	141	141
q7	977	489	496	489
q8	2162	2210	2203	2203
q9	6944	6921	6890	6890
q10	3265	3305	3289	3289
q11	346	221	218	218
q12	346	198	205	198
q13	4574	3809	3827	3809
q14	377	325	316	316
q15	815	1311	1325	1311
q16	445	393	394	393
q17	1029	611	545	545
q18	7467	7047	6960	6960
q19	1529	1402	1449	1402
q20	584	306	307	306
q21	3096	2651	2646	2646
q22	356	278	284	278
Total cold run time: 90991 ms
Total hot run time: 68563 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	30775	31820	31807	31807
q2	276	163	173	163
q3	3686	3639	3617	3617
q4	2396	2361	2358	2358
q5	5779	5866	5790	5790
q6	242	131	128	128
q7	3063	3400	3384	3384
q8	3540	3542	3536	3536
q9	9234	9184	9111	9111
q10	3956	4053	4041	4041
q11	817	918	1350	918
q12	805	669	588	588
q13	4282	3560	3576	3560
q14	528	438	480	438
q15	1032	1331	1330	1330
q16	535	439	467	439
q17	1873	1844	1853	1844
q18	8901	8422	8452	8422
q19	1759	1768	1742	1742
q20	2902	2594	2552	2552
q21	6542	6186	6133	6133
q22	506	413	413	413
Total cold run time: 93429 ms
Total hot run time: 92314 ms

dataroaring

LGTM

doris-robot · 2023-12-13T11:01:28Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.13 seconds
stream load tsv: 590 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17219963917 Bytes

doris-robot · 2023-12-13T11:16:56Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 10e66c33cb70873d2787f6e56bf87fd9c4336380, data reload: false
run tpch-sf100 query with default conf and session variables
q1	52651	31865	31826	31826
q2	369	138	141	138
q3	1576	1244	1234	1234
q4	1114	933	918	918
q5	3194	3324	3163	3163
q6	265	149	142	142
q7	979	492	486	486
q8	2152	2207	2178	2178
q9	6909	6919	6921	6919
q10	3267	3321	3308	3308
q11	343	223	206	206
q12	349	207	211	207
q13	4849	3800	3824	3800
q14	438	342	334	334
q15	756	1273	1341	1273
q16	442	390	384	384
q17	1020	588	567	567
q18	7611	6972	7950	6972
q19	1512	1323	1474	1323
q20	555	305	342	305
q21	3068	2659	2672	2659
q22	347	281	284	281
Total cold run time: 93766 ms
Total hot run time: 68623 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	30845	31933	31825	31825
q2	286	168	179	168
q3	4662	3678	3636	3636
q4	2405	2392	2403	2392
q5	6018	6083	5835	5835
q6	243	139	132	132
q7	3050	3425	3353	3353
q8	3545	3531	3546	3531
q9	9309	9196	9166	9166
q10	4007	4086	4052	4052
q11	844	896	1347	896
q12	788	599	610	599
q13	4306	3594	3593	3593
q14	514	471	447	447
q15	1085	1332	1323	1323
q16	528	453	458	453
q17	1919	1880	1825	1825
q18	8897	8297	8370	8297
q19	1728	1779	1771	1771
q20	2931	2599	2693	2599
q21	6572	6196	6155	6155
q22	522	430	424	424
Total cold run time: 95004 ms
Total hot run time: 92472 ms

1. if we set enable_fallback_to_original_planner to false, the UPDATE sql in regression case test_unique_table_sequence will fail due to：“Table test_uniq_sequence has sequence column, need to specify the sequence column”，The bug is introduced by [fix](sequence column) insert into should require sequence column in all scenario apache#27780 2. fix insert fail in transaction mode, which is introduced by [refactor](Nereids) let insert into compatible with legacy planner apache#27947 3. add cases with session variable enable_fallback_to_original_planner false

pr #28105 and #28031 merging conflict caused this case to fail.

pr apache#28105 and apache#28031 merging conflict caused this case to fail.

1. if we set enable_fallback_to_original_planner to false, the UPDATE sql in regression case test_unique_table_sequence will fail due to：“Table test_uniq_sequence has sequence column, need to specify the sequence column”，The bug is introduced by [fix](sequence column) insert into should require sequence column in all scenario apache#27780 2. fix insert fail in transaction mode, which is introduced by [refactor](Nereids) let insert into compatible with legacy planner apache#27947 3. add cases with session variable enable_fallback_to_original_planner false

pr apache#28105 and apache#28031 merging conflict caused this case to fail.

1. if we set enable_fallback_to_original_planner to false, the UPDATE sql in regression case test_unique_table_sequence will fail due to：“Table test_uniq_sequence has sequence column, need to specify the sequence column”，The bug is introduced by [fix](sequence column) insert into should require sequence column in all scenario apache#27780 2. fix insert fail in transaction mode, which is introduced by [refactor](Nereids) let insert into compatible with legacy planner apache#27947 3. add cases with session variable enable_fallback_to_original_planner false

pr apache#28105 and apache#28031 merging conflict caused this case to fail.

zhannngchen added the not-merge/2.0 do not merge into 2.0 branch label Dec 5, 2023

zhannngchen force-pushed the fix-nerieds-update-seq branch 2 times, most recently from 3a558bb to 028ba1bCompare December 5, 2023 14:33

morrySnow reviewed Dec 6, 2023
View reviewed changes

zhannngchen added 2 commits December 12, 2023 15:44

[fix](sequence column) fix update fail on nereids planner
f76f3dd

address comments
3a7963a

zhannngchen force-pushed the fix-nerieds-update-seq branch from 028ba1b to 3a7963aCompare December 12, 2023 09:38

zhannngchen added 4 commits December 12, 2023 18:11

update cases, test enable_fallback_to_original_planner
67f8887

fix compile
fcf354e

update cases
2c872c4

update cases
71a82eb

remove debug log
10e66c3

zhannngchen requested a review from morrySnow December 13, 2023 04:18

morrySnow approved these changes Dec 13, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 13, 2023

github-actionsBot added the reviewed label Dec 13, 2023

dataroaring approved these changes Dec 13, 2023
View reviewed changes

zhannngchen merged commit 0868526 into apache:masterDec 13, 2023

liaoxin01 mentioned this pull request Dec 15, 2023
[fix](test) fix case of test_unique_table_new_sequence #28442
Merged

zhannngchen pushed a commit that referenced this pull request Dec 15, 2023
[fix](test) fix case of test_unique_table_new_sequence (#28442)
088bb80
pr #28105 and #28031 merging conflict caused this case to fail.

hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Dec 28, 2023
[fix](test) fix case of test_unique_table_new_sequence (apache#28442)
4f14b86
pr apache#28105 and apache#28031 merging conflict caused this case to fail.

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
[fix](test) fix case of test_unique_table_new_sequence (apache#28442)
23072e7
pr apache#28105 and apache#28031 merging conflict caused this case to fail.

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](test) fix case of test_unique_table_new_sequence (apache#28442)
54c3f06
pr apache#28105 and apache#28031 merging conflict caused this case to fail.

Uh oh!

Conversation

zhannngchen commented Dec 5, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

zhannngchen commented Dec 5, 2023

Uh oh!

doris-robot commented Dec 5, 2023

Uh oh!

doris-robot commented Dec 5, 2023

Uh oh!

morrySnowDec 6, 2023

Choose a reason for hiding this comment

Uh oh!

zhannngchenDec 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhannngchen commented Dec 13, 2023

Uh oh!

zhannngchen commented Dec 13, 2023

Uh oh!

github-actionsBot commented Dec 13, 2023

Uh oh!

github-actionsBot commented Dec 13, 2023

Uh oh!

doris-robot commented Dec 13, 2023

Uh oh!

zhannngchen commented Dec 13, 2023

Uh oh!

doris-robot commented Dec 13, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Dec 13, 2023

Uh oh!

doris-robot commented Dec 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhannngchen commented Dec 5, 2023•
edited
Loading