[fix](maxCompute)Fixed the bug that extra nulls are read when reading maxcompute. by hubgeter · Pull Request #40888 · apache/doris

hubgeter · 2024-09-17T01:08:43Z

before pr #40225

Proposed changes

Fixed a bug where when reading maxcompute, if there are null values in a batch, null values will always be read out.

… maxcompute.

doris-robot · 2024-09-17T01:08:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hubgeter · 2024-09-17T01:11:22Z

run buildall

doris-robot · 2024-09-17T01:49:22Z

TPC-H: Total hot run time: 41109 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d63eba5b8d9b47277c7124cc31b19a4f316dbea1, data reload: false
------ Round 1 ----------------------------------
q1	17569	7248	7292	7248
q2	2050	163	154	154
q3	10620	1088	1193	1088
q4	10567	734	703	703
q5	7756	3057	3060	3057
q6	234	153	145	145
q7	1004	608	588	588
q8	9416	2027	2044	2027
q9	6796	6400	6373	6373
q10	7000	2244	2237	2237
q11	447	245	243	243
q12	407	213	218	213
q13	17785	2970	2955	2955
q14	237	206	214	206
q15	547	523	533	523
q16	672	612	614	612
q17	969	825	827	825
q18	7338	6676	6566	6566
q19	1403	1046	1022	1022
q20	586	287	284	284
q21	3953	3282	3057	3057
q22	1145	1025	983	983
Total cold run time: 108501 ms
Total hot run time: 41109 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7230	7201	7177	7177
q2	329	229	219	219
q3	2910	2957	2965	2957
q4	2002	1788	1832	1788
q5	5560	5579	5589	5579
q6	230	142	144	142
q7	2211	1804	1812	1804
q8	3286	3423	3403	3403
q9	8663	8813	8711	8711
q10	3568	3401	3474	3401
q11	578	474	485	474
q12	812	619	587	587
q13	10166	3105	3133	3105
q14	295	260	288	260
q15	581	536	519	519
q16	718	658	679	658
q17	1805	1573	1555	1555
q18	8184	7750	7773	7750
q19	1703	1562	1458	1458
q20	2110	1859	1887	1859
q21	5461	5245	5420	5245
q22	1156	1065	1049	1049
Total cold run time: 69558 ms
Total hot run time: 59700 ms

doris-robot · 2024-09-17T02:01:49Z

TPC-DS: Total hot run time: 197868 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d63eba5b8d9b47277c7124cc31b19a4f316dbea1, data reload: false
query1	1265	854	892	854
query2	6397	2112	2065	2065
query3	10820	3972	3998	3972
query4	64905	28795	23482	23482
query5	4957	476	459	459
query6	415	171	173	171
query7	5451	298	294	294
query8	313	219	222	219
query9	8571	2655	2619	2619
query10	441	274	287	274
query11	17052	15253	15701	15253
query12	163	102	103	102
query13	1451	438	424	424
query14	10732	7533	6703	6703
query15	205	174	185	174
query16	6779	480	470	470
query17	1125	605	567	567
query18	1411	290	287	287
query19	197	147	146	146
query20	135	116	108	108
query21	211	101	106	101
query22	4800	4649	4789	4649
query23	34724	33726	33708	33708
query24	6024	2860	2856	2856
query25	500	386	376	376
query26	635	153	153	153
query27	1671	279	281	279
query28	4254	2430	2403	2403
query29	649	400	422	400
query30	225	153	153	153
query31	950	761	785	761
query32	67	50	48	48
query33	439	284	276	276
query34	904	489	472	472
query35	859	750	722	722
query36	1060	925	932	925
query37	153	83	83	83
query38	4033	3902	3890	3890
query39	1452	1401	1378	1378
query40	201	95	92	92
query41	46	43	42	42
query42	117	96	94	94
query43	518	482	501	482
query44	1130	791	783	783
query45	190	159	160	159
query46	1107	779	756	756
query47	1869	1791	1842	1791
query48	461	358	364	358
query49	698	396	394	394
query50	838	410	406	406
query51	6970	6946	6897	6897
query52	100	85	84	84
query53	244	170	176	170
query54	560	444	446	444
query55	80	72	71	71
query56	280	250	257	250
query57	1199	1075	1081	1075
query58	223	220	263	220
query59	3169	2888	2934	2888
query60	288	260	259	259
query61	101	103	98	98
query62	757	649	653	649
query63	218	192	183	183
query64	1326	682	637	637
query65	3216	3162	3237	3162
query66	673	296	309	296
query67	16014	15708	15512	15512
query68	1230	543	558	543
query69	422	278	281	278
query70	1211	1136	1129	1129
query71	324	269	265	265
query72	4688	3969	3954	3954
query73	748	326	336	326
query74	9148	9051	8985	8985
query75	3319	2624	2632	2624
query76	1270	865	885	865
query77	384	296	299	296
query78	10385	9416	9302	9302
query79	1068	886	857	857
query80	815	567	591	567
query81	510	261	257	257
query82	1173	229	231	229
query83	240	159	163	159
query84	288	112	111	111
query85	807	423	410	410
query86	323	320	291	291
query87	4440	4354	4201	4201
query88	4346	4147	4054	4054
query89	379	358	357	357
query90	1735	303	301	301
query91	165	159	167	159
query92	77	72	71	71
query93	921	881	882	881
query94	689	374	375	374
query95	440	398	398	398
query96	477	483	482	482
query97	3193	3112	3153	3112
query98	230	230	225	225
query99	1464	1296	1277	1277
Total cold run time: 305897 ms
Total hot run time: 197868 ms

doris-robot · 2024-09-17T02:07:16Z

ClickBench: Total hot run time: 32.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d63eba5b8d9b47277c7124cc31b19a4f316dbea1, data reload: false
query1	0.05	0.04	0.05
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.51	0.50	0.49
query6	1.13	0.73	0.72
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.55	0.53	0.54
query11	0.14	0.11	0.10
query12	0.13	0.11	0.10
query13	0.60	0.59	0.60
query14	2.99	2.97	2.91
query15	0.87	0.80	0.82
query16	0.38	0.40	0.38
query17	1.06	1.00	1.05
query18	0.19	0.19	0.18
query19	1.86	1.83	2.01
query20	0.01	0.01	0.00
query21	15.35	0.60	0.60
query22	2.78	2.52	1.92
query23	17.17	0.75	0.94
query24	2.94	1.60	1.32
query25	0.26	0.13	0.14
query26	0.41	0.13	0.13
query27	0.04	0.03	0.04
query28	9.94	1.10	1.06
query29	12.58	3.25	3.20
query30	0.24	0.05	0.06
query31	2.87	0.38	0.39
query32	3.28	0.46	0.46
query33	2.99	3.02	3.05
query34	16.92	4.37	4.47
query35	4.49	4.44	4.36
query36	0.65	0.47	0.48
query37	0.09	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 106.39 s
Total hot run time: 32.92 s

hubgeter · 2024-09-17T03:38:44Z

run buildall

doris-robot · 2024-09-17T04:16:54Z

TPC-H: Total hot run time: 41531 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66ff97b628703bdcd27253021b749d9039c34d36, data reload: false
------ Round 1 ----------------------------------
q1	17570	7389	7238	7238
q2	2046	155	163	155
q3	10863	1113	1196	1113
q4	10523	752	717	717
q5	7765	3091	3045	3045
q6	237	152	150	150
q7	1002	623	606	606
q8	9425	2058	2034	2034
q9	6863	6422	6380	6380
q10	7053	2283	2288	2283
q11	444	249	242	242
q12	407	217	212	212
q13	17797	2967	2965	2965
q14	227	212	210	210
q15	574	517	513	513
q16	682	611	605	605
q17	982	826	793	793
q18	7284	6692	6658	6658
q19	1421	1077	1129	1077
q20	577	282	284	282
q21	3960	3305	3250	3250
q22	1136	1003	1014	1003
Total cold run time: 108838 ms
Total hot run time: 41531 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7265	7252	7247	7247
q2	329	232	226	226
q3	3053	2977	2925	2925
q4	2032	1840	1835	1835
q5	5624	5529	5641	5529
q6	233	145	145	145
q7	2206	1815	1822	1815
q8	3306	3443	3394	3394
q9	8751	8884	8759	8759
q10	3430	3468	3467	3467
q11	580	478	502	478
q12	815	626	612	612
q13	9925	3187	3133	3133
q14	315	293	269	269
q15	581	536	503	503
q16	725	685	685	685
q17	1787	1589	1628	1589
q18	8174	7855	7845	7845
q19	1747	1674	1447	1447
q20	2136	1889	1860	1860
q21	5672	5355	5466	5355
q22	1160	1066	1032	1032
Total cold run time: 69846 ms
Total hot run time: 60150 ms

doris-robot · 2024-09-17T04:29:23Z

TPC-DS: Total hot run time: 199701 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66ff97b628703bdcd27253021b749d9039c34d36, data reload: false
query1	1284	870	855	855
query2	6381	2117	2023	2023
query3	10790	3914	3738	3738
query4	62453	30376	23461	23461
query5	5159	489	476	476
query6	410	171	171	171
query7	5481	298	296	296
query8	314	227	221	221
query9	8469	2659	2628	2628
query10	454	277	297	277
query11	17769	15137	15738	15137
query12	170	104	103	103
query13	1466	420	413	413
query14	9737	7521	7748	7521
query15	203	180	178	178
query16	6740	505	460	460
query17	1151	641	609	609
query18	1543	323	300	300
query19	213	157	152	152
query20	125	106	110	106
query21	209	101	102	101
query22	4632	4633	4673	4633
query23	34642	34145	34447	34145
query24	6017	2965	2895	2895
query25	506	416	406	406
query26	629	161	163	161
query27	1601	282	283	282
query28	3839	2445	2416	2416
query29	671	425	423	423
query30	236	162	163	162
query31	956	781	782	781
query32	73	52	54	52
query33	415	303	299	299
query34	899	489	488	488
query35	854	768	736	736
query36	1088	950	944	944
query37	151	86	89	86
query38	4062	3956	3881	3881
query39	1465	1406	1388	1388
query40	212	97	97	97
query41	50	47	50	47
query42	118	102	98	98
query43	518	502	478	478
query44	1187	815	794	794
query45	193	165	169	165
query46	1133	748	747	747
query47	1874	1883	1820	1820
query48	479	352	357	352
query49	691	412	382	382
query50	836	408	403	403
query51	6882	6883	6982	6883
query52	96	87	88	87
query53	248	182	179	179
query54	568	463	455	455
query55	75	79	73	73
query56	275	265	258	258
query57	1194	1088	1081	1081
query58	221	253	236	236
query59	3091	2933	3157	2933
query60	290	272	280	272
query61	142	104	98	98
query62	755	663	677	663
query63	226	196	201	196
query64	1419	678	683	678
query65	3347	3221	3279	3221
query66	695	309	308	308
query67	16233	15521	15486	15486
query68	1329	856	848	848
query69	471	346	352	346
query70	1180	1161	1093	1093
query71	335	336	330	330
query72	6242	3405	3432	3405
query73	599	583	572	572
query74	9257	8978	8994	8978
query75	3037	3115	2871	2871
query76	1149	875	852	852
query77	387	376	374	374
query78	9475	9271	9292	9271
query79	927	902	870	870
query80	603	599	626	599
query81	461	253	254	253
query82	228	231	229	229
query83	155	155	157	155
query84	265	94	101	94
query85	704	423	365	365
query86	313	333	324	324
query87	4471	4266	4350	4266
query88	4497	4042	4024	4024
query89	368	358	363	358
query90	1432	314	311	311
query91	165	165	163	163
query92	77	75	73	73
query93	897	891	886	886
query94	560	359	362	359
query95	427	420	409	409
query96	486	483	483	483
query97	3140	3130	3116	3116
query98	223	227	226	226
query99	1425	1304	1306	1304
Total cold run time: 301590 ms
Total hot run time: 199701 ms

doris-robot · 2024-09-17T04:34:49Z

ClickBench: Total hot run time: 32.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66ff97b628703bdcd27253021b749d9039c34d36, data reload: false
query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.24	0.06	0.07
query4	1.66	0.10	0.10
query5	0.50	0.51	0.50
query6	1.14	0.73	0.71
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.51	0.48
query10	0.55	0.54	0.54
query11	0.14	0.11	0.11
query12	0.13	0.10	0.10
query13	0.60	0.60	0.59
query14	3.06	3.00	3.01
query15	0.88	0.81	0.82
query16	0.39	0.38	0.37
query17	1.02	1.07	1.03
query18	0.19	0.19	0.19
query19	1.86	1.84	1.98
query20	0.01	0.01	0.01
query21	15.35	0.62	0.62
query22	2.96	2.52	1.57
query23	17.10	0.94	0.91
query24	3.23	1.85	1.05
query25	0.31	0.18	0.07
query26	0.44	0.14	0.14
query27	0.03	0.04	0.04
query28	9.87	1.09	1.06
query29	12.61	3.21	3.18
query30	0.25	0.06	0.05
query31	2.88	0.37	0.38
query32	3.27	0.47	0.46
query33	2.98	3.05	3.02
query34	17.01	4.38	4.39
query35	4.40	4.46	4.44
query36	0.66	0.48	0.48
query37	0.08	0.06	0.05
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.9 s
Total hot run time: 32.68 s

morningman

LGTM

github-actions · 2024-09-18T06:37:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-18T06:37:50Z

PR approved by anyone and no changes requested.

… maxcompute. (apache#40888) before pr apache#40225 ## Proposed changes Fixed a bug where when reading maxcompute, if there are null values in a batch, null values will always be read out.

… API.(#40225 , #40888 ,#41386 ) (#41614) bp #40225 , #40888 ,#41386

… API.(#40225 , #40888 ,#41386 ) (#41610) bp #40225 , #40888 ,#41386 ## Proposed changes Among them, #40225 is the new api of mc, #40888 is used to fix the bug when reading null between the new and old apis, #41386 is used for compatibility between the new and old versions

… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)" This reverts commit 8c0f73c.

… maxcompute. (apache#40888) before pr apache#40225 ## Proposed changes Fixed a bug where when reading maxcompute, if there are null values in a batch, null values will always be read out.

[fix](maxCompute)Fixed the bug that extra nulls are read when reading…
d63eba5
… maxcompute.

rm list databases
66ff97b

morningman approved these changes Sep 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2024

github-actionsBot added the reviewed label Sep 18, 2024

zy-kkk approved these changes Sep 18, 2024
View reviewed changes

suxiaogang223 approved these changes Sep 18, 2024
View reviewed changes

morningman merged commit 64463f5 into apache:masterSep 18, 2024

morningman added the dev/2.1.x-experimental label Sep 18, 2024

hubgeter mentioned this pull request Oct 9, 2024
[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.(#40225 , #40888 ,#41386 ) #41610
Merged

hubgeter mentioned this pull request Oct 9, 2024
[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.(#40225 , #40888 ,#41386 ) #41614
Merged

morningman pushed a commit that referenced this pull request Oct 11, 2024
[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage…
a1f8383
… API.(#40225 , #40888 ,#41386 ) (#41614) bp #40225 , #40888 ,#41386

morningman added the dev/3.0.3-merged label Oct 11, 2024

morningman removed the dev/2.1.x-experimental label Oct 11, 2024

morningman added the dev/2.1.7-merged label Oct 11, 2024

hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 20, 2024
Revert "[Enhancement](MaxCompute)Refactoring maxCompute catalog using…
edab727
… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)" This reverts commit 8c0f73c.

hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 20, 2024
Revert "[Enhancement](MaxCompute)Refactoring maxCompute catalog using…
b551c95
… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)" This reverts commit 8c0f73c.

Navigation Menu

Uh oh!

[fix](maxCompute)Fixed the bug that extra nulls are read when reading maxcompute. - #40888

[fix](maxCompute)Fixed the bug that extra nulls are read when reading maxcompute.#40888
morningman merged 2 commits into
apache:masterfrom
hubgeter:fix_mc_null

hubgeter commented Sep 17, 2024•
edited
Loading

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

hubgeter commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

hubgeter commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

morningman left a comment

Uh oh!

github-actionsBot commented Sep 18, 2024

Uh oh!

github-actionsBot commented Sep 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

hubgeter commented Sep 17, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

hubgeter commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

hubgeter commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

doris-robot commented Sep 17, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 18, 2024

Uh oh!

github-actionsBot commented Sep 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hubgeter commented Sep 17, 2024•
edited
Loading