Skip to content

[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) - #40538

Merged
Jibing-Li merged 1 commit into
apache:branch-2.0from
Jibing-Li:row2.0
Sep 9, 2024
Merged

[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457)#40538
Jibing-Li merged 1 commit into
apache:branch-2.0from
Jibing-Li:row2.0

Conversation

@Jibing-Li

@Jibing-LiJibing-Li commented Sep 9, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

backport: 40457
reverted

@doris-robot

doris-robot commented Sep 9, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actionsgithub-actionsBot added area/planner Issues or PRs related to the query planner kind/test labels Sep 9, 2024
@Jibing-Li
Jibing-Li marked this pull request as ready for review September 9, 2024 06:45
@Jibing-Li

Jibing-Li commented Sep 9, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Sep 9, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Jibing-Li
[improvement](statistics)Return -1 to neredis if report olap table ro…
29bbe33
…w count for new table is not done for all tablets. (apache#40457)
Return -1 to neredis if report olap table row count for new table is not
done for all tablets.
After this change, nereids could know new table is empty or not. When
it's not empty but not reported yet, return -1 as row count to nereids.
@Jibing-Li

Jibing-Li commented Sep 9, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Sep 9, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Sep 9, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49473 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29bbe3357bf7791d5c282eb35e4bb50b0b80f5bf, data reload: false
------ Round 1 ----------------------------------
q1	5112	4402	4331	4331
q2	399	160	192	160
q3	2012	1956	1910	1910
q4	1411	1309	1310	1309
q5	4000	3939	3970	3939
q6	256	137	132	132
q7	2014	1604	1619	1604
q8	2752	2759	2762	2759
q9	10284	9905	9840	9840
q10	3522	3528	3515	3515
q11	369	253	237	237
q12	456	299	297	297
q13	7315	4019	4052	4019
q14	348	327	333	327
q15	512	465	462	462
q16	511	476	465	465
q17	1103	956	948	948
q18	7344	6956	6901	6901
q19	1630	1560	1511	1511
q20	511	324	313	313
q21	4433	4162	4098	4098
q22	503	399	396	396
Total cold run time: 56797 ms
Total hot run time: 49473 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4481	4305	4269	4269
q2	323	227	223	223
q3	4202	4143	4140	4140
q4	2763	2765	2750	2750
q5	7220	7143	7081	7081
q6	236	117	118	117
q7	3268	2843	2814	2814
q8	4332	4435	4497	4435
q9	13658	13544	13623	13544
q10	4285	4277	4272	4272
q11	744	696	714	696
q12	1011	852	839	839
q13	4173	3782	3765	3765
q14	439	420	433	420
q15	513	460	461	460
q16	636	595	591	591
q17	3773	3846	3834	3834
q18	8762	8783	8799	8783
q19	1698	1656	1696	1656
q20	2388	2149	2131	2131
q21	8461	8472	8396	8396
q22	1050	940	913	913
Total cold run time: 78416 ms
Total hot run time: 76129 ms

@doris-robot

doris-robot commented Sep 9, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 212861 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29bbe3357bf7791d5c282eb35e4bb50b0b80f5bf, data reload: false
query1	926	388	410	388
query2	6548	2376	1953	1953
query3	6918	204	202	202
query4	23198	21654	21837	21654
query5	19762	6508	6519	6508
query6	281	227	232	227
query7	4332	301	319	301
query8	279	281	266	266
query9	3152	2676	2602	2602
query10	468	302	304	302
query11	15894	15082	15045	15045
query12	126	74	74	74
query13	1036	437	437	437
query14	17184	13524	13400	13400
query15	391	216	240	216
query16	6493	281	262	262
query17	1794	956	909	909
query18	905	312	317	312
query19	213	154	150	150
query20	87	76	79	76
query21	194	98	100	98
query22	5191	4968	4862	4862
query23	34223	33710	33513	33513
query24	7699	6372	6366	6366
query25	519	417	429	417
query26	1269	167	162	162
query27	2314	295	296	295
query28	6045	2278	2251	2251
query29	2892	2782	2789	2782
query30	240	169	169	169
query31	980	783	724	724
query32	71	56	55	55
query33	463	262	261	261
query34	893	480	466	466
query35	1128	938	913	913
query36	1439	1211	1224	1211
query37	149	59	61	59
query38	3054	2931	2975	2931
query39	1388	1330	1342	1330
query40	295	99	95	95
query41	38	38	36	36
query42	85	91	92	91
query43	699	587	621	587
query44	1174	723	728	723
query45	243	234	228	228
query46	1234	967	984	967
query47	1902	1844	1765	1765
query48	521	427	421	421
query49	660	380	386	380
query50	855	604	635	604
query51	4753	4696	4683	4683
query52	93	83	80	80
query53	237	189	187	187
query54	2651	2483	2463	2463
query55	97	85	97	85
query56	224	200	217	200
query57	1444	1189	1104	1104
query58	235	209	215	209
query59	3654	3160	3262	3160
query60	217	221	197	197
query61	98	99	95	95
query62	764	444	467	444
query63	204	176	180	176
query64	3533	1554	1436	1436
query65	3623	3555	3576	3555
query66	765	416	401	401
query67	16867	17547	15788	15788
query68	9078	666	644	644
query69	491	257	260	257
query70	1635	1416	1332	1332
query71	389	308	304	304
query72	6850	4908	4871	4871
query73	763	328	318	318
query74	6308	5860	5837	5837
query75	4674	3726	3612	3612
query76	4769	1154	1189	1154
query77	652	259	270	259
query78	12555	11435	11679	11435
query79	9229	629	634	629
query80	2559	384	385	384
query81	504	242	236	236
query82	1674	120	96	96
query83	174	130	132	130
query84	259	69	69	69
query85	1276	312	322	312
query86	354	289	273	273
query87	3218	3037	3009	3009
query88	4843	2334	2309	2309
query89	487	280	289	280
query90	1779	217	208	208
query91	156	128	126	126
query92	58	45	50	45
query93	6726	563	575	563
query94	846	213	207	207
query95	1990	1934	1912	1912
query96	652	321	319	319
query97	6467	6368	6428	6368
query98	214	219	201	201
query99	2935	930	853	853
Total cold run time: 321867 ms
Total hot run time: 212861 ms

@doris-robot

doris-robot commented Sep 9, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 29bbe3357bf7791d5c282eb35e4bb50b0b80f5bf, data reload: false
query1	0.02	0.01	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.79	0.06	0.06
query5	0.52	0.52	0.52
query6	1.23	0.60	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.51	0.50	0.48
query10	0.54	0.54	0.54
query11	0.12	0.08	0.09
query12	0.12	0.09	0.08
query13	0.61	0.61	0.61
query14	0.78	0.78	0.78
query15	0.77	0.76	0.77
query16	0.39	0.36	0.39
query17	0.98	1.04	1.02
query18	0.22	0.26	0.24
query19	1.92	1.82	1.80
query20	0.01	0.01	0.01
query21	15.48	0.55	0.54
query22	1.95	1.99	1.44
query23	17.24	0.91	0.81
query24	7.62	1.26	0.67
query25	0.38	0.06	0.05
query26	0.89	0.16	0.16
query27	0.05	0.03	0.04
query28	5.65	0.75	0.72
query29	12.80	2.35	2.22
query30	0.56	0.50	0.52
query31	2.80	0.40	0.38
query32	3.35	0.49	0.48
query33	3.08	3.10	3.04
query34	15.27	4.82	4.80
query35	4.87	4.83	4.86
query36	1.05	1.02	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.17	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.33 s
Total hot run time: 29.99 s

@doris-robot

doris-robot commented Sep 9, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 29bbe3357bf7791d5c282eb35e4bb50b0b80f5bf with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s

@Jibing-Li

Jibing-Li commented Sep 9, 2024

Copy link
Copy Markdown
ContributorAuthor

run p0

@doris-robot

doris-robot commented Sep 9, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.85% (8137/21499)
Line Coverage: 29.59% (66981/226360)
Region Coverage: 29.08% (34562/118851)
Branch Coverage: 24.99% (17809/71272)
Coverage Report: http://coverage.selectdb-in.cc/coverage/29bbe3357bf7791d5c282eb35e4bb50b0b80f5bf_29bbe3357bf7791d5c282eb35e4bb50b0b80f5bf/report/index.html

@Jibing-Li
Jibing-Li merged commit f5f74dd into apache:branch-2.0Sep 9, 2024
@hello-stephenhello-stephen mentioned this pull request Sep 10, 2024
Merged
hello-stephen added a commit that referenced this pull request Sep 10, 2024
@hello-stephen
Revert "[improvement](statistics)Return -1 to neredis if report olap …
28bceac
…table row count for new table is not done for all tablets. (#40457)" (#40617)
Reverts #40538
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/plannerIssues or PRs related to the query plannerkind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Jibing-Li@doris-robot