Skip to content

[fix](statistics) do not enter empty-stats fast path when any partition has data - #67542

Open
xy720 wants to merge 3 commits into
apache:masterfrom
xy720:fix-analyze-rowcount-0
Open

[fix](statistics) do not enter empty-stats fast path when any partition has data#67542
xy720 wants to merge 3 commits into
apache:masterfrom
xy720:fix-analyze-rowcount-0

Conversation

@xy720

@xy720xy720 commented Sep 5, 2026•
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Problem Summary:

在存算分离架构中， 表/物化视图第一次写入/刷新完成后，在CloudTabletStatMgr 尚未从 MetaService 拉到最新 tablet stats 的窗口内，可能会出现rowCountReported = true 但是 OlapAnalysisTask 拿到了 rowCount=0 的过期快照，走进短路分支，直接把 count=0/ndv=0/min/max=null 的空 col stats 写进__internal_schema.column_statistics，污染 CBO 直到下一次成功 analyze 覆盖。

例子：
T1 : 创建新物化视图 mv_a
T2: 手动触发 mv_a 刷新并完成
T3: 触发 ANALYZE，此时 CloudTabletStatMgr 尚未拉取最新 rowcount，isEmptyTable 判空，写入空 col stats
T4: 基于 mv_a 的下游 mv_b 开始刷新，上游 mv 的空 col stats 使估算 cardinality=1，join order 选错，产出 BROADCAST 大表 plan，MEM_LIMIT_EXCEEDED
T5: 手动重跑 analyze 覆盖空 col stats 才恢复

修复：
仅当所有 partition 的 visibleVersion 仍为 PARTITION_INIT_VERSION（真正从未 load 过）才允许写空 stats

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xy720

xy720 commented Sep 5, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Sep 5, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16564 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24cb252499ce0fc3ff9cd0b2f50531b1aa626793, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17589	2990	2992	2990
q2	2115	260	242	242
q3	10199	860	511	511
q4	4671	243	203	203
q5	7682	563	382	382
q6	137	112	91	91
q7	524	493	391	391
q8	9246	944	905	905
q9	3467	2388	2394	2388
q10	6489	813	721	721
q11	389	205	181	181
q12	615	257	206	206
q13	18157	1525	1162	1162
q14	154	146	136	136
q15	q16	435	392	368	368
q17	1338	904	737	737
q18	3103	2267	2227	2227
q19	1265	932	730	730
q20	364	291	203	203
q21	5571	1562	1845	1562
q22	333	265	228	228
Total cold run time: 93843 ms
Total hot run time: 16564 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3361	3297	3290	3290
q2	514	371	370	370
q3	2204	2758	2224	2224
q4	1186	1181	880	880
q5	2189	2138	2112	2112
q6	168	126	86	86
q7	1024	912	846	846
q8	1575	1394	1397	1394
q9	3130	3120	3074	3074
q10	1876	1791	1635	1635
q11	351	268	249	249
q12	462	426	336	336
q13	1486	1538	1143	1143
q14	189	170	158	158
q15	q16	388	404	352	352
q17	3548	3393	3202	3202
q18	4812	4426	4750	4426
q19	852	815	920	815
q20	1013	968	834	834
q21	3738	3041	3182	3041
q22	400	343	308	308
Total cold run time: 34466 ms
Total hot run time: 30775 ms

@hello-stephen

hello-stephen commented Sep 5, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81552 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24cb252499ce0fc3ff9cd0b2f50531b1aa626793, data reload: false
query5	4266	415	332	332
query6	384	134	124	124
query7	4953	423	237	237
query8	291	120	115	115
query9	8693	2882	2885	2882
query10	390	205	185	185
query11	5380	1033	899	899
query12	115	69	71	69
query13	1183	469	331	331
query14	6077	2175	2047	2047
query14_1	1948	1936	1928	1928
query15	175	119	111	111
query16	925	369	336	336
query17	809	453	382	382
query18	2336	322	224	224
query19	179	135	104	104
query20	69	68	69	68
query21	206	98	82	82
query22	5314	5334	5214	5214
query23	6691	6146	6002	6002
query23_1	6004	5941	5988	5941
query24	7254	1054	784	784
query24_1	762	782	785	782
query25	411	280	227	227
query26	1239	227	127	127
query27	2797	421	236	236
query28	4690	1513	1490	1490
query29	907	420	333	333
query30	254	151	128	128
query31	838	399	332	332
query32	202	72	66	66
query33	446	208	178	178
query34	989	844	471	471
query35	406	404	331	331
query36	582	557	555	555
query37	123	80	66	66
query38	1003	845	815	815
query39	492	494	492	492
query39_1	474	449	470	449
query40	200	90	74	74
query41	53	51	51	51
query42	71	70	71	70
query43	235	235	207	207
query44	1013	548	556	548
query45	111	110	101	101
query46	737	820	536	536
query47	742	755	728	728
query48	321	298	229	229
query49	532	235	180	180
query50	783	264	187	187
query51	7997	7927	8011	7927
query52	67	68	60	60
query53	197	208	158	158
query54	234	192	175	175
query55	76	66	59	59
query56	204	176	191	176
query57	680	641	671	641
query58	211	190	220	190
query59	1214	1219	1108	1108
query60	266	177	182	177
query61	144	138	134	134
query62	429	205	174	174
query63	179	147	141	141
query64	2907	765	681	681
query65	1583	1627	1598	1598
query66	2055	270	210	210
query67	9956	9661	12159	9661
query68	2831	1258	715	715
query69	351	214	199	199
query70	651	597	629	597
query71	253	168	165	165
query72	2339	1699	1510	1510
query73	662	554	336	336
query74	1565	1225	1142	1142
query75	1160	1097	949	949
query76	2285	730	556	556
query77	250	244	212	212
query78	3969	3587	3208	3208
query79	2864	864	591	591
query80	1586	342	280	280
query81	494	150	132	132
query82	683	117	96	96
query83	280	211	202	202
query84	300	113	90	90
query85	813	349	292	292
query86	412	180	158	158
query87	1015	958	884	884
query88	2925	2136	2111	2111
query89	292	189	172	172
query90	1933	131	130	130
query91	133	118	100	100
query92	81	62	70	62
query93	1686	1111	676	676
query94	642	256	206	206
query95	516	319	218	218
query96	816	557	271	271
query97	1055	1084	1004	1004
query98	167	143	133	133
query99	478	341	304	304
Total cold run time: 178353 ms
Total hot run time: 81552 ms

@hello-stephen

hello-stephen commented Sep 5, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24cb252499ce0fc3ff9cd0b2f50531b1aa626793, data reload: false
query1	0.00	0.01	0.00
query2	0.08	0.03	0.03
query3	0.25	0.10	0.10
query4	1.60	0.10	0.10
query5	0.17	0.15	0.16
query6	1.22	0.68	0.66
query7	0.04	0.00	0.00
query8	0.05	0.03	0.03
query9	0.30	0.20	0.21
query10	0.33	0.34	0.34
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.33	0.30	0.30
query14	0.44	0.44	0.45
query15	0.36	0.35	0.35
query16	0.21	0.22	0.22
query17	0.71	0.69	0.64
query18	0.18	0.17	0.16
query19	1.20	1.13	1.17
query20	0.01	0.01	0.00
query21	15.44	0.16	0.11
query22	5.05	0.04	0.04
query23	16.17	0.25	0.10
query24	3.05	0.33	0.29
query25	0.11	0.04	0.03
query26	0.72	0.16	0.12
query27	0.03	0.03	0.02
query28	3.62	0.50	0.29
query29	12.50	3.14	2.56
query30	0.25	0.11	0.12
query31	2.76	0.37	0.17
query32	3.50	0.32	0.22
query33	1.35	1.35	1.42
query34	15.39	2.22	1.76
query35	1.73	1.70	1.73
query36	0.48	0.29	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.02
query39	0.03	0.02	0.03
query40	0.12	0.08	0.08
query41	0.10	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.37 s
Total hot run time: 14.46 s

@xy720

xy720 commented Sep 5, 2026

Copy link
Copy Markdown
MemberAuthor

run external

@xy720

xy720 commented Sep 5, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

1 similar comment
@xy720

xy720 commented Sep 5, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowAwaiting requested review from morrySnowmorrySnow is a code owner
@engleflyengleflyAwaiting requested review from engleflyenglefly is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xy720@hello-stephen