Skip to content

[fix](be) Handle zero-argument count aggregate states - #67679

Open
HappenLee wants to merge 1 commit into
apache:masterfrom
HappenLee:fix-count-state-zero-arguments
Open

[fix](be) Handle zero-argument count aggregate states#67679
HappenLee wants to merge 1 commit into
apache:masterfrom
HappenLee:fix-count-state-zero-arguments

Conversation

@HappenLee

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: #58031

Problem Summary:

count_union(count_state()) can crash the BE while initializing an aggregate evaluator. FE represents zero-argument count as count(*) and sends an AggState with an empty subtype list, but DataTypeAggState unconditionally reads the first subtype to infer the nested result type.

Handle zero-argument count with its explicit Int64 result type while preserving the empty argument signature and reusing the existing count implementation. Reject unsupported zero-argument AggState functions with INVALID_ARGUMENT before invoking creators that require an input type. Parameterized aggregate handling and serialization formats remain unchanged.

Release note

Fix a BE crash when constructing zero-argument count aggregate states, including queries using count_union(count_state()) and count_merge(count_state()).

Check List (For Author)

  • Test:
    • Unit Test: 20 ASAN tests passed (DataTypeAggStateZeroArgumentTest.*, Params/DataTypeAggStateTest.*, AggregateFunctionCountTest.*). New coverage includes empty/one/three/8193-row state serialization and merge, plus invalid empty-argument functions.
    • Regression test: test_count_state_zero_arguments passed on an isolated ASAN BE cluster. Generated expected output with -forceGenOut, then passed a normal comparison run. Covers the original crash, empty input, nullable input, 10001 rows, grouped union, and parameterized Decimal aggregates. BE remained alive afterward.
    • BE and FE build passed with ./build.sh --be --fe -j 48; clang-format 16 and build hygiene checks passed.
    • clang-tidy reported no diagnostics on changed lines. The script could not pass because the base branch has an unmatched NOLINTEND in be/src/core/types.h; other emitted diagnostics also refer to unchanged code.
  • Behavior changed:
    • Yes. Zero-argument count AggState construction succeeds instead of crashing; unsupported empty-argument states raise an error.
  • Does this need documentation?
    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?
Issue Number: N/A
Related PR: apache#58031
Problem Summary:
`count_union(count_state())` can crash the BE while initializing an aggregate evaluator. FE represents zero-argument count as `count(*)` and sends an AggState with an empty subtype list, but `DataTypeAggState` unconditionally reads the first subtype to infer the nested result type.
Handle zero-argument count with its explicit Int64 result type while preserving the empty argument signature and reusing the existing count implementation. Reject unsupported zero-argument AggState functions with `INVALID_ARGUMENT` before invoking creators that require an input type. Parameterized aggregate handling and serialization formats remain unchanged.
### Release note
Fix a BE crash when constructing zero-argument count aggregate states, including queries using `count_union(count_state())` and `count_merge(count_state())`.
### Check List (For Author)
- Test:
- [x] Unit Test: 20 ASAN tests passed (`DataTypeAggStateZeroArgumentTest.*`, `Params/DataTypeAggStateTest.*`, `AggregateFunctionCountTest.*`). New coverage includes empty/one/three/8193-row state serialization and merge, plus invalid empty-argument functions.
- [x] Regression test: `test_count_state_zero_arguments` passed on an isolated ASAN BE cluster. Generated expected output with `-forceGenOut`, then passed a normal comparison run. Covers the original crash, empty input, nullable input, 10001 rows, grouped union, and parameterized Decimal aggregates. BE remained alive afterward.
- BE and FE build passed with `./build.sh --be --fe -j 48`; clang-format 16 and build hygiene checks passed.
- clang-tidy reported no diagnostics on changed lines. The script could not pass because the base branch has an unmatched `NOLINTEND` in `be/src/core/types.h`; other emitted diagnostics also refer to unchanged code.
- Behavior changed:
- [x] Yes. Zero-argument count AggState construction succeeds instead of crashing; unsupported empty-argument states raise an error.
- Does this need documentation?
- [x] No.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee

Copy link
Copy Markdown
ContributorAuthor

/review

@HappenLee

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion: LGTM; no blocking issue found.

  • Goal and correctness: The change fixes the exact BE failure by handling the only valid zero-argument aggregate-state signature, canonical count, before indexing its subtype list. The explicit Int64 result agrees with FE Count, BE AggregateFunctionCount, and merge output. Invalid empty signatures now fail with INVALID_ARGUMENT.
  • Scope and parallel paths: The production edit is limited to constructor inference. I traced Thrift, protobuf, segment, and tablet-schema reconstruction through scalar count_state(), constant materialization, union, merge, empty input, grouped aggregation, and multiple input blocks. Nonempty, nullable, and Decimal aggregate paths remain unchanged.
  • Concurrency and lifecycle: No new shared mutable state, thread entry, lock, lock order, static-initialization dependency, ownership cycle, or special cleanup path is introduced. Aggregate state remains arena-owned and the function factory remains protected by its existing call_once initialization.
  • Compatibility and persistence: No FE-BE field, storage layout, serialized payload, function symbol, configuration item, EditLog path, transaction path, or data-write protocol changes. The existing BE execution version is preserved in every reconstruction path, and count continues to use the existing fixed-length state payload.
  • Tests and result oracles: The BE test covers 0, 1, 3, and 8193 states through serialization/deserialization and merge, plus invalid empty signatures. The regression suite exercises the original query path, empty input/union, nullable and constant arguments, 10001 rows, grouped union, and parameterized Decimal aggregates; every checked scalar result in the generated output is consistent. Per the review-runner instructions I did not build or run tests, but the read-only header-hygiene gate passed all checks.
  • Performance, memory, and observability: The new branch is construction-only, uses the existing tracked test buffer and state allocation paths, and adds no hot-path scan or redundant per-row work. The explicit invalid-argument exception is sufficient for the new failure mode; no new metric or log is warranted.
  • User focus: No additional user-provided review focus was supplied.

Non-blocking PR-description note: Related PR: #58031 currently points to an unrelated session-variable persistence change; please correct or remove that link if it was accidental.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16724 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 094f6499cbcb58acfaf99f1d0f942454c871b9eb, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17601	3068	3090	3068
q2	2099	264	224	224
q3	10234	835	526	526
q4	4671	247	202	202
q5	7682	606	398	398
q6	142	118	96	96
q7	516	502	396	396
q8	9242	850	951	850
q9	3511	2409	2397	2397
q10	6502	873	701	701
q11	398	200	179	179
q12	624	270	200	200
q13	18120	1534	1155	1155
q14	157	148	138	138
q15	q16	443	399	366	366
q17	1379	815	754	754
q18	3088	2264	2254	2254
q19	1277	934	784	784
q20	392	285	202	202
q21	5554	1600	1842	1600
q22	334	275	234	234
Total cold run time: 93966 ms
Total hot run time: 16724 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3473	3383	3364	3364
q2	505	405	393	393
q3	2262	2263	2155	2155
q4	1198	1176	910	910
q5	2200	2143	2128	2128
q6	163	120	89	89
q7	1104	930	877	877
q8	1604	1428	1428	1428
q9	3142	3124	3100	3100
q10	1883	1790	1654	1654
q11	364	275	256	256
q12	464	430	345	345
q13	1475	1538	1169	1169
q14	172	165	171	165
q15	q16	394	396	373	373
q17	3562	3255	3245	3245
q18	4820	4455	4785	4455
q19	851	942	889	889
q20	1007	981	803	803
q21	3946	3265	3212	3212
q22	395	360	329	329
Total cold run time: 34984 ms
Total hot run time: 31339 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82183 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 094f6499cbcb58acfaf99f1d0f942454c871b9eb, data reload: false
query5	4274	428	348	348
query6	375	138	131	131
query7	4926	425	229	229
query8	286	124	117	117
query9	8697	2974	2979	2974
query10	399	228	184	184
query11	5376	1058	923	923
query12	131	71	70	70
query13	1184	442	341	341
query14	6188	2240	2135	2135
query14_1	2003	2037	1990	1990
query15	175	124	117	117
query16	924	384	361	361
query17	792	446	374	374
query18	2343	336	236	236
query19	191	136	105	105
query20	69	68	73	68
query21	202	100	87	87
query22	5539	5396	5272	5272
query23	6722	6285	5886	5886
query23_1	6113	6119	6008	6008
query24	7215	1100	786	786
query24_1	778	805	746	746
query25	404	274	234	234
query26	1228	232	131	131
query27	2787	390	244	244
query28	4723	1508	1501	1501
query29	907	431	342	342
query30	263	157	130	130
query31	816	392	329	329
query32	129	70	82	70
query33	449	213	172	172
query34	982	868	480	480
query35	405	401	352	352
query36	575	563	548	548
query37	117	83	76	76
query38	1006	848	813	813
query39	510	485	485	485
query39_1	475	459	465	459
query40	200	92	75	75
query41	56	54	52	52
query42	78	72	74	72
query43	243	242	212	212
query44	1003	550	549	549
query45	107	107	100	100
query46	769	809	513	513
query47	783	772	723	723
query48	315	306	223	223
query49	547	243	193	193
query50	736	259	197	197
query51	8351	8135	7901	7901
query52	74	74	67	67
query53	192	205	155	155
query54	209	202	163	163
query55	70	62	63	62
query56	211	173	170	170
query57	702	669	671	669
query58	205	275	194	194
query59	1217	1249	1103	1103
query60	243	195	196	195
query61	138	134	159	134
query62	376	204	189	189
query63	173	140	138	138
query64	2860	739	565	565
query65	1704	1634	1638	1634
query66	1848	265	228	228
query67	9758	9674	9789	9674
query68	3023	1143	740	740
query69	352	237	189	189
query70	671	601	611	601
query71	246	178	164	164
query72	2310	1728	1598	1598
query73	647	581	342	342
query74	2023	1233	1143	1143
query75	1184	1108	950	950
query76	2361	736	536	536
query77	260	259	215	215
query78	3959	3663	3181	3181
query79	2330	859	581	581
query80	1591	336	279	279
query81	495	158	138	138
query82	609	126	94	94
query83	288	212	191	191
query84	287	112	89	89
query85	771	344	286	286
query86	388	173	176	173
query87	1024	975	910	910
query88	2791	2132	2112	2112
query89	285	196	178	178
query90	2043	131	136	131
query91	131	162	98	98
query92	84	73	63	63
query93	1444	1115	688	688
query94	645	225	238	225
query95	534	324	223	223
query96	797	619	283	283
query97	1044	1049	1027	1027
query98	159	138	135	135
query99	417	348	309	309
Total cold run time: 178433 ms
Total hot run time: 82183 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 094f6499cbcb58acfaf99f1d0f942454c871b9eb, data reload: false
query1	0.00	0.00	0.00
query2	0.07	0.03	0.04
query3	0.24	0.11	0.11
query4	1.60	0.10	0.10
query5	0.17	0.16	0.16
query6	1.26	0.69	0.71
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.30	0.22	0.22
query10	0.36	0.34	0.36
query11	0.18	0.12	0.12
query12	0.15	0.12	0.12
query13	0.32	0.33	0.31
query14	0.46	0.46	0.46
query15	0.37	0.37	0.36
query16	0.22	0.22	0.23
query17	0.73	0.66	0.76
query18	0.19	0.17	0.17
query19	1.20	1.17	1.25
query20	0.02	0.02	0.01
query21	15.45	0.16	0.13
query22	5.06	0.04	0.05
query23	16.16	0.26	0.12
query24	2.97	0.35	0.25
query25	0.10	0.03	0.03
query26	0.76	0.17	0.13
query27	0.04	0.03	0.04
query28	3.66	0.59	0.27
query29	12.45	3.23	2.59
query30	0.25	0.11	0.12
query31	2.76	0.39	0.18
query32	3.51	0.32	0.22
query33	1.50	1.52	1.37
query34	15.37	2.21	1.83
query35	1.81	1.79	1.80
query36	0.45	0.29	0.29
query37	0.05	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 90.61 s
Total hot run time: 14.89 s

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@HappenLee@hello-stephen