Skip to content

[fix](hdfs)Fix be coredump in HDFS reader during profile collection. - #56806

Merged
morningman merged 1 commit into
apache:masterfrom
hubgeter:fix_hdfs_handle
Oct 14, 2025
Merged

[fix](hdfs)Fix be coredump in HDFS reader during profile collection.#56806
morningman merged 1 commit into
apache:masterfrom
hubgeter:fix_hdfs_handle

Conversation

@hubgeter

@hubgeterhubgeter commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: #54926
Issue Number: close#55747

Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read failed, but HDFS reader still held such a dangling pointer, which led to issue #55747

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

hubgeter commented Oct 10, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Oct 10, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190230 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c23464e1782cabe74ae14124319333b947bb49e5, data reload: false
query1	1048	433	403	403
query2	6580	1714	1681	1681
query3	6773	222	222	222
query4	25963	23974	23018	23018
query5	5201	643	465	465
query6	365	242	240	240
query7	4658	496	295	295
query8	305	251	255	251
query9	8702	2564	2544	2544
query10	519	339	291	291
query11	15520	15646	15524	15524
query12	196	129	125	125
query13	2337	561	441	441
query14	11566	9495	9507	9495
query15	250	213	175	175
query16	8283	738	542	542
query17	1838	804	675	675
query18	2334	443	365	365
query19	239	209	185	185
query20	141	145	134	134
query21	222	139	127	127
query22	4630	4646	4566	4566
query23	34908	33901	33151	33151
query24	8395	2350	2367	2350
query25	555	497	441	441
query26	1231	265	153	153
query27	2729	483	364	364
query28	4382	2192	2167	2167
query29	757	592	486	486
query30	291	224	196	196
query31	905	810	758	758
query32	80	70	71	70
query33	581	371	325	325
query34	794	861	520	520
query35	767	825	762	762
query36	996	999	936	936
query37	126	112	89	89
query38	3509	3596	3536	3536
query39	1488	1403	1409	1403
query40	222	128	116	116
query41	60	60	62	60
query42	123	116	108	108
query43	480	487	470	470
query44	1303	842	811	811
query45	186	178	172	172
query46	863	979	627	627
query47	1763	1836	1744	1744
query48	381	414	317	317
query49	772	489	419	419
query50	638	687	405	405
query51	3914	3979	4022	3979
query52	119	107	97	97
query53	234	268	195	195
query54	593	616	521	521
query55	85	84	87	84
query56	322	333	311	311
query57	1200	1219	1116	1116
query58	290	280	268	268
query59	2520	2658	2506	2506
query60	350	344	332	332
query61	159	159	154	154
query62	777	733	666	666
query63	231	197	192	192
query64	4405	1168	894	894
query65	4040	3955	3935	3935
query66	1083	457	346	346
query67	15408	15461	15251	15251
query68	7832	934	609	609
query69	509	344	303	303
query70	1354	1345	1281	1281
query71	455	340	332	332
query72	6137	4870	4851	4851
query73	617	586	364	364
query74	8926	9059	8969	8969
query75	3780	3329	2919	2919
query76	3498	1152	742	742
query77	803	415	321	321
query78	9569	9880	8911	8911
query79	2821	816	585	585
query80	679	565	487	487
query81	509	260	234	234
query82	452	166	134	134
query83	310	285	256	256
query84	309	120	91	91
query85	893	550	425	425
query86	386	325	302	302
query87	3772	3750	3677	3677
query88	3727	2205	2233	2205
query89	382	322	290	290
query90	2013	223	217	217
query91	163	172	148	148
query92	89	69	68	68
query93	2301	971	634	634
query94	712	454	342	342
query95	403	319	321	319
query96	481	588	277	277
query97	2947	2949	2867	2867
query98	249	214	214	214
query99	1444	1401	1289	1289
Total cold run time: 281207 ms
Total hot run time: 190230 ms

@doris-robot

doris-robot commented Oct 10, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c23464e1782cabe74ae14124319333b947bb49e5, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.07
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.28	0.27	0.24
query6	1.20	0.65	0.63
query7	0.04	0.03	0.03
query8	0.06	0.04	0.04
query9	0.64	0.52	0.52
query10	0.58	0.59	0.58
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	1.02	1.05	1.04
query15	0.87	0.87	0.87
query16	0.43	0.40	0.41
query17	1.07	1.06	1.06
query18	0.21	0.20	0.20
query19	1.97	1.89	1.84
query20	0.02	0.01	0.02
query21	15.43	0.92	0.58
query22	0.76	1.22	0.73
query23	14.83	1.39	0.69
query24	7.53	1.86	0.46
query25	0.56	0.28	0.13
query26	0.59	0.15	0.13
query27	0.07	0.05	0.05
query28	9.71	1.36	0.95
query29	12.55	3.98	3.37
query30	0.29	0.15	0.12
query31	2.83	0.60	0.39
query32	3.25	0.54	0.47
query33	3.11	3.14	3.11
query34	15.99	5.55	4.87
query35	4.92	4.90	4.93
query36	0.70	0.53	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.16	0.14
query41	0.09	0.04	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 105 s
Total hot run time: 30.41 s

@hello-stephen

hello-stephen commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.53% (17730/33755)
Line Coverage37.69% (160951/427003)
Region Coverage32.19% (122979/382045)
Branch Coverage33.57% (53913/160576)

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2025
@github-actions

github-actionsBot commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

hello-stephen commented Oct 10, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.09% (23511/33073)
Line Coverage57.53% (245409/426575)
Region Coverage52.54% (203261/386873)
Branch Coverage54.43% (87835/161387)

dataroaring
dataroaring approved these changes Oct 13, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit 032ebae into apache:masterOct 14, 2025
33 of 34 checks passed
github-actionsBot pushed a commit that referenced this pull request Oct 14, 2025
@hubgeterYour Name
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
3f531bc
…56806)
### What problem does this PR solve?
Related PR: #54926
Issue Number: close#55747
Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
@github-actionsgithub-actionsBot mentioned this pull request Oct 14, 2025
Merged
github-actionsBot pushed a commit that referenced this pull request Oct 14, 2025
@hubgeterYour Name
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
33fc0ee
…56806)
### What problem does this PR solve?
Related PR: #54926
Issue Number: close#55747
Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
@github-actionsgithub-actionsBot mentioned this pull request Oct 14, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 16, 2025
@github-actions@hubgeter
branch-4.0: [fix](hdfs)Fix be coredump in HDFS reader during profile …
44a73f0
…collection. #56806 (#56951)
Cherry-picked from #56806
Co-authored-by: daidai <changyuwei@selectdb.com>
morningman pushed a commit that referenced this pull request Oct 16, 2025
@hubgeter@morningman
[fix](hdfs)Fix be coredump in HDFS reader during profile collection. (#…
7e4c05e
…56806)
### What problem does this PR solve?
Related PR: #54926
Issue Number: close#55747
Problem Summary:
PR #54926 destroyed the corresponding hdfs handle cache when HDFS read
failed, but HDFS reader still held such a dangling pointer, which led to
issue #55747
morningman pushed a commit that referenced this pull request Oct 16, 2025
@github-actions@hubgeter
branch-3.1: [fix](hdfs)Fix be coredump in HDFS reader during profile …
8b8425c
…collection. #56806 (#56950)
Cherry-picked from #56806
Co-authored-by: daidai <changyuwei@selectdb.com>
@yiguoleiyiguolei mentioned this pull request Nov 5, 2025
Closed
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Nov 12, 2025
@selectdb-robot@hubgeter
branch-selectdb-doris-3.0: [fix](hdfs)Fix be coredump in HDFS reader …
59ca790
…during profile collection. apache#56806 (apache#5192)
cherry-picks from apache#56806
Related to apache#5076
Co-authored-by: daidai <changyuwei@selectdb.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@dataroaringdataroaringdataroaring approved these changes
@gavinchougavinchouAwaiting requested review from gavinchougavinchou is a code owner

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.xdev/3.1.2-mergeddev/4.0.1-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] be crash when reading hive table from different format with same table

6 participants

@hubgeter@hello-stephen@doris-robot@morningman@dataroaring@yiguolei