Uh oh!
There was an error while loading. Please reload this page.
[opt](topn)Optimize the time for topn to lazy materialized reading of external tables - #52114
Conversation
hello-stephen
commented
Jun 22, 2025
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hubgeter
commented
Jun 22, 2025
run buildall |
hubgeter
commented
Jun 24, 2025
run buildall |
doris-robot
commented
Jun 24, 2025
TPC-H: Total hot run time: 35153 ms |
doris-robot
commented
Jun 24, 2025
TPC-DS: Total hot run time: 186100 ms |
doris-robot
commented
Jun 24, 2025
ClickBench: Total hot run time: 29.16 s |
hubgeter
commented
Jun 25, 2025
run buildall |
doris-robot
commented
Jun 25, 2025
TPC-H: Total hot run time: 33815 ms |
doris-robot
commented
Jun 25, 2025
TPC-DS: Total hot run time: 184906 ms |
doris-robot
commented
Jun 25, 2025
ClickBench: Total hot run time: 29.38 s |
doris-robot
commented
Jun 25, 2025
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Jun 25, 2025
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
hubgeter
commented
Jun 30, 2025
run buildall |
doris-robot
commented
Jun 30, 2025
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
doris-robot
commented
Jun 30, 2025
TPC-H: Total hot run time: 33885 ms |
doris-robot
commented
Jun 30, 2025
TPC-DS: Total hot run time: 184507 ms |
doris-robot
commented
Jun 30, 2025
ClickBench: Total hot run time: 30.09 s |
doris-robot
commented
Jun 30, 2025
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hubgeter
commented
Jul 1, 2025
run buildall |
doris-robot
commented
Jul 1, 2025
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
doris-robot
commented
Jul 1, 2025
TPC-H: Total hot run time: 34244 ms |
doris-robot
commented
Jul 1, 2025
TPC-DS: Total hot run time: 184964 ms |
doris-robot
commented
Jul 1, 2025
ClickBench: Total hot run time: 29.91 s |
hello-stephen
commented
Jul 1, 2025
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
PR approved by at least one committer and no changes requested. |
hubgeter
commented
Jul 7, 2025
run buildall |
doris-robot
commented
Jul 7, 2025
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
hubgeter
commented
Jul 8, 2025
run buildall |
doris-robot
commented
Jul 8, 2025
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
doris-robot
commented
Jul 8, 2025
TPC-H: Total hot run time: 32929 ms |
doris-robot
commented
Jul 8, 2025
TPC-DS: Total hot run time: 186311 ms |
doris-robot
commented
Jul 8, 2025
ClickBench: Total hot run time: 29.31 s |
doris-robot
commented
Jul 8, 2025
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
PR approved by at least one committer and no changes requested. |
…ses (#65759) ### What problem does this PR solve? Related pr #52114 Problem Summary: Problem Summary: TopN lazy materialization resolved deferred column indexes with output slot names. Queries that renamed Hive columns therefore produced -1 indexes, and external row-id fetch could fill those columns with NULL when positional reading was used. Resolve the index from the already traced original base column so the descriptor and index share one column identity. Add a focused planner unit test and a Hive ORC Explain regression for the alias path. Fix incorrect NULL values from aliased external-table columns when TopN lazy materialization is used.
…ses (apache#65759) Related pr apache#52114 Problem Summary: Problem Summary: TopN lazy materialization resolved deferred column indexes with output slot names. Queries that renamed Hive columns therefore produced -1 indexes, and external row-id fetch could fill those columns with NULL when positional reading was used. Resolve the index from the already traced original base column so the descriptor and index share one column identity. Add a focused planner unit test and a Hive ORC Explain regression for the alias path. Fix incorrect NULL values from aliased external-table columns when TopN lazy materialization is used.
…ses (apache#65759) Related pr apache#52114 Problem Summary: Problem Summary: TopN lazy materialization resolved deferred column indexes with output slot names. Queries that renamed Hive columns therefore produced -1 indexes, and external row-id fetch could fill those columns with NULL when positional reading was used. Resolve the index from the already traced original base column so the descriptor and index share one column identity. Add a focused planner unit test and a Hive ORC Explain regression for the alias path. Fix incorrect NULL values from aliased external-table columns when TopN lazy materialization is used.
…ses (apache#65759) Related pr apache#52114 Problem Summary: Problem Summary: TopN lazy materialization resolved deferred column indexes with output slot names. Queries that renamed Hive columns therefore produced -1 indexes, and external row-id fetch could fill those columns with NULL when positional reading was used. Resolve the index from the already traced original base column so the descriptor and index share one column identity. Add a focused planner unit test and a Hive ORC Explain regression for the alias path. Fix incorrect NULL values from aliased external-table columns when TopN lazy materialization is used.
…ses (apache#65759) ### What problem does this PR solve? Related pr apache#52114 Problem Summary: Problem Summary: TopN lazy materialization resolved deferred column indexes with output slot names. Queries that renamed Hive columns therefore produced -1 indexes, and external row-id fetch could fill those columns with NULL when positional reading was used. Resolve the index from the already traced original base column so the descriptor and index share one column identity. Add a focused planner unit test and a Hive ORC Explain regression for the alias path. Fix incorrect NULL values from aliased external-table columns when TopN lazy materialization is used.
…ses (apache#65759) ### What problem does this PR solve? Related pr apache#52114 Problem Summary: Problem Summary: TopN lazy materialization resolved deferred column indexes with output slot names. Queries that renamed Hive columns therefore produced -1 indexes, and external row-id fetch could fill those columns with NULL when positional reading was used. Resolve the index from the already traced original base column so the descriptor and index share one column identity. Add a focused planner unit test and a Hive ORC Explain regression for the alias path. Fix incorrect NULL values from aliased external-table columns when TopN lazy materialization is used.
### What problem does this PR solve? Problem Summary: In `RowIdStorageReader::read_batch_external_row`, the runtime profile entry `GetBlockAvgTime` was populated with `*init_reader_avg_ms` instead of `*get_block_avg_ms`, so the profile showed the init-reader average twice and the actual get-block average was never reported. This was a typo introduced in apache#52114. ### Release note None ### Check List (For Author) - Test: No need to test (trivial profile string fix) - Behavior changed: No - Does this need documentation: No
What problem does this PR solve?
Related PR: #51329
Problem Summary:
Topn lazy materialize was introduced in pr#51329 , but the implementation had performance issues when reading external tables. This pr is used for optimization.
RowIDFetchertoMATERIALIZATION_OPERATOR.The example is as follows:
1FE 2BE
sql :select * from ali_hive.tpch100_orc.lineitem order by l_partkey limit 10;
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)