Uh oh!
There was an error while loading. Please reload this page.
[opt](iceberg)support iceberg in batch mode - #46398
Conversation
Thearas
commented
Jan 3, 2025
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
f72acc5 to
33d6d4eComparewuwenchi
commented
Jan 7, 2025
run buildall |
doris-robot
commented
Jan 7, 2025
TPC-H: Total hot run time: 32871 ms |
doris-robot
commented
Jan 7, 2025
TPC-DS: Total hot run time: 197112 ms |
doris-robot
commented
Jan 7, 2025
ClickBench: Total hot run time: 31.32 s |
wuwenchi
commented
Jan 8, 2025
run buildall |
doris-robot
commented
Jan 8, 2025
TPC-H: Total hot run time: 32391 ms |
doris-robot
commented
Jan 8, 2025
TPC-DS: Total hot run time: 187196 ms |
doris-robot
commented
Jan 8, 2025
ClickBench: Total hot run time: 31.06 s |
b10ea0e to
4dd49a2Comparewuwenchi
commented
Jan 13, 2025
run buildall |
doris-robot
commented
Jan 13, 2025
TPC-H: Total hot run time: 33404 ms |
doris-robot
commented
Jan 13, 2025
TPC-DS: Total hot run time: 187592 ms |
doris-robot
commented
Jan 13, 2025
ClickBench: Total hot run time: 31.82 s |
| } catch (InterruptedException e) { | ||
| throw new IOException(e); | ||
| } catch (UndeclaredThrowableException e) { | ||
| if (e.getCause() instanceof RuntimeException) { |
There was a problem hiding this comment.
You can use Util.getRootCause(e)
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| @Override | ||
| public boolean isBatchMode() { | ||
| return sessionVariable.getNumPartitionsInBatchMode() > 0; |
There was a problem hiding this comment.
find a better to to determine using batch mode or not
| long realFileSplitSize = getRealFileSplitSize(0); | ||
| CloseableIterable<FileScanTask> fileScanTasks = null; | ||
| public CloseableIterable<FileScanTask> planFileScanTask(TableScan scan) { |
There was a problem hiding this comment.
| publicCloseableIterable<FileScanTask> planFileScanTask(TableScanscan) { | |
| privateCloseableIterable<FileScanTask> planFileScanTask(TableScanscan) { |
Uh oh!
There was an error while loading. Please reload this page.
| CloseableIterable<FileScanTask> splitFiles; | ||
| try { | ||
| fileScanTasks = TableScanUtil.splitFiles(scan.planFiles(), realFileSplitSize); | ||
| splitFiles = TableScanUtil.splitFiles(scan.planFiles(), targetSplitSize); |
wuwenchi
commented
Jan 22, 2025
run buildall |
doris-robot
commented
Jan 22, 2025
TPC-H: Total hot run time: 32426 ms |
doris-robot
commented
Jan 22, 2025
TPC-DS: Total hot run time: 194235 ms |
doris-robot
commented
Jan 22, 2025
TeamCity be ut coverage result: |
doris-robot
commented
Jan 22, 2025
ClickBench: Total hot run time: 30.84 s |
wuwenchi
commented
Jan 22, 2025
run buildall |
doris-robot
commented
Jan 22, 2025
TPC-H: Total hot run time: 32103 ms |
doris-robot
commented
Jan 23, 2025
TPC-H: Total hot run time: 32252 ms |
doris-robot
commented
Jan 23, 2025
TPC-DS: Total hot run time: 187828 ms |
doris-robot
commented
Jan 23, 2025
ClickBench: Total hot run time: 31.51 s |
doris-robot
commented
Jan 23, 2025
TeamCity be ut coverage result: |
1. use thread pool in planFiles. 2. support batch mode. 3. When executing `doAs`, if the exception type is `UndeclaredThrowableException`, we should get the specific cause inside. Because the message of UndeclaredThrowableException is null, the message will not be obtained if the exception is captured externally 4. Modify the default configuration: fetch_splits_max_wait_time_ms: from 4000 to 1000 remote_split_source_batch_size: from 10240 to 1000
### What problem does this PR solve? 1. use thread pool in planFiles. 2. support batch mode. 3. When executing `doAs`, if the exception type is `UndeclaredThrowableException`, we should get the specific cause inside. Because the message of UndeclaredThrowableException is null, the message will not be obtained if the exception is captured externally 4. Modify the default configuration: fetch_splits_max_wait_time_ms: from 4000 to 1000 remote_split_source_batch_size: from 10240 to 1000
### What problem does this PR solve? Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related #46398
### What problem does this PR solve? 1. use thread pool in planFiles. 2. support batch mode. 3. When executing `doAs`, if the exception type is `UndeclaredThrowableException`, we should get the specific cause inside. Because the message of UndeclaredThrowableException is null, the message will not be obtained if the exception is captured externally 4. Modify the default configuration: fetch_splits_max_wait_time_ms: from 4000 to 1000 remote_split_source_batch_size: from 10240 to 1000
…#49489) ### What problem does this PR solve? Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related apache#46398
…#49489) Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related apache#46398
1. use thread pool in planFiles. 2. support batch mode. 3. When executing `doAs`, if the exception type is `UndeclaredThrowableException`, we should get the specific cause inside. Because the message of UndeclaredThrowableException is null, the message will not be obtained if the exception is captured externally 4. Modify the default configuration: fetch_splits_max_wait_time_ms: from 4000 to 1000 remote_split_source_batch_size: from 10240 to 1000
…#49489) Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related apache#46398
What problem does this PR solve?
doAs, if the exception type isUndeclaredThrowableException, we should get the specific cause inside. Because the message of UndeclaredThrowableException is null, the message will not be obtained if the exception is captured externallyfetch_splits_max_wait_time_ms: from 4000 to 1000
remote_split_source_batch_size: from 10240 to 1000
ATTN:
There is a cache client pool in BE to obtain splits from FE.
When FE restarts, the clients will report errors due to invalidation.
This will cause an error when querying data in batch mode for the first time:
Don't worry, just query again.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)