Uh oh!
There was an error while loading. Please reload this page.
Support concurrent export of query results - #6539
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| } else { | ||
| _storage_type = TStorageBackendType::BROKER; | ||
| } | ||
| _fragment_instance_id.hi = 12345678987654321; |
There was a problem hiding this comment.
In order to be compatible with the old version of Fe and the new version of be
There was a problem hiding this comment.
Add comment to explain.
BTW, I suggest to add FileNamePrefix column in the result of outfile operation. For easy to get full name of exported files. Such as:
+------------+-----------+-----------+-------------+------------------------------------------+
| FileNumber | TotalRows | FileSize | URL |FileNamePrefix |
+------------+-----------+-----------+-------------+------------------------------------------+
| 1 | 123605 | 361061014 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10325|
| 1 | 128180 | 374334318 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10326|
| 1 | 125156 | 365569023 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10327|
| 1 | 124096 | 362395588 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10328|
| 1 | 124862 | 364727515 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10329|
| 1 | 124520 | 363649600 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10330|
| 1 | 124447 | 363479285 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10331|
| 1 | 125134 | 365490611 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10332|
+------------+-----------+-----------+-------------+------------------------------------------+
There was a problem hiding this comment.
It's not easy to add, there will be compatibility issues
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This pr mainly supports
1. Export query result sets concurrently
2. Query result set export supports s3 protocol
Among them, there are several preconditions for concurrently exporting query result sets
1. Enable concurrent export variables
2. The query itself can be exported concurrently
(some queries containing sort nodes at the top level cannot be exported concurrently)
3. Export the s3 protocol used instead of the broker
After exporting the result set concurrently,
the file prefix is changed to outfile_{query_instance_id}_filenumber.{file_format}3203362 to
29e5e79ComparePR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
PR approved by at least one committer and no changes requested. |
legacy planner impl PR: apache#6539
legacy planner impl PR: apache#6539
legacy planner impl PR: apache#6539
legacy planner impl PR: apache#6539
Proposed changes
This pr mainly supports
Among them, there are several preconditions for concurrently exporting query result sets
(some queries containing sort nodes at the top level cannot be exported concurrently)
After exporting the result set concurrently,
the file prefix is changed to outfile_{query_instance_id}_filenumber.{file_format}
Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.