Uh oh!
There was an error while loading. Please reload this page.
[Fix](auto-partition) fix auto partition load lost data in multi sender - #34740
[Fix](auto-partition) fix auto partition load lost data in multi sender#34740zclllyybb wants to merge 6 commits into
Conversation
doris-robot
commented
May 12, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
zclllyybb
commented
May 12, 2024
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
May 12, 2024
TeamCity be ut coverage result: |
zclllyybb
commented
May 12, 2024
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
May 12, 2024
TeamCity be ut coverage result: |
doris-robot
commented
May 12, 2024
TPC-H: Total hot run time: 40797 ms |
doris-robot
commented
May 12, 2024
TPC-DS: Total hot run time: 186958 ms |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
clang-tidy review says "All clean, LGTM! 👍" |
zclllyybb
commented
May 13, 2024
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
zclllyybb
commented
May 13, 2024
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
May 13, 2024
TeamCity be ut coverage result: |
doris-robot
commented
May 13, 2024
TPC-H: Total hot run time: 41730 ms |
doris-robot
commented
May 13, 2024
TPC-DS: Total hot run time: 187323 ms |
zclllyybb
commented
May 14, 2024
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
May 14, 2024
TeamCity be ut coverage result: |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
zclllyybb
commented
May 15, 2024
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
Uh oh!
There was an error while loading. Please reload this page.
| } | ||
| }; | ||
| struct WriteBlockCallbackContext { |
There was a problem hiding this comment.
Only one variable, no need this structure?
There was a problem hiding this comment.
introduced in progress of this pr. for more change possible in the future I think this is more friendly so didn't remove this.
doris-robot
commented
May 15, 2024
TPC-H: Total hot run time: 41113 ms |
doris-robot
commented
May 15, 2024
TPC-DS: Total hot run time: 187038 ms |
zclllyybb
commented
May 15, 2024
run buildall |
doris-robot
commented
May 15, 2024
ClickBench: Total hot run time: 30.37 s |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
May 15, 2024
TeamCity be ut coverage result: |
zclllyybb
commented
May 24, 2024
close this since #35287 covered |
…er (#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace #34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
…er (#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace #34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
…er (apache#35287) Change `use_cnt` mechanism for incremental (auto partition) channels and streams, it's now dynamically counted. Use `close_wait()` of regular partitions as a synchronize point to make sure all sinks are in close phase before closing any incremental (auto partition) channels and streams. Add dummy (fake) partition and tablet if there is no regular partition in the auto partition table. Replace apache#34740 Co-authored-by: zhaochangle <zhaochangle@selectdb.com>
Proposed changes
Issue Number: close #xxx
Before if we have multi sender but some of their sink node have no data of some specific partition, then this sink won't know it has to close it. So tablets channel won't be closed correctly.
Now change close time needed count to same with open in tablets channel.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...