Uh oh!
There was an error while loading. Please reload this page.
[opt](merge-on-write) Skip the alignment process of some rowsets in partial update - #38487
Merged
Conversation
doris-robot
commented
Jul 29, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
bobhan1
commented
Jul 29, 2024
ContributorAuthor
run buildall |
Uh oh!
There was an error while loading. Please reload this page.
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
2 similar comments
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Jul 29, 2024
TPC-H: Total hot run time: 42183 ms |
bobhan1force-pushed
the
opt-partial-update
branch
2 times, most recently
from
July 29, 2024 12:05
cb7cc76 to
fcc24a4Comparedoris-robot
commented
Jul 29, 2024
TPC-DS: Total hot run time: 169362 ms |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
bobhan1
commented
Jul 29, 2024
ContributorAuthor
run buildall |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
bobhan1
commented
Jul 29, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jul 29, 2024
TPC-H: Total hot run time: 41702 ms |
doris-robot
commented
Jul 29, 2024
TPC-DS: Total hot run time: 170035 ms |
doris-robot
commented
Jul 29, 2024
ClickBench: Total hot run time: 29.76 s |
bobhan1force-pushed
the
opt-partial-update
branch
2 times, most recently
from
July 30, 2024 04:37
7a220ae to
6e4f5ddCompareContributor
clang-tidy review says "All clean, LGTM! 👍" |
2 similar comments
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
Contributor
clang-tidy review says "All clean, LGTM! 👍" |
bobhan1
commented
Jul 30, 2024
ContributorAuthor
run buildall |
bobhan1force-pushed
the
opt-partial-update
branch
2 times, most recently
from
July 30, 2024 06:32
6adb68a to
58ef523Compare| @@ -1192,6 +1192,16 @@ void BaseTablet::_remove_sentinel_mark_from_delete_bitmap(DeleteBitmapPtr delete | |||
| Status BaseTablet::update_delete_bitmap(const BaseTabletSPtr& self, TabletTxnInfo* txn_info, | |||
Contributor
There was a problem hiding this comment.
warning: function 'update_delete_bitmap' has cognitive complexity of 91 (threshold 50) [readability-function-cognitive-complexity]
Status BaseTablet::update_delete_bitmap(const BaseTabletSPtr& self, TabletTxnInfo* txn_info,
^Additional context
be/src/olap/base_tablet.cpp:1194: +1, including nesting penalty of 0, nesting level increased to 1
DBUG_EXECUTE_IF("BaseTablet::update_delete_bitmap.enable_spin_wait", {
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/olap/base_tablet.cpp:1194: +2, including nesting penalty of 1, nesting level increased to 2
DBUG_EXECUTE_IF("BaseTablet::update_delete_bitmap.enable_spin_wait", {
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/olap/base_tablet.cpp:1200: +3, including nesting penalty of 2, nesting level increased to 3
if (self->tablet_id() == block_tablet_id) {
^be/src/olap/base_tablet.cpp:1201: +4, including nesting penalty of 3, nesting level increased to 4
DBUG_EXECUTE_IF("BaseTablet::update_delete_bitmap.block", DBUG_BLOCK);
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/olap/base_tablet.cpp:1201: +5, including nesting penalty of 4, nesting level increased to 5
DBUG_EXECUTE_IF("BaseTablet::update_delete_bitmap.block", DBUG_BLOCK);
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/olap/base_tablet.cpp:1201: +6, including nesting penalty of 5, nesting level increased to 6
DBUG_EXECUTE_IF("BaseTablet::update_delete_bitmap.block", DBUG_BLOCK);
^be/src/util/debug_points.h:49: expanded from macro 'DBUG_BLOCK'
while (DebugPoints::instance()->is_enable(DP_NAME)) { \
^be/src/olap/base_tablet.cpp:1214: +1
txn_info->partial_update_info && txn_info->partial_update_info->is_partial_update;
^be/src/olap/base_tablet.cpp:1215: +1, including nesting penalty of 0, nesting level increased to 1
if (is_partial_update) {
^be/src/olap/base_tablet.cpp:1216: nesting level increased to 2
transient_rs_writer = DORIS_TRY(self->create_transient_rowset_writer(
^be/src/common/status.h:700: expanded from macro 'DORIS_TRY'
({ \
^be/src/olap/base_tablet.cpp:1216: +3, including nesting penalty of 2, nesting level increased to 3
transient_rs_writer = DORIS_TRY(self->create_transient_rowset_writer(
^be/src/common/status.h:703: expanded from macro 'DORIS_TRY'
if (!res.has_value()) [[unlikely]] { \
^be/src/olap/base_tablet.cpp:1228: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(std::dynamic_pointer_cast<BetaRowset>(rowset)->load_segments(&segments));
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1228: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(std::dynamic_pointer_cast<BetaRowset>(rowset)->load_segments(&segments));
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1234: +1, including nesting penalty of 0, nesting level increased to 1
if (self->tablet_state() == TABLET_NOTREADY) {
^be/src/olap/base_tablet.cpp:1239: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(self->get_all_rs_id_unlocked(cur_version - 1, &cur_rowset_ids));
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1239: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(self->get_all_rs_id_unlocked(cur_version - 1, &cur_rowset_ids));
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1261: +1, including nesting penalty of 0, nesting level increased to 1
if (is_partial_update) {
^be/src/olap/base_tablet.cpp:1274: +2, including nesting penalty of 1, nesting level increased to 2
if (!rowsets_skip_alignment.empty()) {
^be/src/olap/base_tablet.cpp:1279: +1, including nesting penalty of 0, nesting level increased to 1
if (!rowsets_skip_alignment.empty()) {
^be/src/olap/base_tablet.cpp:1282: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments, rowsets_skip_alignment,
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1282: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments, rowsets_skip_alignment,
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1284: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(token->wait());
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1284: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(token->wait());
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1289: +1, including nesting penalty of 0, nesting level increased to 1
if (segments.size() <= 1) {
^be/src/olap/base_tablet.cpp:1290: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments, specified_rowsets, delete_bitmap,
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1290: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments, specified_rowsets, delete_bitmap,
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1293: +1, nesting level increased to 1
} else {
^be/src/olap/base_tablet.cpp:1295: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments, specified_rowsets, delete_bitmap,
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1295: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments, specified_rowsets, delete_bitmap,
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1298: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(token->wait());
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1298: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(token->wait());
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1306: +1, including nesting penalty of 0, nesting level increased to 1
if (config::enable_merge_on_write_correctness_check && rowset->num_rows() != 0) {
^be/src/olap/base_tablet.cpp:1306: +1
if (config::enable_merge_on_write_correctness_check && rowset->num_rows() != 0) {
^be/src/olap/base_tablet.cpp:1311: +2, including nesting penalty of 1, nesting level increased to 2
if (!st.ok()) {
^be/src/olap/base_tablet.cpp:1316: +1, including nesting penalty of 0, nesting level increased to 1
if (transient_rs_writer) {
^be/src/olap/base_tablet.cpp:1318: +2, including nesting penalty of 1, nesting level increased to 2
DBUG_EXECUTE_IF("Tablet.update_delete_bitmap.partial_update_write_rowset_fail", {
^be/src/util/debug_points.h:36: expanded from macro 'DBUG_EXECUTE_IF'
if (UNLIKELY(config::enable_debug_points)) { \
^be/src/olap/base_tablet.cpp:1318: +3, including nesting penalty of 2, nesting level increased to 3
DBUG_EXECUTE_IF("Tablet.update_delete_bitmap.partial_update_write_rowset_fail", {
^be/src/util/debug_points.h:38: expanded from macro 'DBUG_EXECUTE_IF'
if (dp) { \
^be/src/olap/base_tablet.cpp:1319: +4, including nesting penalty of 3, nesting level increased to 4
if (rand() % 100 < (100 * dp->param("percent", 0.5))) {
^be/src/olap/base_tablet.cpp:1327: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(transient_rs_writer->flush());
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1327: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(transient_rs_writer->flush());
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1329: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(transient_rs_writer->build(transient_rowset));
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1329: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(transient_rs_writer->build(transient_rowset));
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/olap/base_tablet.cpp:1345: nesting level increased to 1
[](size_t sum, const segment_v2::SegmentSharedPtr& s) { return sum += s->num_rows(); });
^be/src/olap/base_tablet.cpp:1347: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(self->save_delete_bitmap(txn_info, txn_id, delete_bitmap,
^be/src/common/status.h:618: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/olap/base_tablet.cpp:1347: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(self->save_delete_bitmap(txn_info, txn_id, delete_bitmap,
^be/src/common/status.h:620: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^doris-robot
commented
Jul 30, 2024
TPC-H: Total hot run time: 42383 ms |
doris-robot
commented
Jul 30, 2024
TPC-DS: Total hot run time: 170484 ms |
doris-robot
commented
Jul 30, 2024
ClickBench: Total hot run time: 29.81 s |
bobhan1
commented
Jul 31, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jul 31, 2024
TPC-H: Total hot run time: 42179 ms |
doris-robot
commented
Jul 31, 2024
TPC-DS: Total hot run time: 170451 ms |
doris-robot
commented
Jul 31, 2024
ClickBench: Total hot run time: 30.24 s |
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
gavinchou pushed a commit
that referenced
this pull request
Aug 7, 2024
…artial update (#38487) If a rowset is produced by compaction before the commit phase of the partial update load and is not included in `txn_info->rowset_ids`, we can skip the alignment process of that rowset because data remains the same before and after compaction. But we still need to calculate the the delete bitmap for that rowset.
feiniaofeiafei pushed a commit
to feiniaofeiafei/doris
that referenced
this pull request
Aug 9, 2024
…artial update (apache#38487) If a rowset is produced by compaction before the commit phase of the partial update load and is not included in `txn_info->rowset_ids`, we can skip the alignment process of that rowset because data remains the same before and after compaction. But we still need to calculate the the delete bitmap for that rowset.
Closed
GoGoWen pushed a commit
to GoGoWen/incubator-doris
that referenced
this pull request
Aug 27, 2024
…artial update apache#38487 (apache#38686) ## Proposed changes picks apache#38487 regression cases are commented currently.
Closed
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Jan 20, 2025
…artial update apache#38487 (apache#38686) picks apache#38487 regression cases are commented currently.
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Jan 20, 2025
…artial update apache#38487 (apache#38686) picks apache#38487 regression cases are commented currently.
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…artial update (apache#38487) If a rowset is produced by compaction before the commit phase of the partial update load and is not included in `txn_info->rowset_ids`, we can skip the alignment process of that rowset because data remains the same before and after compaction. But we still need to calculate the the delete bitmap for that rowset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
If a rowset is produced by compaction before the commit phase of the partial update load and is not included in
txn_info->rowset_ids, we can skip the alignment process of that rowset because data remains the same before and after compaction. But we still need to calculate the the delete bitmap for that rowset.branch-2.1-pick: #38682
branch-2.0-pick: #38686