Skip to content

[fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369) - #39707

Merged
yiguolei merged 6 commits into
apache:branch-2.1from
zhannngchen:cp-38369-2.1
Aug 21, 2024
Merged

[fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369)#39707
yiguolei merged 6 commits into
apache:branch-2.1from
zhannngchen:cp-38369-2.1

Conversation

@zhannngchen

@zhannngchenzhannngchen commented Aug 21, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

cherry-pick #38369 and #38800

@zhannngchen
[fix](merge-on-write) segcompaction should process delete bitmap if n…
115393e
…ecessary (apache#38369)
Issue Number: close #xxx
When loading data to a unique key table with sequence column, some data
in current load job might be marked as delete due to a lower sequence
value.
If there's many segments in such load job, segcompaction might be
triggered, which don't process the delete bitmap currently, will cause
data correctness issue
For example:
1. we have 4 segments in current load job initially, and due to seq
column, some rows are marked as deleted
2. after segcompaction, if we don't process the delete bitmap, it's
content is still corresponding to the old segment layout, and row
7,14,15 is not mark deleted correctly on new generated segment 1.
3. in this PR, we convert old delete bitmap to fit new segment layout,
it use similar way as base/cumulative compaction to convert delete
bitmaps on old layout to new one, but the rowid conversion is simpler
![whiteboard_exported_image-2](https://github.com/user-attachments/assets/a419b6a4-e583-457a-bf4e-56d9bd2a3544)
@zhannngchen
[fix](segcompaction) fix convert delete bitmap core (apache#38800)
e2d072b
## Proposed changes
Issue Number: close #xxx
introduced by apache#38369
@doris-robot

doris-robot commented Aug 21, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhannngchen

zhannngchen commented Aug 21, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Aug 21, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
@zhannngchen

zhannngchen commented Aug 21, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Aug 21, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
github-actions[bot]
github-actionsBot reviewed Aug 21, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
Comment threadbe/test/olap/segcompaction_mow_test.cpp
@zhannngchen

zhannngchen commented Aug 21, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@yiguolei
yiguolei merged commit 0e694f1 into apache:branch-2.1Aug 21, 2024
zhannngchen added a commit to zhannngchen/incubator-doris that referenced this pull request Aug 22, 2024
@zhannngchen
[fix](merge-on-write) segcompaction should process delete bitmap if n…
28143ff
…ecessary (apache#38369) (apache#39707)
Issue Number: close #xxx
cherry-pick apache#38369 and apache#38800
@yiguoleiyiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhannngchen@doris-robot@yiguolei