Skip to content

[Fix](Partial update) Fix wrong position using in segment writer - #22782

Merged
zhannngchen merged 11 commits into
apache:masterfrom
Yukang-Lian:Fix_Segment_Writer_Wrong_Pos
Aug 16, 2023
Merged

[Fix](Partial update) Fix wrong position using in segment writer#22782
zhannngchen merged 11 commits into
apache:masterfrom
Yukang-Lian:Fix_Segment_Writer_Wrong_Pos

Conversation

@Yukang-Lian

@Yukang-LianYukang-Lian commented Aug 9, 2023•
edited
Loading

Copy link
Copy Markdown
Collaborator

Proposed changes

Issue Number: close #xxx

In the process of writing data from block to the segment by the partial update, there will be three different positions, block_pos, delta_pos and segment_pos, and their relationship is shown in the figure below. This PR fixes the use of the wrong positions.

block segment
2 -> 0
3 -> 1 *
4 -> 2
5 -> 3
Take the behavior example with *, we will add block 2-5 rows to segment 0-3 rows. Here * row block_pos = 3, segment_pos = 1, delta_pos = 3-2 = 1.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc..

@github-actions

github-actionsBot commented Aug 9, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

zhannngchen
zhannngchen reviewed Aug 10, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
@github-actions

github-actionsBot commented Aug 10, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Aug 10, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian

Yukang-Lian commented Aug 10, 2023

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

zhannngchen
zhannngchen reviewed Aug 14, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
@Yukang-Lian

Yukang-Lian commented Aug 14, 2023

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian

Yukang-Lian commented Aug 14, 2023

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

zhannngchen
zhannngchen reviewed Aug 14, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
zhannngchen
zhannngchen reviewed Aug 14, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
zhannngchen
zhannngchen reviewed Aug 14, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
zhannngchen
zhannngchen reviewed Aug 14, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Yukang-Lianforce-pushed the Fix_Segment_Writer_Wrong_Pos branch from cccbbb1 to 003a32bCompareAugust 14, 2023 09:09
@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian

Yukang-Lian commented Aug 14, 2023

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 14, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@Yukang-Lian

Yukang-Lian commented Aug 15, 2023

Copy link
Copy Markdown
CollaboratorAuthor

run p0

zhannngchen
zhannngchen reviewed Aug 16, 2023
View reviewed changes
Comment threadbe/src/olap/rowset/segment_v2/segment_writer.cpp Outdated
@Yukang-Lian
Yukang-Lianforce-pushed the Fix_Segment_Writer_Wrong_Pos branch from f980b52 to 1183a58CompareAugust 16, 2023 08:01
@Yukang-Lian

Yukang-Lian commented Aug 16, 2023

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 16, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen

hello-stephen commented Aug 16, 2023

Copy link
Copy Markdown
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.75 seconds
stream load tsv: 545 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 95.8 seconds inserted 10000000 Rows, about 104K ops/s
storage size: 17162116647 Bytes

zhannngchen
zhannngchen approved these changes Aug 16, 2023
View reviewed changes

@zhannngchenzhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 16, 2023
@github-actions

github-actionsBot commented Aug 16, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@zhannngchen
zhannngchen merged commit 0aa57d1 into apache:masterAug 16, 2023
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Oct 7, 2023
@Yukang-Lian@bobhan1
pick "[Fix](Partial update) Fix wrong position using in segment writer…
bbaaead
…apache#22782"
@bobhan1bobhan1 mentioned this pull request Oct 7, 2023
Merged
xiaokang pushed a commit that referenced this pull request Oct 8, 2023
@bobhan1
[branch-2.0-pick](merge-on-write) picks PRs about delete processing o…
bfcb46d
…n merge-on-write unique table (#25071)
* pick "[Fix](Partial update) Fix wrong position using in segment writer #22782"
* pick "[Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #24011"
* pick "[Fix](merge-on-write) Correct the alignment process when the existing rows with same key has marked delete sign #24877"
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchenzhannngchenzhannngchen approved these changes
@bobhan1bobhan1bobhan1 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Yukang-Lian@hello-stephen@zhannngchen@bobhan1@xiaokang