Skip to content

[fix](merge-on-write) fix duplicate key in schema change - #25705

Merged
xiaokang merged 6 commits into
apache:masterfrom
liaoxin01:fix_sc_master
Oct 25, 2023
Merged

[fix](merge-on-write) fix duplicate key in schema change#25705
xiaokang merged 6 commits into
apache:masterfrom
liaoxin01:fix_sc_master

Conversation

@liaoxin01

@liaoxin01liaoxin01 commented Oct 20, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

  1. It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
  2. The remaining NOTREADY tablet in the schema change failure should be dropped.
  3. When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@liaoxin01

liaoxin01 commented Oct 20, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Oct 20, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@liaoxin01
liaoxin01 marked this pull request as draft October 20, 2023 12:36
@liaoxin01
liaoxin01 marked this pull request as ready for review October 24, 2023 11:21
@liaoxin01

liaoxin01 commented Oct 24, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@liaoxin01

liaoxin01 commented Oct 24, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Oct 24, 2023

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.52 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162279163 Bytes

@apacheapache deleted a comment from github-actionsBotOct 25, 2023
zhannngchen
zhannngchen previously approved these changes Oct 25, 2023
View reviewed changes

@zhannngchenzhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment threadbe/src/vec/exec/scan/new_olap_scan_node.cpp Outdated
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions

github-actionsBot commented Oct 25, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Oct 25, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@liaoxin01

liaoxin01 commented Oct 25, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Oct 25, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/olap/tablet.cpp
@liaoxin01

liaoxin01 commented Oct 25, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

zhannngchen
zhannngchen approved these changes Oct 25, 2023
View reviewed changes

@zhannngchenzhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions

github-actionsBot commented Oct 25, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot

doris-robot commented Oct 25, 2023

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.62 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162172721 Bytes

xiaokang
xiaokang approved these changes Oct 25, 2023
View reviewed changes

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang
xiaokang merged commit f31c1d8 into apache:masterOct 25, 2023
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Oct 25, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)
4b77cac
@liaoxin01liaoxin01 mentioned this pull request Oct 25, 2023
Merged
xiaokang pushed a commit that referenced this pull request Oct 25, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (#25705) (#2…
6f8ea15
…5914)
dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
@liaoxin01王翔宇
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)
7c3e600
It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
The remaining NOTREADY tablet in the schema change failure should be dropped.
When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
@liaoxin01gnehil
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)…
ce76137
… (apache#25914)
@xiaokangxiaokang mentioned this pull request Dec 4, 2023
Closed
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@liaoxin01胥剑旭
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)
f1330f9
It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
The remaining NOTREADY tablet in the schema change failure should be dropped.
When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.
@liaoxin01
liaoxin01 deleted the fix_sc_master branch February 6, 2024 12:29
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@xiaokangxiaokangxiaokang approved these changes
@zhannngchenzhannngchenzhannngchen approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.3-mergedmerge_conflictp0_bp0_wreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liaoxin01@doris-robot@xiaokang@zhannngchen