Skip to content

[fix](txn lazy commit) fix txn lazy commit conflict with schema change - #55349

Merged
dataroaring merged 1 commit into
apache:masterfrom
swjtu-zhanglei:master-fix18
Sep 1, 2025
Merged

[fix](txn lazy commit) fix txn lazy commit conflict with schema change#55349
dataroaring merged 1 commit into
apache:masterfrom
swjtu-zhanglei:master-fix18

Conversation

@swjtu-zhanglei

@swjtu-zhangleiswjtu-zhanglei commented Aug 27, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor
  • step1: MS txn lazy commit convert tmp rowsets and make txn visible commit failed due to unexpected fdb error
  • step2: BE schema change job convert historical data and prepare/commit rowsets with the same txn_id and tablet_id for new tablet rowset meta, tmp rowset key is decided by (txn_id, tablet_id)
  • step3: MS retry lazy commit will convert tmp rowset which write by schema change job

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Aug 27, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

w41ter
w41ter previously approved these changes Aug 27, 2025
View reviewed changes
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2025
@github-actions

github-actionsBot commented Aug 27, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 27, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@swjtu-zhanglei

swjtu-zhanglei commented Aug 27, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 27, 2025
@doris-robot

doris-robot commented Aug 27, 2025

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 100.00% (11/11) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage83.60% (1427/1707)
Line Coverage67.24% (25391/37760)
Region Coverage68.32% (12652/18518)
Branch Coverage58.14% (6713/11546)

@swjtu-zhanglei

swjtu-zhanglei commented Aug 29, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@swjtu-zhanglei
[fix](txn lazy commit) fix txn lazy commit conflict with schema change
2870574
* step1: MS txn lazy commit convert tmp rowsets and make txn visible commit failed due to unexpected fdb error
* step2: BE schema change job convert historical data and prepare/commit rowsets with the same txn_id and tablet_id
for new tablet rowset meta, tmp rowset key is decided by (txn_id, tablet_id)
* step3: MS retry lazy commit will convert tmp rowset which write by schema change job
@swjtu-zhanglei

swjtu-zhanglei commented Aug 29, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 29, 2025

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 100.00% (11/11) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage83.66% (1439/1720)
Line Coverage67.35% (25736/38213)
Region Coverage68.39% (12799/18714)
Branch Coverage58.27% (6814/11694)

@github-actions

github-actionsBot commented Aug 31, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 31, 2025
bobhan1
bobhan1 reviewed Sep 1, 2025
View reviewed changes
Comment threadcloud/test/txn_lazy_commit_test.cpp
int64_t tablet_id = tablet_id_base + i;
check_tablet_idx_db_id(txn, db_id, tablet_id);
check_tmp_rowset_exist(txn, tablet_id, txn_id);
check_rowset_meta_exist(txn, tablet_id, 2);

@bobhan1bobhan1Sep 1, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also check check_rowset_meta_not_exist(txn, tablet_id, 3)?

@dataroaring
dataroaring merged commit 0f1f5da into apache:masterSep 1, 2025
32 checks passed
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Sep 5, 2025
@swjtu-zhanglei
[fix](txn lazy commit) fix txn lazy commit conflict with schema change (
e4e7639
apache#55349)
* step1: MS txn lazy commit convert tmp rowsets and make txn visible
commit failed due to unexpected fdb error
* step2: BE schema change job convert historical data and prepare/commit
rowsets with the same txn_id and tablet_id for new tablet rowset meta,
tmp rowset key is decided by (txn_id, tablet_id)
* step3: MS retry lazy commit will convert tmp rowset which write by
schema change job
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
None
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@morrySnowmorrySnow mentioned this pull request Sep 5, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 5, 2025
@swjtu-zhanglei
branch-3.1: [fix](txn lazy commit) fix txn lazy commit conflict with …
76b1260
…schema change #55349 (#55701)
picked from #55349
@swjtu-zhangleiswjtu-zhanglei mentioned this pull request Sep 11, 2025
Merged
16 tasks
@morrySnowmorrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@bobhan1bobhan1bobhan1 approved these changes
@w41terw41terw41ter left review comments
@dataroaringdataroaringAwaiting requested review from dataroaring

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.9-mergeddev/3.1.1-mergedp0_wreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@swjtu-zhanglei@Thearas@doris-robot@gavinchou@w41ter@bobhan1@dataroaring@morrySnow