Skip to content

[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap - #46969

Merged
dataroaring merged 4 commits into
apache:masterfrom
hust-hhb:fix-txn-size
Jan 16, 2025
Merged

[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap#46969
dataroaring merged 4 commits into
apache:masterfrom
hust-hhb:fix-txn-size

Conversation

@hust-hhb

@hust-hhbhust-hhb commented Jan 14, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

fdb_txn_size is not key size+value size, actually txn size is the triple of key size plus value size at least for writing, so use txn->approximate_bytes() to calculating txn size is more accurate. However, txn->approximate_bytes() is not 100% accurate either, in my test, when txn->approximate_bytes() bingger than 8.3MB, it will meet Transaction exceeds byte limit error.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb

hust-hhb commented Jan 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

zhannngchen
zhannngchen reviewed Jan 14, 2025
View reviewed changes
Comment threadcloud/src/common/config.h
zhannngchen
zhannngchen reviewed Jan 14, 2025
View reviewed changes
Comment threadcloud/src/meta-service/meta_service.cpp Outdated
zhannngchen
zhannngchen reviewed Jan 14, 2025
View reviewed changes
Comment threadcloud/src/meta-service/meta_service.cpp Outdated
@hust-hhb

hust-hhb commented Jan 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

zhannngchen
zhannngchen approved these changes Jan 15, 2025
View reviewed changes

@zhannngchenzhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 15, 2025
@github-actions

github-actionsBot commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jan 15, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 91d3a52 into apache:masterJan 16, 2025
@dataroaringdataroaring added the usercase Important user case type label label Jan 16, 2025
github-actionsBot pushed a commit that referenced this pull request Jan 16, 2025
@hust-hhbYour Name
[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size w…
9162a85
…hen updating delete bitmap (#46969)
fdb_txn_size is not key size+value size, actually txn size is the triple
of key size plus value size at least for writing, so use
txn->approximate_bytes() to calculating txn size is more accurate.
However, txn->approximate_bytes() is not 100% accurate either, in my
test, when txn->approximate_bytes() bingger than 8.3MB, it will meet
Transaction exceeds byte limit error.
@github-actionsgithub-actionsBot mentioned this pull request Jan 16, 2025
Merged
zhannngchen pushed a commit that referenced this pull request Jan 17, 2025
@github-actions@hust-hhb@CalvinKirs
branch-3.0: [fix](cloud-mow) Fix the issue of inaccurate estimation o…
415aa23
…f txn size when updating delete bitmap #46969 (#47046)
Cherry-picked from #46969
Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
@hust-hhbhust-hhb mentioned this pull request Feb 11, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Feb 11, 2025
@hust-hhb
[test](cloud-mow)Add update big delete bitmap case (#47097)
cbcebb1
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
github-actionsBot pushed a commit that referenced this pull request Feb 11, 2025
@hust-hhbYour Name
[test](cloud-mow)Add update big delete bitmap case (#47097)
7933e5f
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
@gavinchougavinchou mentioned this pull request Feb 18, 2025
Closed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb@lzyy2024
[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size w…
004d22c
…hen updating delete bitmap (apache#46969)
fdb_txn_size is not key size+value size, actually txn size is the triple
of key size plus value size at least for writing, so use
txn->approximate_bytes() to calculating txn size is more accurate.
However, txn->approximate_bytes() is not 100% accurate either, in my
test, when txn->approximate_bytes() bingger than 8.3MB, it will meet
Transaction exceeds byte limit error.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb@lzyy2024
[test](cloud-mow)Add update big delete bitmap case (apache#47097)
4ede535
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhbkoarz
[test](cloud-mow)Add update big delete bitmap case (apache#47097)
1b065e2
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025
@hust-hhb
branch-3.0: [fix](cloud-mow) Fix the issue of inaccurate estimation o…
a4ad5f1
…f txn size when updating delete bitmap apache#46969 (apache#3920)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41terw41terw41ter approved these changes
@zhannngchenzhannngchenzhannngchen approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.4-mergedreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hust-hhb@Thearas@w41ter@zhannngchen@dataroaring