Skip to content

[Opt](TabletSchema) reuse TabletColumn info to reduce mem (#42448) - #43349

Merged
eldenmoon merged 1 commit into
apache:branch-2.1from
eldenmoon:mem-opt-2.1
Nov 11, 2024
Merged

[Opt](TabletSchema) reuse TabletColumn info to reduce mem (#42448)#43349
eldenmoon merged 1 commit into
apache:branch-2.1from
eldenmoon:mem-opt-2.1

Conversation

@eldenmoon

@eldenmooneldenmoon commented Nov 6, 2024

Copy link
Copy Markdown
Member

@doris-robot

doris-robot commented Nov 6, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon

eldenmoon commented Nov 6, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@eldenmoon

eldenmoon commented Nov 6, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@github-actions

github-actionsBot commented Nov 6, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Nov 6, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@eldenmoon
[Opt](TabletSchema) reuse TabletColumn info to reduce mem (apache#42448)
9e80851
1. When there are a large number of identical TabletColumns in the
cluster, which usually occurs when VARIANT type columns are modified and
added, each Rowset has an individual TabletSchema. Excessive
TabletSchemas can lead to significant memory overhead. Reusing memory
for identical TabletColumns would greatly reduce this memory
consumption.
2. Serialized TabletSchema as LRU cache key could also increase memusage
when large sets of schemas are in LRU cache, so inorder to reduce the
memory footprint we just record the key signature caculated by
generating an UUID by hash algorithm， and lookup the key signature in
LRU cache， and check the key in case of hash collision
@eldenmoon

eldenmoon commented Nov 6, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@github-actions

github-actionsBot commented Nov 6, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Nov 6, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.33% (9471/26070)
Line Coverage: 27.87% (77979/279747)
Region Coverage: 26.53% (39994/150742)
Branch Coverage: 23.31% (20311/87122)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9e80851483d9f5b9c9701cbefcdbad4a2b913c22_9e80851483d9f5b9c9701cbefcdbad4a2b913c22/report/index.html

@eldenmoon
eldenmoon merged commit e073b57 into apache:branch-2.1Nov 11, 2024
@yiguoleiyiguolei mentioned this pull request Jan 19, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@eldenmoon@doris-robot