[improvement](bitmap) Use shared_ptr in BitmapValue to avoid deep copying by mrhhsg · Pull Request #19101 · apache/doris

mrhhsg · 2023-04-26T07:23:51Z

Proposed changes

Currently bitmapvalue type is copied between columns, it cost a lot of memory. Use a shared ptr in bitmap value to avoid copy data.

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

mrhhsg · 2023-04-26T07:27:17Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-04-26T07:36:51Z

+ BitmapValue& operator=(const BitmapValue& other) {
+ _type = other._type;
+ _sv = other._sv;
+ _bitmap = other._bitmap;
+ return *this;
+ }


warning: use '= default' to define a trivial copy-assignment operator [modernize-use-equals-default]
Suggested change
BitmapValue&operator=(constBitmapValue&other) {
_type=other._type;
_sv=other._sv;
_bitmap=other._bitmap;
return*this;
}
BitmapValue&operator=(constBitmapValue&other) =default;

mrhhsg · 2023-04-26T08:39:10Z

run buildall

github-actions · 2023-04-26T08:44:19Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-04-26T08:46:49Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.65 seconds
stream load tsv: 421 seconds loaded 74807831229 Bytes, about 169 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230504033341_clickbench_pr_137893.html

mrhhsg · 2023-04-28T00:25:32Z

run buildall

github-actions · 2023-04-28T00:29:10Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-04-28T03:13:35Z

run buildall

github-actions · 2023-04-28T03:16:36Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-05-04T01:59:10Z

run buildall

github-actions · 2023-05-04T02:02:34Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-05-04T02:33:47Z

run buildall

github-actions · 2023-05-04T02:35:56Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-05-20T11:22:16Z

+ return;
+ }
+
+ if (_bitmap.use_count() > 1) {


感觉这里用use count 可能不太好，不如记录一个标记位，比如is_owner 这样的，然后再copy constructor 还有operator=的时候，都把is_owner 设置为false；prepare for write的时候，设置为true。

如果is owner == true，就直接return了。我觉得一个bool 的判断效率要比 shared ptr use count 高

…ying

mrhhsg · 2023-05-22T02:59:45Z

run buildall

github-actions · 2023-05-22T03:02:29Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-05-22T04:55:23Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.94 seconds
stream load tsv: 423 seconds loaded 74807831229 Bytes, about 168 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 79.4 seconds inserted 10000000 Rows, about 125K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230522045521_clickbench_pr_148057.html

github-actions · 2023-05-22T07:30:50Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-05-22T09:06:49Z

run buildall

hello-stephen · 2023-05-22T10:39:55Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 36.68 seconds
stream load tsv: 433 seconds loaded 74807831229 Bytes, about 164 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 80.4 seconds inserted 10000000 Rows, about 124K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230522103952_clickbench_pr_148293.html

yiguolei

LGTM

github-actions · 2023-05-24T08:12:42Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-05-24T08:12:44Z

PR approved by anyone and no changes requested.

…ying (apache#19101) Currently bitmapvalue type is copied between columns, it cost a lot of memory. Use a shared ptr in bitmap value to avoid copy data.

…ying (#19101) (#21271) Currently bitmapvalue type is copied between columns, it cost a lot of memory. Use a shared ptr in bitmap value to avoid copy data.

…ying (apache#19101) (apache#21271) Currently bitmapvalue type is copied between columns, it cost a lot of memory. Use a shared ptr in bitmap value to avoid copy data.

mrhhsg force-pushed the opt_bitmap branch from 62a005f to dc882d9Compare April 26, 2023 07:24

github-actionsBot reviewed Apr 26, 2023
View reviewed changes

mrhhsg force-pushed the opt_bitmap branch from dc882d9 to d81f917Compare April 26, 2023 08:36

morningman added the dev/1.2.5 label Apr 27, 2023

mrhhsg force-pushed the opt_bitmap branch from d81f917 to 19eb1f4Compare April 28, 2023 00:25

mrhhsg force-pushed the opt_bitmap branch from 19eb1f4 to 0e6b36eCompare April 28, 2023 03:13

mrhhsg force-pushed the opt_bitmap branch from 0e6b36e to a11d4e4Compare May 4, 2023 01:58

mrhhsg force-pushed the opt_bitmap branch from a11d4e4 to f40f0f0Compare May 4, 2023 02:32

yiguolei reviewed May 20, 2023
View reviewed changes

[improvement](bitmap) Use shared_ptr in BitmapValue to avoid deep cop…
863fab8
…ying

mrhhsg force-pushed the opt_bitmap branch from f40f0f0 to 863fab8Compare May 22, 2023 02:58

yiguolei reviewed May 22, 2023
View reviewed changes

Comment threadbe/src/util/bitmap_value.h

yiguolei reviewed May 22, 2023
View reviewed changes

Comment threadbe/src/util/bitmap_value.h

fix: _is_shared was not set
eb76d85

yiguolei approved these changes May 24, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 24, 2023

github-actionsBot added the reviewed label May 24, 2023

yiguolei merged commit e5eed53 into apache:masterMay 24, 2023

morningman added the need-cherry-pick label May 27, 2023

mrhhsg mentioned this pull request Jun 28, 2023
[improvement](bitmap) Use shared_ptr in BitmapValue to avoid deep copying #21271
Merged

mrhhsg deleted the opt_bitmap branch August 18, 2023 06:31

-BitmapValue&operator=(constBitmapValue&other) {
-_type=other._type;
-_sv=other._sv;
-_bitmap=other._bitmap;
-return*this;
- }
+BitmapValue&operator=(constBitmapValue&other) =default;

Uh oh!

Conversation

mrhhsg commented Apr 26, 2023• edited by yiguolei Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

mrhhsg commented Apr 26, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 26, 2023

Choose a reason for hiding this comment

Uh oh!

mrhhsg commented Apr 26, 2023

Uh oh!

github-actionsBot commented Apr 26, 2023

Uh oh!

hello-stephen commented Apr 26, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrhhsg commented Apr 28, 2023

Uh oh!

github-actionsBot commented Apr 28, 2023

Uh oh!

mrhhsg commented Apr 28, 2023

Uh oh!

github-actionsBot commented Apr 28, 2023

Uh oh!

mrhhsg commented May 4, 2023

Uh oh!

github-actionsBot commented May 4, 2023

Uh oh!

mrhhsg commented May 4, 2023

Uh oh!

github-actionsBot commented May 4, 2023

Uh oh!

yiguoleiMay 20, 2023

Choose a reason for hiding this comment

Uh oh!

yiguoleiMay 20, 2023

Choose a reason for hiding this comment

Uh oh!

mrhhsg commented May 22, 2023

Uh oh!

github-actionsBot commented May 22, 2023

Uh oh!

Uh oh!

Uh oh!

hello-stephen commented May 22, 2023

Uh oh!

github-actionsBot commented May 22, 2023

Uh oh!

mrhhsg commented May 22, 2023

Uh oh!

hello-stephen commented May 22, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 24, 2023

Uh oh!

github-actionsBot commented May 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrhhsg commented Apr 26, 2023•
edited by yiguolei
Loading

hello-stephen commented Apr 26, 2023•
edited
Loading