[Fix](rowset) When a rowset is cooled down, it is directly deleted. This can result in data query misses in the second phase of a two-phase query. by eldenmoon · Pull Request #21741 · apache/doris

eldenmoon · 2023-07-12T02:54:11Z

Proposed changes

related pr #20732

There are two reasons for moving the logic of delayed deletion from the Tablet to the StorageEngine. The first reason is to consolidate the logic and unify the delayed operations. The second reason is that delayed garbage collection during queries can cause rowsets to remain in the "stale rowsets" state, preventing the timely deletion of rowset metadata, It may cause rowset metadata too large.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

eldenmoon · 2023-07-12T02:56:25Z

run buildall

eldenmoon · 2023-07-12T02:59:39Z

run buildall

github-actions · 2023-07-12T03:01:51Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-07-12T03:02:18Z

clang-tidy review says "All clean, LGTM! 👍"

eldenmoon · 2023-07-12T03:04:00Z

run buildall

github-actions · 2023-07-12T03:07:09Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-12T03:39:57Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 55.23 seconds
stream load tsv: 503 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162201973 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712113957_clickbench_pr_176842.html

hello-stephen · 2023-07-12T03:40:10Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 56.99 seconds
stream load tsv: 501 seconds loaded 74807831229 Bytes, about 142 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17167713010 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712114009_clickbench_pr_176840.html

hello-stephen · 2023-07-12T03:56:04Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.89 seconds
stream load tsv: 450 seconds loaded 74807831229 Bytes, about 158 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 56 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 26.1 seconds inserted 10000000 Rows, about 383K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712035601_clickbench_pr_176839.html

hello-stephen · 2023-07-12T04:00:24Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.63 seconds
stream load tsv: 452 seconds loaded 74807831229 Bytes, about 157 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 56 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
insert into select: 26.3 seconds inserted 10000000 Rows, about 380K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712040022_clickbench_pr_176835.html

eldenmoon · 2023-07-12T08:14:38Z

run buildall

github-actions · 2023-07-12T08:21:24Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-07-12T08:23:15Z

clang-tidy review says "All clean, LGTM! 👍"

eldenmoon · 2023-07-12T08:31:18Z

run buildall

github-actions · 2023-07-12T08:38:41Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-12T09:07:30Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.1 seconds
stream load tsv: 503 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17167434455 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712170729_clickbench_pr_177058.html

hello-stephen · 2023-07-12T09:09:31Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 53.46 seconds
stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17168016719 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712170930_clickbench_pr_177059.html

hello-stephen · 2023-07-12T09:12:47Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.38 seconds
stream load tsv: 454 seconds loaded 74807831229 Bytes, about 157 MB/s
stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s
stream load orc: 56 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select: 26.2 seconds inserted 10000000 Rows, about 381K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712091244_clickbench_pr_177057.html

eldenmoon · 2023-07-12T12:08:19Z

run buildall

github-actions · 2023-07-12T12:16:13Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-07-12T12:36:19Z

PR approved by anyone and no changes requested.

hello-stephen · 2023-07-12T12:44:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.07 seconds
stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17168907269 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712204407_clickbench_pr_177299.html

…his can result in data query misses in the second phase of a two-phase query. related pr apache#20732 There are two reasons for moving the logic of delayed deletion from the Tablet to the StorageEngine. The first reason is to consolidate the logic and unify the delayed operations. The second reason is that delayed garbage collection during queries can cause rowsets to remain in the "stale rowsets" state, preventing the timely deletion of rowset metadata, It may cause rowset metadata too large.

eldenmoon · 2023-07-12T14:22:03Z

run buildall

github-actions · 2023-07-12T14:28:18Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-07-12T15:00:07Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.45 seconds
stream load tsv: 502 seconds loaded 74807831229 Bytes, about 142 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17165354409 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230712230006_clickbench_pr_177431.html

dataroaring

LGTM

…his can result in data query misses in the second phase of a two-phase query. (#21741) * [Fix](rowset) When a rowset is cooled down, it is directly deleted. This can result in data query misses in the second phase of a two-phase query. related pr #20732 There are two reasons for moving the logic of delayed deletion from the Tablet to the StorageEngine. The first reason is to consolidate the logic and unify the delayed operations. The second reason is that delayed garbage collection during queries can cause rowsets to remain in the "stale rowsets" state, preventing the timely deletion of rowset metadata, It may cause rowset metadata too large. * not use unused rowsets

…his can result in data query misses in the second phase of a two-phase query. (apache#21741) * [Fix](rowset) When a rowset is cooled down, it is directly deleted. This can result in data query misses in the second phase of a two-phase query. related pr apache#20732 There are two reasons for moving the logic of delayed deletion from the Tablet to the StorageEngine. The first reason is to consolidate the logic and unify the delayed operations. The second reason is that delayed garbage collection during queries can cause rowsets to remain in the "stale rowsets" state, preventing the timely deletion of rowset metadata, It may cause rowset metadata too large. * not use unused rowsets

TsukiokaKogane · 2024-03-08T11:38:17Z

this pr could cause unused rowset not gc, fixed in #30904

github-actionsBot added the kind/test label Jul 12, 2023

eldenmoon force-pushed the 2pr-rs-delete branch from d2d4ee7 to d0cdacaCompare July 12, 2023 02:55

eldenmoon force-pushed the 2pr-rs-delete branch from d0cdaca to 720d564Compare July 12, 2023 02:59

platoneko reviewed Jul 12, 2023
View reviewed changes

Comment threadbe/src/olap/storage_engine.cpp

platoneko reviewed Jul 12, 2023
View reviewed changes

Comment threadbe/src/olap/storage_engine.h Outdated

eldenmoon force-pushed the 2pr-rs-delete branch from abe8202 to e7f6089Compare July 12, 2023 08:15

eldenmoon force-pushed the 2pr-rs-delete branch from e7f6089 to c937224Compare July 12, 2023 08:31

platoneko reviewed Jul 12, 2023
View reviewed changes

Comment threadbe/src/olap/storage_engine.cpp

platoneko approved these changes Jul 12, 2023
View reviewed changes

github-actionsBot added the reviewed label Jul 12, 2023

eldenmoon added 2 commits July 12, 2023 22:21

not use unused rowsets
3dd39f1

eldenmoon force-pushed the 2pr-rs-delete branch from 45b98ab to 3dd39f1Compare July 12, 2023 14:21

dataroaring approved these changes Jul 13, 2023
View reviewed changes

dataroaring added the dev/2.0.0 2.0.0 release label Jul 13, 2023

dataroaring merged commit 9cad929 into apache:masterJul 13, 2023

eldenmoon deleted the 2pr-rs-delete branch July 13, 2023 05:23

xiaokang added dev/2.0.0-merged and removed dev/2.0.0 2.0.0 release labels Jul 13, 2023

Uh oh!

Conversation

eldenmoon commented Jul 12, 2023

Proposed changes

Further comments

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

eldenmoon commented Jul 12, 2023

Uh oh!

github-actionsBot commented Jul 12, 2023

Uh oh!

hello-stephen commented Jul 12, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

TsukiokaKogane commented Mar 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants