[fix](cooldown) Fix potential data loss when clone task's dst tablet is cooldown replica by platoneko · Pull Request #17644 · apache/doris

platoneko · 2023-03-10T04:36:33Z

Proposed changes

Issue Number: close #xxx

Problem summary

If this replica is cooldown replica, MUST generate a new cooldown_meta_id to avoid use cooldown_meta_id generated in old cooldown term which may lead to such situation:
Replica A is cooldown replica, cooldown_meta_id=2,
Replica B: cooldown_replica=A, cooldown_meta_id=1
Replica A: full clone Replica B, cooldown_meta_id=1, but remote cooldown_meta is still with cooldown_meta_id=2
After tablet report. FE finds all replicas' cooldowned data is consistent(cooldown_meta_id=1)
Replica A: confirm_unused_remote_files, delete some cooldowned data of cooldown_meta_id=2
Replica B: follow_cooldown_data, cooldown_meta_id=2, data lost

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-03-10T04:39:59Z

clang-tidy review says "All clean, LGTM! 👍"

platoneko · 2023-03-13T03:34:42Z

run beut

platoneko · 2023-03-13T03:34:47Z

run feut

platoneko · 2023-03-13T03:34:57Z

run p0

platoneko · 2023-03-13T03:35:04Z

run clickbench

dataroaring · 2023-03-19T02:25:45Z

run buildall

platoneko · 2023-03-19T03:36:23Z

run buildall

github-actions · 2023-03-19T03:40:14Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-03-19T04:16:56Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.18 seconds
stream load tsv: 478 seconds loaded 74807831229 Bytes, about 149 MB/s
stream load json: 26 seconds loaded 2358488459 Bytes, about 86 MB/s
stream load orc: 76 seconds loaded 1101869774 Bytes, about 13 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230319054840_clickbench_pr_116669.html

dataroaring

LGTM

github-actions · 2023-08-18T11:06:04Z

clang-tidy review says "All clean, LGTM! 👍"

xiaokang · 2023-08-24T10:13:45Z

run buildall

dataroaring

LGTM

dataroaring · 2023-08-27T02:03:18Z

run buildall

xiaokang

LGTM

github-actions · 2023-08-27T02:09:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-08-27T02:09:28Z

PR approved by anyone and no changes requested.

github-actions · 2023-08-27T02:09:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-27T10:15:10Z

clang-tidy review says "All clean, LGTM! 👍"

xiaokang · 2023-08-27T11:04:24Z

run buildall

platoneko · 2023-09-01T03:24:38Z

run buildall

platoneko · 2023-09-01T03:48:12Z

run buildall

github-actions · 2023-09-01T03:54:05Z

clang-tidy review says "All clean, LGTM! 👍"

platoneko · 2023-09-01T05:29:58Z

run buildall

github-actions · 2023-09-01T05:36:08Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-01T06:11:24Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.98 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162195977 Bytes

dataroaring

LGTM

github-actions · 2023-09-01T07:13:03Z

PR approved by at least one committer and no changes requested.

…is cooldown replica (#17644) Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com> Co-authored-by: Kang <kxiao.tiger@gmail.com>

platoneko force-pushed the cooldown_replica_as_clone_dst_tablet branch from 952b095 to 5305f10Compare March 19, 2023 03:35

dataroaring added the dev/2.0.0 2.0.0 release label Jul 25, 2023

xiaokang added dev/2.0.1 and removed dev/2.0.0 2.0.0 release labels Jul 26, 2023

xiaokang added the p0_l label Aug 18, 2023

dataroaring previously approved these changes Aug 18, 2023
View reviewed changes

platoneko added 3 commits August 18, 2023 18:57

Generate new cooldown_meta_id when clone dst tablet is cooldown replica
ff879c7

log created clone task
fa1c06d

improve log
7780330

platoneko dismissed dataroaring’s stale review via 7780330August 18, 2023 10:59

platoneko force-pushed the cooldown_replica_as_clone_dst_tablet branch from 5305f10 to 7780330Compare August 18, 2023 10:59

dataroaring previously approved these changes Aug 27, 2023
View reviewed changes

Merge branch 'master' into cooldown_replica_as_clone_dst_tablet
7d45e9d

xiaokang previously approved these changes Aug 27, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2023

github-actionsBot added the reviewed label Aug 27, 2023

Merge branch 'master' into cooldown_replica_as_clone_dst_tablet
b652347

xiaokang added dev/2.0.2 and removed dev/2.0.1 labels Aug 28, 2023

Merge branch 'master' into cooldown_replica_as_clone_dst_tablet
6a455f7

platoneko dismissed stale reviews from dataroaring and xiaokang via 6a455f7September 1, 2023 05:29

platoneko force-pushed the cooldown_replica_as_clone_dst_tablet branch from 1477542 to 6a455f7Compare September 1, 2023 05:29

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 1, 2023

dataroaring approved these changes Sep 1, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2023

ByteYue approved these changes Sep 1, 2023
View reviewed changes

dataroaring merged commit 9d2fc78 into apache:masterSep 1, 2023

xiaokang added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 1, 2023

xiaokang mentioned this pull request Sep 28, 2023
Release Note 2.0.2 #25011
Closed

Doris-Breakwater mentioned this pull request Jul 26, 2026
[Bug] Cold data (cooldown remote files) can be wrongly deleted after full clone, causing data loss #66051
Open

Uh oh!

Conversation

platoneko commented Mar 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actionsBot commented Mar 10, 2023

Uh oh!

platoneko commented Mar 13, 2023

Uh oh!

platoneko commented Mar 13, 2023

Uh oh!

platoneko commented Mar 13, 2023

Uh oh!

platoneko commented Mar 13, 2023

Uh oh!

dataroaring commented Mar 19, 2023

Uh oh!

platoneko commented Mar 19, 2023

Uh oh!

github-actionsBot commented Mar 19, 2023

Uh oh!

hello-stephen commented Mar 19, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 18, 2023

Uh oh!

xiaokang commented Aug 24, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring commented Aug 27, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 27, 2023

Uh oh!

github-actionsBot commented Aug 27, 2023

Uh oh!

github-actionsBot commented Aug 27, 2023

Uh oh!

github-actionsBot commented Aug 27, 2023

Uh oh!

xiaokang commented Aug 27, 2023

Uh oh!

platoneko commented Sep 1, 2023

Uh oh!

platoneko commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

platoneko commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

hello-stephen commented Sep 1, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

platoneko commented Mar 10, 2023•
edited
Loading

hello-stephen commented Mar 19, 2023•
edited
Loading