[fix](replica) handle replica version missing info to avoid -214 error by morningman · Pull Request #8209 · apache/doris

morningman · 2022-02-23T10:38:50Z

Proposed changes

Issue Number: close#8208

Problem Summary:

In the original tablet reporting information, the version missing information is done by combining
two pieces of information as follows:

the maximum consecutive version number
the version_miss field

The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying.

After the change, we directly use the version checking logic used in the query, and set version_miss to true
if a missing version is found

and on the FE processing side. Originally, only the bad replica information was syncronized among FEs,
but not the version missing information. As a result, the non-master FE is not aware of the missing version information.

In the new design, we deprecate the original log persistence class BackendTabletsInfo and use the new
BackendReplicasInfo to record replica reporting information and write both bad and version missing
information to metadata so that other FEs can synchronize these information.

Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (Yes)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

yangzhg · 2022-03-03T05:37:16Z

 }

-OLAPStatus Tablet::check_version_integrity(const Version& version) {
+OLAPStatus Tablet::check_version_integrity(const Version& version, bool quite) {


what does quite mean?

if !quite, it will print log for missing versions.

You mean quiet? slient may be better

yangzhg

LGTM

yangzhg

LGTM

github-actions · 2022-03-09T03:23:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2022-03-09T03:23:49Z

PR approved by anyone and no changes requested.

maybe typo

yangzhg

LGTM

github-actions · 2022-03-09T03:36:21Z

PR approved by at least one committer and no changes requested.

apache#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

#8444) 1. This bug was introduced by #8209. Error in fe.warn.log: ``` java.lang.IllegalStateException: 560278 at com.google.common.base.Preconditions.checkState(Preconditions.java:508) ~[spark-dpp-0.15-SNAPSHOT.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.TabletInvertedIndex.getReplica(TabletInvertedIndex.java:462) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayBackendReplicasInfo(Catalog.java:6941) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:626) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayJournal(Catalog.java:2446) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:116) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:74) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) [palo-fe.jar:0.15-SNAPSHOT] ``` Since the reporting of a tablet and the deletion of a tablet are two independent events and are not mutually exclusive, it may happen that the tablet is deleted first and the reporting is done later. 2. Change the tablet report info. Now, the version of a tablet report from BE is the largest continuous version. Eg, versions: [1,2,3,5,7], the report version of this tablet will be 3.

apache#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

this bug was introduced from apache#8209

this bug was introduced from #8209

#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

apache#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

apache#8444) 1. This bug was introduced by apache#8209. Error in fe.warn.log: ``` java.lang.IllegalStateException: 560278 at com.google.common.base.Preconditions.checkState(Preconditions.java:508) ~[spark-dpp-0.15-SNAPSHOT.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.TabletInvertedIndex.getReplica(TabletInvertedIndex.java:462) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayBackendReplicasInfo(Catalog.java:6941) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:626) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayJournal(Catalog.java:2446) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:116) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:74) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) [palo-fe.jar:0.15-SNAPSHOT] ``` Since the reporting of a tablet and the deletion of a tablet are two independent events and are not mutually exclusive, it may happen that the tablet is deleted first and the reporting is done later. 2. Change the tablet report info. Now, the version of a tablet report from BE is the largest continuous version. Eg, versions: [1,2,3,5,7], the report version of this tablet will be 3.

this bug was introduced from apache#8209

morningman added the dev/backlog waiting to be merged in future dev branch label Feb 23, 2022

yangzhg reviewed Mar 3, 2022
View reviewed changes

yangzhg previously approved these changes Mar 8, 2022
View reviewed changes

morningmanand others added 4 commits March 9, 2022 10:54

1
202e6e3

2
013d3b5

2
f2564f2

3
934cc61

morningman dismissed yangzhg’s stale review via 934cc61March 9, 2022 03:00

morningman force-pushed the bug214 branch from 540d97c to 934cc61Compare March 9, 2022 03:00

rebase
92818a7

yangzhg previously approved these changes Mar 9, 2022
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 9, 2022

github-actionsBot added the reviewed label Mar 9, 2022

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Mar 9, 2022

fix
244ba7e

yangzhg approved these changes Mar 9, 2022
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 9, 2022

morningman merged commit 826467e into apache:masterMar 9, 2022

morningman mentioned this pull request Mar 11, 2022
[fix](report) fix bug that tablet may already be delete when reporting #8444
Merged

morningman pushed a commit to morningman/doris that referenced this pull request Mar 12, 2022
[fix] BE crash when reporting tablet
88c5343
this bug was introduced from apache#8209

morningman mentioned this pull request Mar 12, 2022
[fix] BE crash when reporting tablet #8453
Merged

morningman added a commit that referenced this pull request Mar 12, 2022
[fix] BE crash when reporting tablet (#8453)
5f8e948
this bug was introduced from #8209

morningman added a commit that referenced this pull request Mar 12, 2022
[fix] BE crash when reporting tablet (#8453)
32da525
this bug was introduced from #8209

morningman added dev/merged-1.0.1-deprecated PR has been merged into dev-1.0.1 and removed dev/backlog waiting to be merged in future dev branch labels Mar 14, 2022

zhengte pushed a commit to zhengte/incubator-doris that referenced this pull request Mar 15, 2022
[fix] BE crash when reporting tablet (apache#8453)
e0f494b
this bug was introduced from apache#8209

morningman mentioned this pull request Apr 27, 2022
[Bug] failed to initialize storage reader. tablet=924991.xxxx, res=-214, backend=xxxx #9267
Closed
3 tasks

Navigation Menu

Uh oh!

[fix](replica) handle replica version missing info to avoid -214 error - #8209

[fix](replica) handle replica version missing info to avoid -214 error#8209
morningman merged 6 commits into
apache:masterfrom
morningman:bug214

morningman commented Feb 23, 2022

Uh oh!

yangzhgMar 3, 2022

Uh oh!

morningmanMar 5, 2022

Uh oh!

yangzhgMar 9, 2022

Uh oh!

yangzhg left a comment

Uh oh!

yangzhg left a comment

Uh oh!

github-actionsBot commented Mar 9, 2022

Uh oh!

github-actionsBot commented Mar 9, 2022

Uh oh!

yangzhg left a comment

Uh oh!

github-actionsBot commented Mar 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

morningman commented Feb 23, 2022

Proposed changes

Problem Summary:

Checklist(Required)

Further comments

Uh oh!

yangzhgMar 3, 2022

Choose a reason for hiding this comment

Uh oh!

morningmanMar 5, 2022

Choose a reason for hiding this comment

Uh oh!

yangzhgMar 9, 2022

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 9, 2022

Uh oh!

github-actionsBot commented Mar 9, 2022

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants