Skip to content

[fix](fe-meta) fix paimon file catalog meta issue and replication num analysis issue - #24681

Merged
yiguolei merged 1 commit into
apache:masterfrom
morningman:meta_catalog_replay
Sep 21, 2023
Merged

[fix](fe-meta) fix paimon file catalog meta issue and replication num analysis issue#24681
yiguolei merged 1 commit into
apache:masterfrom
morningman:meta_catalog_replay

Conversation

@morningman

Copy link
Copy Markdown
Contributor

Proposed changes

Fix 2 issues:

  1. The replication num analysis will throw exception when there is no BE alive when FE restart.
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = errCode = 2, detailMessage = replication num should be less than the number of available backends. replication num is 1, available backend num is 0
at org.apache.doris.common.util.PropertyAnalyzer.analyzeReplicaAllocation(PropertyAnalyzer.java:1018) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.TableProperty.buildReplicaAllocation(TableProperty.java:483) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.TableProperty.read(TableProperty.java:529) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.OlapTable.readFields(OlapTable.java:1370) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Table.read(Table.java:383) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Database.readFields(Database.java:631) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.InternalCatalog.loadDb(InternalCatalog.java:3075) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.loadDb(Env.java:1764) ~[doris-fe.jar:1.2-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_341]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_341]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_341]
at org.apache.doris.persist.meta.MetaReader.read(MetaReader.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.loadImage(Env.java:1701) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:130) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
  1. Paimon File Catalog meta issue

Fix#23366

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morningman

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morningman

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morningman

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.15 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162211193 Bytes

morningman added a commit that referenced this pull request Sep 20, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguoleiyiguolei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 21, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@yiguolei
yiguolei merged commit 5e82aeb into apache:masterSep 21, 2023
xiaokang pushed a commit that referenced this pull request Sep 24, 2023
@xiaokangxiaokang mentioned this pull request Sep 28, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.2-mergedp0_breviewed

Projects

None yet

4 participants

@morningman@doris-robot@yiguolei@zddr