[Improve](metadata)Start the script to set metadata_failure_recovery by CalvinKirs · Pull Request #24308 · apache/doris

CalvinKirs · 2023-09-13T09:09:54Z

Backend

Many people set meta_recovery in fe.conf, but often forget to change it back after doing it, so fe keeps using it like this, and as a result, problems with the metadata later occur.

We can add this parameter to start_fe.sh --metadata_failure_recovery to do this. In this way, if the user starts fe at the same time, it will automatically fail next time.

Changes

allowed use start_fe.sh -- metadata_failure_recovery or start_fe.sh --daemon -- metadata_failure_recovery
Setting up metadata recovery via configuration files is no longer allowed

Test

2023-09-13 17:30:05,808 INFO (main|1) [BDBEnvironment.setup():101] start group reset
2023-09-13 17:30:06,913 INFO (MASTER fe_38192775_8511_425f_9d31_39aaf9f69841(-1)|1) [BDBEnvironment.setup():105] group has been reset.

github-actions · 2023-09-13T09:10:55Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_fe.sh line 229:
nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${METADATA_FAILURE_RECOVERY} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
^--------------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
Did you mean: nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
In bin/start_fe.sh line 232:
${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} ${METADATA_FAILURE_RECOVERY} "$@" </dev/null
^--------------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
Did you mean: ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
For more information:
https://www.shellcheck.net/wiki/SC2248 -- Prefer double quoting even when v...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:
----------
--- bin/start_fe.sh.orig
+++ bin/start_fe.sh
@@ -55,7 +55,7 @@
;;
--metadata_failure_recovery)
METADATA_FAILURE_RECOVERY="-r"
- shift + shift
;;
--helper)
HELPER="$2"
@@ -226,10 +226,10 @@
echo "Internal Error. USE IMAGE_TOOL like : ./start_fe.sh --image image_path"
fi
elif [[ "${RUN_DAEMON}" -eq 1 ]]; then
- nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${METADATA_FAILURE_RECOVERY} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
+ nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${METADATA_FAILURE_RECOVERY} "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
else
export DORIS_LOG_TO_STDERR=1
- ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} ${METADATA_FAILURE_RECOVERY} "$@" </dev/null
+ ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} ${METADATA_FAILURE_RECOVERY} "$@" </dev/null
fi
echo $! >"${pidfile}"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename

CalvinKirs · 2023-09-13T09:34:46Z

run buildall

github-actions · 2023-09-13T09:35:25Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors

'shellcheck ' found no issues.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:
----------
--- bin/start_fe.sh.orig
+++ bin/start_fe.sh
@@ -55,7 +55,7 @@
;;
--metadata_failure_recovery)
METADATA_FAILURE_RECOVERY="-r"
- shift + shift
;;
--helper)
HELPER="$2"
@@ -226,10 +226,10 @@
echo "Internal Error. USE IMAGE_TOOL like : ./start_fe.sh --image image_path"
fi
elif [[ "${RUN_DAEMON}" -eq 1 ]]; then
- nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
+ nohup ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} "${METADATA_FAILURE_RECOVERY}" "$@" >>"${LOG_DIR}/fe.out" 2>&1 </dev/null &
else
export DORIS_LOG_TO_STDERR=1
- ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
+ ${LIMIT:+${LIMIT}} "${JAVA}" ${final_java_opt:+${final_java_opt}} -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError="kill -9 %p" org.apache.doris.DorisFE ${HELPER:+${HELPER}} ${OPT_VERSION:+${OPT_VERSION}} "${METADATA_FAILURE_RECOVERY}" "$@" </dev/null
fi
echo $! >"${pidfile}"
----------
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename

yiguolei

LGTM

github-actions · 2023-09-13T09:42:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-13T09:42:57Z

PR approved by anyone and no changes requested.

CalvinKirs · 2023-09-13T09:57:26Z

run buildall

zy-kkk

LGTM

doris-robot · 2023-09-13T10:07:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46 seconds
stream load tsv: 595 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162208254 Bytes

sollhui

LGTM

yiguolei

LGTM

yiguolei · 2023-09-14T01:14:23Z

run clickbench

yiguolei · 2023-09-14T01:15:09Z

run ckbench

doris-robot · 2023-09-14T01:54:43Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.58 seconds
stream load tsv: 605 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162244970 Bytes

…24308)

CalvinKirs force-pushed the metadata-recover branch from c3b603f to 74b92ffCompare September 13, 2023 09:34

yiguolei previously approved these changes Sep 13, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 13, 2023

github-actionsBot added the reviewed label Sep 13, 2023

CalvinKirs dismissed yiguolei’s stale review via 808b394September 13, 2023 09:54

[Improve](metadata)Start the script to set metadata_failure_recovery
c672779

CalvinKirs force-pushed the metadata-recover branch from 808b394 to c672779Compare September 13, 2023 09:55

zy-kkk approved these changes Sep 13, 2023
View reviewed changes

sollhui approved these changes Sep 13, 2023
View reviewed changes

yiguolei added the dev/2.0.2 label Sep 14, 2023

yiguolei approved these changes Sep 14, 2023
View reviewed changes

yiguolei added the kind/behavior-changed label Sep 14, 2023

yiguolei merged commit 64337a8 into apache:masterSep 14, 2023

CalvinKirs deleted the metadata-recover branch September 14, 2023 02:49

xiaokang added merge_conflict dev/2.0.2-merged and removed dev/2.0.2 labels Sep 14, 2023

xiaokang pushed a commit that referenced this pull request Sep 14, 2023
[Improve](metadata)Start the script to set metadata_failure_recovery (#…
ae9d176
…24308)

xiaokang mentioned this pull request Sep 28, 2023
Release Note 2.0.2 #25011
Closed

xiaokang mentioned this pull request Dec 3, 2023
Release Note 2.0.3 #27909
Closed

xiaokang mentioned this pull request Dec 28, 2023
[fix](compatibility) add back metadata_failure_recovery config for compatibility #29206
Merged

Uh oh!

Conversation

CalvinKirs commented Sep 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backend

Changes

Test

Uh oh!

github-actionsBot commented Sep 13, 2023

sh-checker report

Uh oh!

CalvinKirs commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

sh-checker report

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

CalvinKirs commented Sep 13, 2023

Uh oh!

zy-kkk left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

sollhui left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei commented Sep 14, 2023

Uh oh!

yiguolei commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CalvinKirs commented Sep 13, 2023•
edited
Loading

`sh-checker report`

`sh-checker report`