[refactor](libhdfs) introduce hadoop libhdfs by morningman · Pull Request #18204 · apache/doris

morningman · 2023-03-29T05:52:43Z

Proposed changes

Issue Number: close #xxx

Problem summary

Introduce hadoop libhdfs
For Linux-X86 platform, use the hadoop libhdfs
For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform

Co-authored-by: adonis0147 adonis0147@gmail.com

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-03-29T05:53:40Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-29T14:53:00Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 101:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 257:
if [[ -z $JAVA_OPTS ]]; then
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: if [[ -z ${JAVA_OPTS} ]]; then
In bin/start_be.sh line 260:
JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: JAVA_OPTS="-Xmx1024m -DlogPath=${DORIS_HOME}/log/jni.log -Xloggc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
For more information:
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-29T14:55:01Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 101:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 257:
if [[ -z $JAVA_OPTS ]]; then
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: if [[ -z ${JAVA_OPTS} ]]; then
In bin/start_be.sh line 260:
JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: JAVA_OPTS="-Xmx1024m -DlogPath=${DORIS_HOME}/log/jni.log -Xloggc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
For more information:
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

github-actions · 2023-03-29T14:56:28Z

+ options = (JavaVMOption*)calloc(no_args, sizeof(JavaVMOption));
+ options[0].optionString = const_cast<char*>(classpath.c_str());
+ java_opts = getenv("JAVA_OPTS");
+ if (java_opts != NULL) {


warning: use nullptr [modernize-use-nullptr]
Suggested change
if (java_opts != NULL) {
if (java_opts != nullptr) {

morningman · 2023-03-29T16:10:36Z

run buildall

morningman · 2023-03-29T23:52:12Z

run buildall

adonis0147 · 2023-03-30T02:44:03Z

+
+ if (java_opts != nullptr) {
+ free(java_opts);
+ }
+ free(options);


It's better to use RAII.

morningman · 2023-03-30T05:43:14Z

run buildall

github-actions · 2023-03-30T05:43:39Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 101:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 259:
if [[ -z $JAVA_OPTS ]]; then
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: if [[ -z ${JAVA_OPTS} ]]; then
In bin/start_be.sh line 262:
JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: JAVA_OPTS="-Xmx1024m -DlogPath=${DORIS_HOME}/log/jni.log -Xloggc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
For more information:
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

morningman · 2023-03-30T05:51:14Z

run buildall

morningman · 2023-03-30T13:34:38Z

run buildall

github-actions · 2023-03-30T13:37:36Z

clang-tidy review says "All clean, LGTM! 👍"

adonis0147

LGTM

github-actions · 2023-03-30T13:54:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-03-30T13:54:40Z

PR approved by anyone and no changes requested.

adonis0147

LGTM

morningman · 2023-03-30T15:09:32Z

run buildall

github-actions · 2023-03-30T15:10:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-03-30T15:14:18Z

clang-tidy review says "All clean, LGTM! 👍"

1. Introduce hadoop libhdfs 2. For Linux-X86 platform, use the hadoop libhdfs 3. For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform Co-authored-by: adonis0147 <adonis0147@gmail.com>

github-actionsBot added the area/vectorization label Mar 29, 2023

morningman mentioned this pull request Mar 29, 2023
[refactor](libhdfs) introduce hadoop libhdfs #17482
Closed
5 tasks

morningman force-pushed the hadoop_hdfs branch from adf343e to 86c0311Compare March 29, 2023 14:53

github-actionsBot reviewed Mar 29, 2023
View reviewed changes

morningman changed the title ~~[Draft](libhdfs) introduce hadoop libhdfs~~[refactor](libhdfs) introduce hadoop libhdfsMar 30, 2023

adonis0147 reviewed Mar 30, 2023
View reviewed changes

morningman added 5 commits March 30, 2023 12:29

1. intro libhdfs
a0f4799

fix meta tool
216b1f9

fix jvm
baac148

fix libhdfs3
3c75e4b

fix review
1c51072

morningman force-pushed the hadoop_hdfs branch from 623b6f9 to 1c51072Compare March 30, 2023 05:42

fix shell
8adec90

fix jni
9d1ffb5

adonis0147 previously approved these changes Mar 30, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

github-actionsBot added the reviewed label Mar 30, 2023

1
0477c3b

morningman dismissed adonis0147’s stale review via 0477c3bMarch 30, 2023 15:08

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

adonis0147 approved these changes Mar 30, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

morningman merged commit 7e61a85 into apache:masterMar 31, 2023

morningman mentioned this pull request Apr 9, 2023
[refactor](libhdfs) introduce hadoop libhdfs #18495
Closed
5 tasks

Uh oh!

Conversation

morningman commented Mar 29, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actionsBot commented Mar 29, 2023

sh-checker report

Uh oh!

github-actionsBot commented Mar 29, 2023

sh-checker report

Uh oh!

github-actionsBot commented Mar 29, 2023

sh-checker report

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotMar 29, 2023

Choose a reason for hiding this comment

Uh oh!

morningman commented Mar 29, 2023

Uh oh!

morningman commented Mar 29, 2023

Uh oh!

adonis0147Mar 30, 2023

Choose a reason for hiding this comment

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

github-actionsBot commented Mar 30, 2023

sh-checker report

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

github-actionsBot commented Mar 30, 2023

Uh oh!

adonis0147 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 30, 2023

Uh oh!

github-actionsBot commented Mar 30, 2023

Uh oh!

adonis0147 left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Mar 30, 2023

Uh oh!

github-actionsBot commented Mar 30, 2023

Uh oh!

github-actionsBot commented Mar 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

morningman commented Mar 29, 2023•
edited
Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`