[refactor](libhdfs) introduce hadoop libhdfs by morningman · Pull Request #17482 · apache/doris

morningman · 2023-03-07T02:15:49Z

Proposed changes

Issue Number: close #xxx

Problem summary

Introduce hadoop libhdfs
For Linux-X86 platform, use the hadoop libhdfs
For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform

Still WIP, this PR is just for review.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-03-07T02:16:47Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

morningman · 2023-03-07T02:23:18Z

+
+#pragma once
+
+#if defined(__x86_64__)


Should also check system os?

morningman · 2023-03-07T02:23:48Z

 Status HDFSStorageBackend::list(const std::string& remote_path, bool contain_md5, bool recursion,
 std::map<std::string, FileStat>* files) {
 CHECK_HDFS_CLIENT(_hdfs_fs);
+#if 0


Comment out temporarily, this class will be refactored

github-actions · 2023-03-27T08:24:24Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-27T13:39:32Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-27T13:49:00Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2023-03-28T07:00:40Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

…#17202)" This reverts commit 1a549ed.

github-actions · 2023-03-28T14:51:33Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In bin/start_be.sh line 100:
export CLASSPATH="${DORIS_HOME}/conf/:$DORIS_CLASSPATH"
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
In bin/start_be.sh line 262:
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$jvm_arch/server:$JAVA_HOME/jre/lib/$jvm_arch:$LD_LIBRARY_PATH
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^-------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/${jvm_arch}/server:${JAVA_HOME}/jre/lib/${jvm_arch}:${LD_LIBRARY_PATH}
In bin/start_be.sh line 263:
export LD_LIBRARY_PATH=$DORIS_HOME/lib/hadoop_hdfs/native:$LD_LIBRARY_PATH
^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
Did you mean: export LD_LIBRARY_PATH=${DORIS_HOME}/lib/hadoop_hdfs/native:${LD_LIBRARY_PATH}
In bin/start_be.sh line 264:
export LIBHDFS_OPTS="${JAVA_OPTS}"
^----------^ SC2154 (warning): JAVA_OPTS is referenced but not assigned.
In bin/start_be.sh line 269:
echo "CLASSPATH: ${CLASSPATH}\n"
^-------------------------^ SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 270:
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
In bin/start_be.sh line 271:
echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}\n"
^-- SC2028 (info): echo may not expand escape sequences. Use printf.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- JAVA_OPTS is referenced but not a...
https://www.shellcheck.net/wiki/SC2028 -- echo may not expand escape sequen...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around vari...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

morningman · 2023-03-29T05:53:03Z

see #18204

morningman marked this pull request as draft March 7, 2023 02:15

morningman commented Mar 7, 2023
View reviewed changes

morningman force-pushed the intro_libhdfs branch from 186ae2a to 991f5e0Compare March 27, 2023 08:23

github-actionsBot added area/planner Issues or PRs related to the query planner area/vectorization labels Mar 27, 2023

morningman force-pushed the intro_libhdfs branch from 991f5e0 to 49f8437Compare March 27, 2023 13:38

morningman force-pushed the intro_libhdfs branch from 49f8437 to 8ff52f2Compare March 27, 2023 13:47

morningman added 3 commits March 28, 2023 08:58

remove file utils compile pass ut pass
904a952

fix ut
f915b44

1. intro libhdfs
b8408bb

morningman force-pushed the intro_libhdfs branch from 8ff52f2 to b8408bbCompare March 28, 2023 06:59

morningman added 2 commits March 28, 2023 19:05

Revert "[Chore](third-party) upgrade thrift from 0.13 to 0.16 (apache…
09115cc
…#17202)" This reverts commit 1a549ed.

fix meta tool
3f2d891

morningman closed this Mar 29, 2023

Navigation Menu

Uh oh!

[refactor](libhdfs) introduce hadoop libhdfs - #17482

[refactor](libhdfs) introduce hadoop libhdfs#17482
morningman wants to merge 5 commits into
apache:masterfrom
morningman:intro_libhdfs

morningman commented Mar 7, 2023

Uh oh!

github-actionsBot commented Mar 7, 2023

Uh oh!

morningmanMar 7, 2023

Uh oh!

morningmanMar 7, 2023

Uh oh!

github-actionsBot commented Mar 27, 2023

Uh oh!

github-actionsBot commented Mar 27, 2023

Uh oh!

github-actionsBot commented Mar 27, 2023

Uh oh!

github-actionsBot commented Mar 28, 2023

Uh oh!

github-actionsBot commented Mar 28, 2023

Uh oh!

morningman commented Mar 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant


		#pragma once

		#if defined(__x86_64__)

Uh oh!

Conversation

morningman commented Mar 7, 2023

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actionsBot commented Mar 7, 2023

sh-checker report

Uh oh!

morningmanMar 7, 2023

Choose a reason for hiding this comment

Uh oh!

morningmanMar 7, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 27, 2023

sh-checker report

Uh oh!

github-actionsBot commented Mar 27, 2023

sh-checker report

Uh oh!

github-actionsBot commented Mar 27, 2023

sh-checker report

Uh oh!

github-actionsBot commented Mar 28, 2023

sh-checker report

Uh oh!

github-actionsBot commented Mar 28, 2023

sh-checker report

Uh oh!

morningman commented Mar 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`