Skip to content

Backport "HBASE-28686 MapReduceBackupCopyJob should support custom DistCp options (#6017)" to branch-2 - #6093

Closed
ndimiduk wants to merge 1 commit into
apache:branch-2from
ndimiduk:28686-branch-2
Closed

Backport "HBASE-28686 MapReduceBackupCopyJob should support custom DistCp options (#6017)" to branch-2#6093
ndimiduk wants to merge 1 commit into
apache:branch-2from
ndimiduk:28686-branch-2

Conversation

@ndimiduk

@ndimidukndimiduk commented Jul 18, 2024

Copy link
Copy Markdown
Member

No description provided.

@rmdmattinglyRay Mattingly@ndimiduk
HBASE-28686 MapReduceBackupCopyJob should support custom DistCp optio…
099d48f
…ns (apache#6017)
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@ndimidukndimiduk added the backport This PR is a back port of some issue or issues already committed to master label Jul 18, 2024
@Apache-HBase

Apache-HBase commented Jul 18, 2024

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 39sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall2m 32sbranch-2 passed
+1 💚compile0m 16sbranch-2 passed
+1 💚javadoc0m 13sbranch-2 passed
+1 💚shadedjars5m 17sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
-1 ❌mvninstall2m 24s/patch-mvninstall-root.txtroot in the patch failed.
-1 ❌compile0m 15s/patch-compile-hbase-backup.txthbase-backup in the patch failed.
-0 ⚠️javac0m 15s/patch-compile-hbase-backup.txthbase-backup in the patch failed.
+1 💚javadoc0m 12sthe patch passed
+1 💚shadedjars5m 19spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 ❌unit0m 15s/patch-unit-hbase-backup.txthbase-backup in the patch failed.
18m 27s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#6093
Optional Testsjavac javadoc unit compile shadedjars
unameLinux f7adbd582070 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 099d48f
Default JavaTemurin-1.8.0_412-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/testReport/
Max. process+thread count67 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 18, 2024

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec1m 4sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ branch-2 Compile Tests _
+1 💚mvninstall4m 29sbranch-2 passed
+1 💚compile0m 36sbranch-2 passed
+1 💚checkstyle0m 16sbranch-2 passed
+1 💚spotbugs0m 43sbranch-2 passed
+1 💚spotless0m 57sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall3m 27sthe patch passed
+1 💚compile0m 30sthe patch passed
+1 💚javac0m 30sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 10sthe patch passed
+1 💚spotbugs0m 45sthe patch passed
-1 ❌hadoopcheck3m 9sThe patch causes 20 errors with Hadoop v2.10.2.
+1 💚spotless1m 6spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
33m 18s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6093
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux 32accbcdce6e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 099d48f
Default JavaEclipse Adoptium-11.0.23+9
hadoopcheckhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-general-check/output/patch-javac-2.10.2.txt
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 18, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 51sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall4m 35sbranch-2 passed
+1 💚compile0m 44sbranch-2 passed
+1 💚javadoc0m 23sbranch-2 passed
+1 💚shadedjars6m 44sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall3m 47sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚javadoc0m 16sthe patch passed
+1 💚shadedjars6m 9spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit11m 37shbase-backup in the patch passed.
36m 48s
SubsystemReport/Notes
DockerClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#6093
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 7a570803db05 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 099d48f
Default JavaEclipse Adoptium-11.0.23+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/testReport/
Max. process+thread count3824 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 18, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 53sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall4m 50sbranch-2 passed
+1 💚compile0m 34sbranch-2 passed
+1 💚javadoc0m 21sbranch-2 passed
+1 💚shadedjars6m 41sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall4m 31sthe patch passed
+1 💚compile0m 22sthe patch passed
+1 💚javac0m 22sthe patch passed
+1 💚javadoc0m 21sthe patch passed
+1 💚shadedjars6m 38spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit14m 34shbase-backup in the patch passed.
40m 54s
SubsystemReport/Notes
DockerClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6093
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 37c6f3b538e5 5.4.0-177-generic #197-Ubuntu SMP Thu Mar 28 22:45:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 099d48f
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/testReport/
Max. process+thread count2993 (vs. ulimit of 30000)
modulesC: hbase-backup U: hbase-backup
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk

ndimiduk commented Jul 18, 2024

Copy link
Copy Markdown
MemberAuthor

@rmdmattingly it looks like we need an addendum for Hadoop2. That version of DistCP does not have the constant DistCpConstants#CONF_LABEL_DIRECT_WRITE and indeed may not support the feature.

@rmdmattingly

rmdmattingly commented Jul 19, 2024

Copy link
Copy Markdown
Contributor

I opened up this replacement that should build: #6102

@ndimiduk

ndimiduk commented Jul 22, 2024

Copy link
Copy Markdown
MemberAuthor

Superseded by #6102.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backportThis PR is a back port of some issue or issues already committed to master

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ndimiduk@Apache-HBase@rmdmattingly