Skip to content

HDFS-14738：Reading and writing HDFS data through HDFS nfs3 server often getting stuck - #1300

Closed
alexkingli wants to merge 2 commits into
apache:trunkfrom
alexkingli:alex_dev_branch
Closed

HDFS-14738：Reading and writing HDFS data through HDFS nfs3 server often getting stuck#1300
alexkingli wants to merge 2 commits into
apache:trunkfrom
alexkingli:alex_dev_branch

Conversation

@alexkingli

@alexkinglialexkingli commented Aug 15, 2019

Copy link
Copy Markdown

If the user group does not exist, there will be a lot of invoking the updateMapIncr . And it is the synchronized methods，it really affects performance.

@alexkinglialexkingli changed the title update updateMapIncr to avoid loading information form os every timeHDFS-14738：Reading and writing HDFS data through HDFS nfs3 server often getting stuckAug 15, 2019
@hadoop-yetus

hadoop-yetus commented Aug 16, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec45Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
-1test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1mvninstall1066trunk passed
+1compile1070trunk passed
+1checkstyle40trunk passed
+1mvnsite79trunk passed
+1shadedclient837branch has no errors when building and testing our client artifacts.
+1javadoc60trunk passed
0spotbugs123Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs120trunk passed
_ Patch Compile Tests _
+1mvninstall48the patch passed
+1compile1091the patch passed
+1javac1091the patch passed
+1checkstyle40the patch passed
+1mvnsite81the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedclient648patch has no errors when building and testing our client artifacts.
+1javadoc63the patch passed
+1findbugs134the patch passed
_ Other Tests _
-1unit507hadoop-common in the patch failed.
+1asflicense40The patch does not generate ASF License warnings.
6024
ReasonTests
Failed junit testshadoop.security.TestShellBasedIdMapping
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/artifact/out/Dockerfile
GITHUB PR#1300
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
unameLinux 7e2dab2da4c1 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 9b8359b
Default Java1.8.0_222
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/testReport/
Max. process+thread count1417 (vs. ulimit of 5500)
modulesC: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/1/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

hadoop-yetus commented Aug 20, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec551Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author1The patch does not contain any @author tags.
-1test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1mvninstall1133trunk passed
+1compile1049trunk passed
+1checkstyle36trunk passed
+1mvnsite72trunk passed
+1shadedclient735branch has no errors when building and testing our client artifacts.
+1javadoc55trunk passed
0spotbugs111Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs110trunk passed
_ Patch Compile Tests _
+1mvninstall53the patch passed
+1compile1020the patch passed
+1javac1020the patch passed
+1checkstyle35the patch passed
+1mvnsite64the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedclient624patch has no errors when building and testing our client artifacts.
+1javadoc55the patch passed
+1findbugs123the patch passed
_ Other Tests _
-1unit512hadoop-common in the patch failed.
+1asflicense39The patch does not generate ASF License warnings.
6325
ReasonTests
Failed junit testshadoop.security.TestShellBasedIdMapping
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/artifact/out/Dockerfile
GITHUB PR#1300
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
unameLinux 3ea77b56f5e7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 6244502
Default Java1.8.0_222
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/testReport/
Max. process+thread count1380 (vs. ulimit of 5500)
modulesC: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/2/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

hadoop-yetus commented Aug 22, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec41Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
-1test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1mvninstall1125trunk passed
+1compile1239trunk passed
+1checkstyle39trunk passed
+1mvnsite75trunk passed
+1shadedclient765branch has no errors when building and testing our client artifacts.
+1javadoc60trunk passed
0spotbugs120Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs118trunk passed
_ Patch Compile Tests _
+1mvninstall43the patch passed
+1compile1050the patch passed
+1javac1050the patch passed
+1checkstyle39the patch passed
+1mvnsite72the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedclient625patch has no errors when building and testing our client artifacts.
+1javadoc56the patch passed
+1findbugs122the patch passed
_ Other Tests _
-1unit544hadoop-common in the patch failed.
+1asflicense39The patch does not generate ASF License warnings.
6111
ReasonTests
Failed junit testshadoop.security.TestShellBasedIdMapping
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/Dockerfile
GITHUB PR#1300
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
unameLinux 15eb07c9efaf 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / ee7c261
Default Java1.8.0_212
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/testReport/
Max. process+thread count1424 (vs. ulimit of 5500)
modulesC: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/3/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

hadoop-yetus commented Aug 27, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec57Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
-1test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1mvninstall1371trunk passed
+1compile1258trunk passed
+1checkstyle51trunk passed
+1mvnsite94trunk passed
+1shadedclient981branch has no errors when building and testing our client artifacts.
+1javadoc75trunk passed
0spotbugs150Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs148trunk passed
_ Patch Compile Tests _
+1mvninstall54the patch passed
+1compile1172the patch passed
+1javac1172the patch passed
+1checkstyle50the patch passed
+1mvnsite82the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedclient702patch has no errors when building and testing our client artifacts.
+1javadoc71the patch passed
+1findbugs130the patch passed
_ Other Tests _
-1unit532hadoop-common in the patch failed.
+1asflicense47The patch does not generate ASF License warnings.
6961
ReasonTests
Failed junit testshadoop.security.TestShellBasedIdMapping
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/artifact/out/Dockerfile
GITHUB PR#1300
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
unameLinux 3a08b1fd0c60 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 3329257
Default Java1.8.0_222
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/testReport/
Max. process+thread count1513 (vs. ulimit of 5500)
modulesC: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/4/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

hadoop-yetus commented Sep 3, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec97Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
-1test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1mvninstall1329trunk passed
+1compile1233trunk passed
+1checkstyle47trunk passed
+1mvnsite92trunk passed
+1shadedclient927branch has no errors when building and testing our client artifacts.
+1javadoc60trunk passed
0spotbugs141Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs139trunk passed
_ Patch Compile Tests _
+1mvninstall49the patch passed
+1compile1148the patch passed
+1javac1148the patch passed
+1checkstyle45the patch passed
+1mvnsite77the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedclient774patch has no errors when building and testing our client artifacts.
+1javadoc72the patch passed
+1findbugs133the patch passed
_ Other Tests _
-1unit571hadoop-common in the patch failed.
+1asflicense50The patch does not generate ASF License warnings.
6900
ReasonTests
Failed junit testshadoop.security.TestShellBasedIdMapping
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/artifact/out/Dockerfile
GITHUB PR#1300
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
unameLinux eb6d25b5071b 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 915cbc9
Default Java1.8.0_222
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/testReport/
Max. process+thread count1378 (vs. ulimit of 5500)
modulesC: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-1300/5/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

hadoop-yetus commented Jul 31, 2020

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 39sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚mvninstall20m 30strunk passed
+1 💚compile21m 31strunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚compile18m 35strunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚checkstyle0m 48strunk passed
+1 💚mvnsite1m 28strunk passed
+1 💚shadedclient16m 51sbranch has no errors when building and testing our client artifacts.
-1 ❌javadoc0m 39shadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚javadoc0m 59strunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗spotbugs2m 13sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs2m 11strunk passed
_ Patch Compile Tests _
+1 💚mvninstall0m 55sthe patch passed
+1 💚compile20m 4sthe patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚javac20m 4sthe patch passed
+1 💚compile17m 50sthe patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚javac17m 50sthe patch passed
+1 💚checkstyle0m 45sthe patch passed
+1 💚mvnsite1m 26sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedclient14m 6spatch has no errors when building and testing our client artifacts.
-1 ❌javadoc0m 40shadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚javadoc0m 57sthe patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚findbugs2m 29sthe patch passed
_ Other Tests _
-1 ❌unit9m 26shadoop-common in the patch passed.
+1 💚asflicense0m 50sThe patch does not generate ASF License warnings.
155m 35s
ReasonTests
Failed junit testshadoop.security.TestShellBasedIdMapping
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/Dockerfile
GITHUB PR#1300
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
unameLinux 139168d963b2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / e756fe3
Default JavaPrivate Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadochttps://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadochttps://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
unithttps://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Resultshttps://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/testReport/
Max. process+thread count3449 (vs. ulimit of 5500)
modulesC: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console outputhttps://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1300/3/console
versionsgit=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@github-actions

github-actionsBot commented Dec 20, 2025

Copy link
Copy Markdown
Contributor

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

HDFSStale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alexkingli@hadoop-yetus@jojochuang