Uh oh!
There was an error while loading. Please reload this page.
YARN-8470. Fix a NPE in identifyContainersToPreemptOnNode() - #416
Closed
gg7 wants to merge 1 commit into
Closed
Conversation
I encountered this issue while running 3.1.0: ``` 2018-09-10 13:42:39,437 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: Container container_1536156801471_0071_01_000055 completed with event FINISHED, but corresponding RMContainer doesn't exist. 2018-09-10 13:42:39,881 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread, FSPreemptionThread, that exited unexpectedly: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.identifyContainersToPreemptOnNode(FSPreemptionThread.java:207) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.identifyContainersToPreemptForOneContainer(FSPreemptionThread.java:161) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.identifyContainersToPreempt(FSPreemptionThread.java:121) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.run(FSPreemptionThread.java:81) 2018-09-10 13:42:39,886 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Shutting down the resource manager. 2018-09-10 13:42:39,891 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: a critical thread, FSPreemptionThread, that exited unexpectedly: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.identifyContainersToPreemptOnNode(FSPreemptionThread.java:207) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.identifyContainersToPreemptForOneContainer(FSPreemptionThread.java:161) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.identifyContainersToPreempt(FSPreemptionThread.java:121) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSPreemptionThread.run(FSPreemptionThread.java:81) ``` I'm guessing a better fix would be to synchronise the removal of applications, but this simple patch should be an improvement IMO. Signed-off-by: George G <git@gg7.io>
hadoop-yetus
commented
Jul 19, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Jul 26, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 2, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 8, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 16, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 20, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 22, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Aug 27, 2019
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
commented
Sep 3, 2019
💔 -1 overall
This message was automatically generated. |
sjrand
reviewed
Oct 8, 2019
| if (app == null) { | ||
| // e.g. "INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: Container container_1536156801471_0071_01_000096 completed with event FINISHED, but corresponding RMContainer doesn't exist." | ||
| LOG.warn("app == null, giving up in identifyContainersToPreemptOnNode()"); | ||
| return null; |
There was a problem hiding this comment.
Should we just continue instead of returning null since we might still be able to find preemptable containers on this node?
shanthoosh added a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
When zookeeper session failures occur in a stream processor, leaves the group(zkClient is closed) and joins the group again. The last step in that shutdown sequence is zkClient.close(). In some scenarios, it throws the following exception, org.I0Itec.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException at org.I0Itec.zkclient.ZkClient.close(ZkClient.java:1278) at org.apache.samza.zk.ZkControllerImpl.stop(ZkControllerImpl.java:92) at org.apache.samza.zk.ZkJobCoordinator.stop(ZkJobCoordinator.java:141) In existing implementation this is not handled, there by killing the stream processor. The following codepath triggers this exception: `StreamProcessor.stop -> ZkJobCoordinator.stop() -> zkController.stop() -> zkUtils.close` This exception causes the integration test to fail occasionally and can cause LocalApplicationRunner.waitForFinish method call to block indefinitely(since this callback event success, updates the latch state required for waitForFinish to end). Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com> Reviewers: Jagadish <jagadish@apache.org> Closesapache#416 from shanthoosh/zk_utils_close
Contributor
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I encountered this issue while running 3.1.0:
I'm guessing a better fix would be to synchronise the removal of applications, but this simple patch should be an improvement IMO.