Uh oh!
There was an error while loading. Please reload this page.
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 - #1450
Conversation
Apache-HBase
commented
Apr 7, 2020
(!) A patch to the testing environment has been detected. |
2 similar comments
Apache-HBase
commented
Apr 7, 2020
(!) A patch to the testing environment has been detected. |
Apache-HBase
commented
Apr 7, 2020
(!) A patch to the testing environment has been detected. |
Apache-HBase
commented
Apr 7, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 7, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 7, 2020
🎊 +1 overall
This message was automatically generated. |
saintstack
commented
Apr 7, 2020
Will merge this to master in morning unless objection. |
| ## @stability evolving | ||
| function hadoopcheck_docker_support | ||
| { | ||
| DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "--ulimit" "nproc=12500") |
There was a problem hiding this comment.
I don't think this should be here. we want this proc limit change all the time, not just when the hadoopcheck test is enabled.
There was a problem hiding this comment.
I don't follow. The new 12500 number is put everywhere, not just here conditional on when hadoopcheck runs. Thanks for taking a look.
There was a problem hiding this comment.
the reference you've added here is in the function that defines what is needed when we run docker specifically just when we're doing the hadoopcheck test.
if we need it all the time, then it should be elsewhere. It looks it should be coming in from the YETUS_ARGS that have been added to the wrapper. If that's the case, then removing it here should have no impact on what happens. If removing it here makes it so things don't do what you expect, then something else is broken because the inclusion here is only going to happen when the hadoopcheck test is enabled.
There was a problem hiding this comment.
Let me push new patch w/ your change. Thanks @busbey
saintstack
commented
Apr 8, 2020
Address @busbey comment. |
Apache-HBase
commented
Apr 8, 2020
(!) A patch to the testing environment has been detected. |
2 similar comments
Apache-HBase
commented
Apr 8, 2020
(!) A patch to the testing environment has been detected. |
Apache-HBase
commented
Apr 8, 2020
(!) A patch to the testing environment has been detected. |
Apache-HBase
commented
Apr 8, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 8, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 8, 2020
🎊 +1 overall
This message was automatically generated. |
| flaky_args=("${flaky_args[@]}" --urls "${JENKINS_URL}/job/HBase-Flaky-Tests/job/${BRANCH_NAME}" --is-yetus False --max-builds 30) | ||
| docker build -t hbase-dev-support dev-support | ||
| docker run -v "${WORKSPACE}":/hbase --workdir=/hbase hbase-dev-support python dev-support/flaky-tests/report-flakies.py --mvn -v "${flaky_args[@]}" | ||
| docker run --ulimit nproc=12500 -v "${WORKSPACE}":/hbase --workdir=/hbase hbase-dev-support python dev-support/flaky-tests/report-flakies.py --mvn -v "${flaky_args[@]}" |
There was a problem hiding this comment.
Yikes. This is just hiding over here in the corner. TODO: update flaky reporting to use yetus to run builds.
There was a problem hiding this comment.
Yeah, one way to run all builds would be sweet.
| @@ -159,4 +159,4 @@ done | |||
| echo "Successfully built ${IMAGE_NAME}." | |||
There was a problem hiding this comment.
Oh no, I didn't know this script existed. I probably broke it with the improvements that went in with HBASE-23945.
| # Up it. See HBASE-19902 for how we arrived at this number. | ||
| # NOTE: I don't think changing this has an effect. Set the | ||
| # --proclimit passed to yetus. This seems to do what we | ||
| # need changing proclimit. |
There was a problem hiding this comment.
Ah, yeah. In yetus, the env variable is PROC_LIMIT. You found a bug :)
There was a problem hiding this comment.
Let me try filing an issue.
| YETUS_ARGS=("--ignore-unknown-options=true" "${YETUS_ARGS[@]}") | ||
| # Why are these not being picked up from hbase-personality? | ||
| YETUS_ARGS=("--proclimit=10000" "${YETUS_ARGS[@]}") | ||
| YETUS_ARGS=("--proclimit=12500" "${YETUS_ARGS[@]}") |
There was a problem hiding this comment.
If the above change for PROC_LIMIT works, this line can be dropped.
There was a problem hiding this comment.
Yeah. For now they needed. Let me file an issue.
| # changing these to higher values may cause problems | ||
| # with other jobs on systemd-enabled machines | ||
| YETUS_ARGS+=("--proclimit=10000") | ||
| YETUS_ARGS+=("--proclimit=12500") |
Signed-off-by: Reid Chan <reidchan@apache.org> Signed-off-by: Sean Busbey <busbey@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
saintstack
commented
Apr 11, 2020
After push, I went to file issue against YETUS and noticed YETUS expects PROC_LIMIT, not PROCLIMIT as I have in our personality (an old error but relevant here). I reverted. Let me see if using PROC_LIMIT instead helps here. New PR coming. |
ndimiduk
commented
Apr 11, 2020
Oops. I meant to say that you found a bug in our personality, not in Yetus. That wasn't clear in my earlier comment. |
saintstack
commented
Apr 13, 2020
bq. Oops. I meant to say that you found a bug in our personality, not in Yetus. That wasn't clear in my earlier comment. Took me a while but now I get what you meant. |
…e#1450) Signed-off-by: Reid Chan <reidchan@apache.org> Signed-off-by: Sean Busbey <busbey@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
…e#1450) Signed-off-by: Reid Chan <reidchan@apache.org> Signed-off-by: Sean Busbey <busbey@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
…e#1450) Signed-off-by: Reid Chan <reidchan@apache.org> Signed-off-by: Sean Busbey <busbey@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
…e#1450) Signed-off-by: Reid Chan <reidchan@apache.org> Signed-off-by: Sean Busbey <busbey@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
No description provided.