[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 300m to 340m) by HyukjinKwon · Pull Request #21845 · apache/spark

HyukjinKwon · 2018-07-23T05:35:16Z

What changes were proposed in this pull request?

Currently, looks we hit the time limit time to time. Looks better increasing the time a bit.

For instance, please see #21822

For clarification, current Jenkins timeout is 400m. This PR just proposes to fix the test script to increase it correspondingly.

This PR does not target to change the build configuration

How was this patch tested?

Jenkins tests.

HyukjinKwon · 2018-07-23T05:35:24Z

cc @rxin

SparkQA · 2018-07-23T07:05:01Z

Test build #93429 has finished for PR 21845 at commit c57b745.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-23T07:05:02Z

Test build #93430 has finished for PR 21845 at commit 7afc5c5.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-07-23T07:21:36Z

retest this please

…0m to 350m)

hvanhovell · 2018-07-23T08:53:16Z

@HyukjinKwon do we have any idea why we are hitting a timeout?

HyukjinKwon · 2018-07-23T08:56:33Z

I am not really sure on that. I asked the same question before and got no answer before. Just vaguely roughly guess there's something wrong in Jenkins cluster - I have roughly been kind of keen to check build time and to me seems suddenly increased (in some cases or some machines(?)).

HyukjinKwon · 2018-07-23T09:02:43Z

Just given observation for the builds in #21822, most of timeouts looked happened in amp-jenkins-worker-06 machine FWIW.

SparkQA · 2018-07-23T11:59:28Z

Test build #93436 has finished for PR 21845 at commit 7afc5c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2018-07-23T16:32:45Z

This helps, but it is not sustainable to keep increasing the threshold. What we need to do is to look at test time distribution and figure out what test suites are unnecessarily long and actually cut down the time there. @HyukjinKwon Would you be interested in doing that?

HyukjinKwon · 2018-07-23T17:00:09Z

of course i am as usual. I actually already have been being taking care of it. Thing is the tests are just being added even if they are duplicated of something. I feel like it's a bit excessive so far. In genetal, I don't think there are particular tests especially taking a lot of time IMHO. What we should do is that we put some efforts to deduplicate the tests.

HyukjinKwon · 2018-07-24T01:29:06Z

@rxin, btw you want me close this one or get this in? Will take a look for the build and tests thing again during this week for sure anyway.

rxin · 2018-07-24T01:36:19Z

Are more pull requests failing due to time out right now?

…

On Mon, Jul 23, 2018 at 6:30 PM Hyukjin Kwon ***@***.***> wrote: @rxin <https://github.com/rxin>, btw you want me close this one or get this in? Will take a look for the build and tests thing again during this week for sure anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21845 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATvPAz8b9kFd3V6puP3zYfyw-GSv2BGks5uJnimgaJpZM4VaY0E> .

HyukjinKwon · 2018-07-24T06:43:49Z

Yup, looks so in your PR #21822 (comment)

rxin · 2018-07-24T06:49:10Z

If that's the only one I think that PR itself needs to be fixed (significantly increases test runtime), and I wouldn't increase the time here.

…

On Mon, Jul 23, 2018 at 11:44 PM Hyukjin Kwon ***@***.***> wrote: Yup, looks so in your PR #21822 (comment) <#21822 (comment)> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21845 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATvPLusPAUegZcctlDTG2lWAJO1pjlDks5uJsJagaJpZM4VaY0E> .

HyukjinKwon · 2018-07-24T06:52:01Z

Hm, yea then. I actually opened this PR to make the tests passed in your PR. Let me leave this closed then and reopen when we hit the issue next time.

dilipbiswal · 2018-07-24T11:18:00Z

@HyukjinKwon I saw the following test run for 11 minutes on jenkins for one of my PR. Not sure if its a transient problem. Just thought, i should let you know. On the nightly runs, should we have test that runs for that long ?

SPARK-22499: Least and greatest should not generate codes beyond 64KB (11 minutes, 38 seconds)

HyukjinKwon · 2018-07-24T11:35:47Z

Ah, of course we shouldn't. I roughly checked few builds in #21822 and made a fix for the test #21855

Thanks for letting me know.

dilipbiswal · 2018-07-24T11:41:34Z

@HyukjinKwon Super. Thanks a lot for fixing.

HyukjinKwon · 2018-08-07T05:50:18Z

I am reopening this per #21898 (comment)

cc @cloud-fan, @rxin and @shaneknapp

cloud-fan · 2018-08-07T06:24:36Z

Ideally we should figure out which tests take abnormal long time and fix them. But I'd like to increase the timeout first, if #21898 keeps hitting timeout. #21898 is an important feature to Spark 2.4, and we should not block it for some infra problems. cc @rxin

SparkQA · 2018-08-07T07:05:01Z

Test build #94345 has finished for PR 21845 at commit 7afc5c5.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-08-07T07:12:06Z

retest this please

HyukjinKwon · 2018-08-07T07:18:16Z

I think we still intermediately meet this limit issue. For instance:

#22001 (comment)
#21923 (comment)
#21909 (comment)
#21777 (comment)
#21156 (comment)
#21109 (comment)
#20838 (comment)
#20345 (comment)

I saw here multiple times - #21991 (comment)

#21991 (comment)
#21930 (comment)
#21972 (comment)

SparkQA · 2018-08-07T11:46:57Z

Test build #94349 has finished for PR 21845 at commit 7afc5c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2018-08-07T16:52:16Z

i'm also more than happy to bump the timeout in the PRB build, but i think that's just putting duct tape on a band-aid and spray painting it to hide the layers of tape.

the builds and tests just take too long. i know that solving this problem is far beyond the scope of this PR, but build duration really needs some attention.

HyukjinKwon · 2018-08-07T23:49:35Z

 # format: http://linux.die.net/man/1/timeout
- # must be less than the timeout configured on Jenkins (currently 350m)
- tests_timeout = "300m"
+ # must be less than the timeout configured on Jenkins (currently 400m)


Updated by http://apache-spark-developers-list.1001551.n3.nabble.com/build-system-bumped-pull-request-builder-job-timeout-to-400mins-td24647.html

HyukjinKwon · 2018-08-08T02:06:37Z

Let me push this in late tonight or early tomorrow.

SparkQA · 2018-08-08T04:26:54Z

Test build #94396 has finished for PR 21845 at commit 08b4ebe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-08-10T01:09:47Z

I am getting this in. We are seeing more - #22011 (comment)

HyukjinKwon · 2018-08-10T01:11:56Z

Increased 330 -> 340 since even 330 looks not enough.

HyukjinKwon · 2018-08-10T01:12:42Z

Merged to master.

SparkQA · 2018-08-10T06:18:18Z

Test build #94533 has finished for PR 21845 at commit 51f8792.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2018-08-13T19:47:09Z

- # must be less than the timeout configured on Jenkins (currently 350m)
- tests_timeout = "300m"
+ # must be less than the timeout configured on Jenkins (currently 400m)
+ tests_timeout = "340m"


we're STILL seeing test timeouts. let's bump this to 400m and i'll up the timeout in jenkins to 430m.

Woah .. got it ..

…enkins build (from 340m to 400m) ## What changes were proposed in this pull request? This PR targets to increase the timeout from 340 to 400m. Please also see #21845 (comment) ## How was this patch tested? N/A Closes#22098 from HyukjinKwon/SPARK-24886-1. Authored-by: hyukjinkwon <gurwls223@apache.org> Signed-off-by: hyukjinkwon <gurwls223@apache.org>

HyukjinKwon changed the title ~~[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 300m to 350m)~~[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 300m to 330m)Jul 23, 2018

HyukjinKwon added 2 commits July 23, 2018 15:31

Fix the testing script to increase timeout for Jenkins build (from 30…
c8f5bcc
…0m to 350m)

Looks 330m good enough actually for now
d67cd40

HyukjinKwon closed this Jul 24, 2018

HyukjinKwon mentioned this pull request Jul 25, 2018
[SPARK-24624][SQL][PYTHON] Support mixture of Python UDF and Scalar Pandas UDF #21650
Closed

HyukjinKwon reopened this Aug 7, 2018

Now it's 400m
08b4ebe

HyukjinKwon force-pushed the SPARK-24886 branch from 7afc5c5 to 08b4ebeCompare August 7, 2018 23:49

HyukjinKwon commented Aug 7, 2018
View reviewed changes

330m -> 340m
51f8792

HyukjinKwon changed the title ~~[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 300m to 330m)~~[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 300m to 340m)Aug 10, 2018

asfgit closed this in 6c7bb57Aug 10, 2018

shaneknapp reviewed Aug 13, 2018
View reviewed changes

HyukjinKwon mentioned this pull request Aug 14, 2018
[SPARK-24886][INFRA] Fix the testing script to increase timeout for Jenkins build (from 340m to 400m) #22098
Closed

HyukjinKwon deleted the SPARK-24886 branch October 16, 2018 12:45

Uh oh!

Conversation

HyukjinKwon commented Jul 23, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Jul 23, 2018

Uh oh!

SparkQA commented Jul 23, 2018

Uh oh!

SparkQA commented Jul 23, 2018

Uh oh!

HyukjinKwon commented Jul 23, 2018

Uh oh!

hvanhovell commented Jul 23, 2018

Uh oh!

HyukjinKwon commented Jul 23, 2018

Uh oh!

HyukjinKwon commented Jul 23, 2018

Uh oh!

SparkQA commented Jul 23, 2018

Uh oh!

rxin commented Jul 23, 2018

Uh oh!

HyukjinKwon commented Jul 23, 2018

Uh oh!

HyukjinKwon commented Jul 24, 2018

Uh oh!

rxin commented Jul 24, 2018 via email

Uh oh!

HyukjinKwon commented Jul 24, 2018

Uh oh!

rxin commented Jul 24, 2018 via email

Uh oh!

HyukjinKwon commented Jul 24, 2018

Uh oh!

dilipbiswal commented Jul 24, 2018

Uh oh!

HyukjinKwon commented Jul 24, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dilipbiswal commented Jul 24, 2018

Uh oh!

HyukjinKwon commented Aug 7, 2018

Uh oh!

cloud-fan commented Aug 7, 2018

Uh oh!

SparkQA commented Aug 7, 2018

Uh oh!

HyukjinKwon commented Aug 7, 2018

Uh oh!

HyukjinKwon commented Aug 7, 2018

Uh oh!

SparkQA commented Aug 7, 2018

Uh oh!

shaneknapp commented Aug 7, 2018

Uh oh!

HyukjinKwonAug 7, 2018

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Aug 8, 2018

Uh oh!

SparkQA commented Aug 8, 2018

Uh oh!

HyukjinKwon commented Aug 10, 2018

Uh oh!

HyukjinKwon commented Aug 10, 2018

Uh oh!

HyukjinKwon commented Aug 10, 2018

Uh oh!

SparkQA commented Aug 10, 2018

Uh oh!

shaneknappAug 13, 2018

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonAug 14, 2018

Choose a reason for hiding this comment

Uh oh!

Reviewers

HyukjinKwon commented Jul 23, 2018•
edited
Loading

HyukjinKwon commented Jul 24, 2018•
edited
Loading