Skip to content

Test timeout issues - #683

Merged
mk-mxp merged 6 commits into
exercism:mainfrom
mk-mxp:fix-robot-name
Apr 21, 2024
Merged

Test timeout issues#683
mk-mxp merged 6 commits into
exercism:mainfrom
mk-mxp:fix-robot-name

Conversation

@mk-mxp

@mk-mxpmk-mxp commented Apr 14, 2024

Copy link
Copy Markdown
Contributor

PHPUnit 10.5 assertArrayNotHasKey()

PHPUnit ^10.3 has an issue with assertArrayNotHasKey(). Only this (and the opposite assertArrayHasKey()) are affected. The issue is not listed in PHPUnit's issue list / mentioned in Pull Requests, but the issue is gone in 11.0.0.

This contains a workaround for this and enables upgrading to PHPUnit 10.5. I replaced that assertion with a functionally equivalent one, but that prints some misleading output in addition to the helpful message. So it should be reverted when upgrading to PHPUnit 11.

CI limit for test runtime

I also added a time limit of 18 (lowest value possible on my machine with equal performance to Online Test Runner) x 3 (to compensate lower CI performance) = 54 seconds for each exercise test class using timeout command. This should prevent running into performance degradation issues in the future.

As discussed in the forum there is no recommendation or guiding data for such a limit. Only guesses about "test runner is expected to be slower than GitHub CI". But the production limit of 20 seconds exists and is enforced. So I took a guess based on my measurements.

Part of #652

@github-actions

Copy link
Copy Markdown
Contributor

This PR touches files which potentially affect the outcome of the tests of an exercise. This will cause all students' solutions to affected exercises to be re-tested.

If this PR does not affect the result of the test (or, for example, adds an edge case that is not worth rerunning all tests for), please add the following to the merge-commit message which will stops student's tests from re-running. Please copy-paste to avoid typos.

[no important files changed]

For more information, refer to the documentation. If you are unsure whether to add the message or not, please ping @exercism/maintainers-admin in a comment. Thank you!

@mk-mxpmk-mxp self-assigned this Apr 14, 2024
@mk-mxpmk-mxp added x:action/fix Fix an issue x:knowledge/intermediate Quite a bit of Exercism knowledge required x:type/ci Work on Continuous Integration (e.g. GitHub Actions workflows) x:type/content Work on content (e.g. exercises, concepts) x:size/medium Medium amount of work x:rep/medium Medium amount of reputation labels Apr 14, 2024
@mk-mxp
mk-mxp removed the request for review from homersimpsonsApril 14, 2024 14:24
@mk-mxp

Copy link
Copy Markdown
ContributorAuthor

@homersimpsons Sorry, I thought I had figured that out right. But I somehow haven't remembered things correctly...

@mk-mxpmk-mxp closed this Apr 14, 2024
@mk-mxpmk-mxp reopened this Apr 15, 2024
@mk-mxp

Copy link
Copy Markdown
ContributorAuthor

@homersimpsons Another attempt using measured data now. I put the data into the forum, and a longish comment into the test script.

Comment threadexercises/practice/robot-name/RobotNameTest.php
Comment threadbin/test.sh Outdated

@homersimpsonshomersimpsons left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your investigation. I'm proposing a new wording for the comment about the timeout. But I still approve the pull request as-is.

Comment threadbin/test.sh Outdated
Co-authored-by: homersimpsons <guillaume.alabre@gmail.com>
@mk-mxp
mk-mxp merged commit cd8a756 into exercism:mainApr 21, 2024
@mk-mxp
mk-mxp deleted the fix-robot-name branch April 21, 2024 13:14
tomasnorre pushed a commit to tomasnorre/exercism-php that referenced this pull request May 7, 2024
Co-authored-by: homersimpsons <guillaume.alabre@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

x:action/fixFix an issuex:knowledge/intermediateQuite a bit of Exercism knowledge requiredx:rep/mediumMedium amount of reputationx:size/mediumMedium amount of workx:type/ciWork on Continuous Integration (e.g. GitHub Actions workflows)x:type/contentWork on content (e.g. exercises, concepts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@mk-mxp@homersimpsons