Skip to content

bpo-11105: use a lower recursion limit for infinite recursion tests - #26550

Merged
isidentical merged 1 commit into
python:mainfrom
isidentical:bpo-11105-recursion-windows
Jun 8, 2021
Merged

bpo-11105: use a lower recursion limit for infinite recursion tests#26550
isidentical merged 1 commit into
python:mainfrom
isidentical:bpo-11105-recursion-windows

Conversation

@isidentical

@isidenticalisidentical commented Jun 5, 2021

Copy link
Copy Markdown
Member

@isidenticalisidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@isidentical

Copy link
Copy Markdown
MemberAuthor

@Fidget-Spinner does this patch prevent the crash for your end (do you have resources to test it out?). I highly doubt so (since these functions are giant I don't expect them to be inlined), though if it does then it would be good to know.

@Fidget-Spinner

Fidget-Spinner commented Jun 5, 2021

Copy link
Copy Markdown
Member

@Fidget-Spinner does this patch prevent the crash for your end (do you have resources to test it out?). I highly doubt so (since these functions are giant I don't expect them to be inlined), though if it does then it would be good to know.

Thanks for working on this! Yup I can test it out, I use MSVC often.

Your doubts were right, this is sadly still causing a stack overflow. BTW, testing with buildbots should catch it, the AMD64 Windows10 Pro 3.x and AMD64 Windows10 3.x buildbots both caught it the previous commit. For PR pushes, their names are AMD64 Windows10 Pro PR and AMD64 Windows10 PR (the workers are bolen-windows10 and kloth-win64). Both seem to have errored out though they haven't updated their statuses yet.

@isidentical

isidentical commented Jun 5, 2021

Copy link
Copy Markdown
MemberAuthor

Both seem to have errored out though they haven't updated their statuses yet.

Cool! I was worried that they are not available on the public list, but apparently, they are! Thanks for letting me know and testing this out.

@isidentical
isidenticalforce-pushed the bpo-11105-recursion-windows branch from b246c21 to 7f83161CompareJune 5, 2021 20:29
@isidenticalisidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@isidentical
isidenticalforce-pushed the bpo-11105-recursion-windows branch from 7f83161 to d7b496fCompareJune 5, 2021 20:47
@isidenticalisidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@isidentical
isidenticalforce-pushed the bpo-11105-recursion-windows branch from d7b496f to f31c841CompareJune 5, 2021 22:39
@isidenticalisidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @isidentical for commit f31c8411cc51590aee9bc8c2b3042911316b546e 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021
@pythonpython deleted a comment from bedevere-botJun 5, 2021
@pythonpython deleted a comment from bedevere-botJun 5, 2021
@pythonpython deleted a comment from bedevere-botJun 5, 2021
@isidentical

Copy link
Copy Markdown
MemberAuthor

Seems like the error in test_ast doesn't happen anymore though we still get a stack overflow from somewhere else which doesn't make much sense since there is no ast compilation related cover on that test.

test_recursion_in_except_handler (test.test_exceptions.ExceptionTests) ... D:\buildarea\3.x.bolen-windows10\build>exit /b -1073741571 Windows fatal exception: stack overflow
Current thread 0x00000a54 (most recent call first):
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_exceptions.py", line 1132 in test_recursion_in_except_handler
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\case.py", line 549 in _callTestMethod
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\case.py", line 592 in run
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\case.py", line 652 in __call__
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 122 in run
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 84 in __call__
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 122 in run
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 84 in __call__
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 122 in run
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 84 in __call__
File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\runner.py", line 176 in run
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\support\__init__.py", line 960 in _run_suite
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\support\__init__.py", line 1083 in run_unittest
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 210 in _test_module
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 154 in _runtest
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 194 in runtest
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 321 in rerun_failed_tests
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 698 in _main
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 641 in main
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 719 in main
File "D:\buildarea\3.x.bolen-windows10\build\lib\test\__main__.py", line 2 in <module>
File "D:\buildarea\3.x.bolen-windows10\build\lib\runpy.py", line 86 in _run_code
File "D:\buildarea\3.x.bolen-windows10\build\lib\runpy.py", line 196 in _run_module_as_main

@Fidget-Spinner

Copy link
Copy Markdown
Member

Seems like the error in test_ast doesn't happen anymore though we still get a stack overflow from somewhere else which doesn't make much sense since there is no ast compilation related cover on that test.

test_recursion_in_except_handler (test.test_exceptions.ExceptionTests) ... 

I just checked on main branch, this specific test also causes a stack overflow there. So maybe that test failure was previously masked by this one. Your patch fixes the AST stack overflow stack overflow issue for windows. Thanks!

Something I found strange is that the exceptions recursion test was added in 4e7a69b but it passed all along. I'm starting to suspect maybe there's some other change elsewhere that caused both test_exception and test_ast to fail. I'll start bisecting with test_exception soon. So please wait before merging this PR, thanks!

@Fidget-Spinner

Fidget-Spinner commented Jun 6, 2021

Copy link
Copy Markdown
Member

Ok, bisect tells me that this commit caused test_exception to fail:

commit f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970
[bpo-39573](https://bugs.python.org/issue39573): Py_TYPE becomes a static inline function (GH-26493)
Convert the Py_TYPE() and Py_SIZE() macros to static inline
functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be
used to set an object type and size.

@isidentical
isidenticalforce-pushed the bpo-11105-recursion-windows branch from f31c841 to 1bf3518CompareJune 8, 2021 11:08
@isidenticalisidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @isidentical for commit 1bf3518db90f3e1c6c8eb1d8daf925e0ae5ee649 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021
@isidenticalisidentical changed the title bpo-11105: [WIP] reduce stack usagebpo-11105: use a lower recursion limit for infinite recursion testsJun 8, 2021
@Fidget-Spinner

Copy link
Copy Markdown
Member

@isidentical I think if you rebase to the latest 6d518bb, test with buildbots should mostly pass. commit f3fa63e was reverted.

@isidentical
isidenticalforce-pushed the bpo-11105-recursion-windows branch from 1bf3518 to e01e32dCompareJune 8, 2021 11:29
@isidenticalisidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @isidentical for commit e01e32d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021

@Fidget-SpinnerFidget-Spinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The windows 10 tests are passing. Thanks for your efforts. Without your help I would've taken a lot longer to track down the other windows buildbot failures.

testcase.assertRaisesRegex(TypeError, msg, tp, *args, **kwds)

@contextlib.contextmanager
def infinite_recursion(max_depth=75):

@Fidget-SpinnerFidget-SpinnerJun 8, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor suggestion: I think adding a docstring would help others know where/why/when they need this.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, missed this one! Yeap, forgot to do that. Will add a follow up PR.

@isidentical
isidentical marked this pull request as ready for review June 8, 2021 16:54
@isidentical
isidentical merged commit e58d762 into python:mainJun 8, 2021
@miss-islington

Copy link
Copy Markdown
Contributor

Thanks @isidentical for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

@miss-islington

Copy link
Copy Markdown
Contributor

Thanks @isidentical for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

@miss-islington

Copy link
Copy Markdown
Contributor

Sorry, @isidentical, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker e58d762c1fb4ad5e021d016c80c2bc4513632d2f 3.9

@miss-islington

Copy link
Copy Markdown
Contributor

Sorry @isidentical, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker e58d762c1fb4ad5e021d016c80c2bc4513632d2f 3.10

isidentical added a commit to isidentical/cpython that referenced this pull request Jun 8, 2021
(cherry picked from commit e58d762)
Co-authored-by: Batuhan Taskaya <batuhan@python.org>
@bedevere-bot

Copy link
Copy Markdown

GH-26605 is a backport of this pull request to the 3.9 branch.

isidentical added a commit to isidentical/cpython that referenced this pull request Jun 8, 2021
(cherry picked from commit e58d762)
Co-authored-by: Batuhan Taskaya <batuhan@python.org>
@bedevere-botbedevere-bot removed the needs backport to 3.10 only security fixes label Jun 8, 2021
@bedevere-bot

Copy link
Copy Markdown

GH-26607 is a backport of this pull request to the 3.10 branch.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@isidentical@Fidget-Spinner@bedevere-bot@miss-islington@the-knights-who-say-ni