bpo-43693: Turn localspluskinds into an object

[gvanrossum]

This makes memory management for code objects cleaner (no more special casing for ownership of the kinds array).

https://bugs.python.org/issue43693

[gvanrossum]

I wonder if we're not using up too many magic numbers during pre-alpha...

[ericsnowcurrently]

what are our options?

[gvanrossum]

what are our options?

Going back to the first magic number for 3.11a1. Everything is for the same bpo issue it seems.

[gvanrossum]

@ericsnowcurrently I think I have one failing test left, test_ctypes. In particular:

======================================================================
FAIL: test_frozentable (ctypes.test.test_values.PythonValuesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/guido/cpython/Lib/ctypes/test/test_values.py", line 87, in test_frozentable
self.assertEqual(items, expected, "PyImport_FrozenModules example "
AssertionError: Lists differ: [('__hello__', 133), ('__phello__', -133), ('__phello__.spam', 133)] != [('__hello__', 128), ('__phello__', -128), ('__phello__.spam', 128)]
First differing element 0:
('__hello__', 133)
('__hello__', 128)
- [('__hello__', 133), ('__phello__', -133), ('__phello__.spam', 133)]
? ^^ ^^ ^^
+ [('__hello__', 128), ('__phello__', -128), ('__phello__.spam', 128)]
? ^^ ^^ ^^
: PyImport_FrozenModules example in Doc/library/ctypes.rst may be out of date
----------------------------------------------------------------------

Have you run into this?

[ericsnowcurrently]

Yeah, you have to update the test to match the new memory size

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit b09738c 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[gvanrossum]

I don't see how these changes could make test_ssl fail (only on Mac), and when I run that test myself it passes. I assume it's flakey.

[gvanrossum]

Happy buildbots, except refleaks on AMD64 Windows8.1 is still churning.

[ericsnowcurrently]

mostly LGTM

I've left a few comments for some slight adjustments but otherwise everything looks correct. Thanks for taking the time to do this. FWIW, I now see the value of using PyBytesObject instead of a raw array like I had, so thanks for educating me! 😄

I'm approving the PR now, assuming that you'll make the relevant changes (or ask) before merging.

[ericsnowcurrently]

It seems strange to pass one in and return the other. Perhaps create both in this function?

Suggested change

	staticPyObject*
	compute_localsplus_info(structcompiler*c, PyObject*names)
	staticint
	compute_localsplus_info(structcompiler*c, intnlocalsplus, PyObject**pnames, PyObject**pkinds)

[ericsnowcurrently]

...and something like "init_localsplus_info" is probably a better name with the shift in responsibility for this function.

[gvanrossum]

It seems strange to pass one in and return the other. Perhaps create both in this function?

I find the C idiom for returning multiple values pretty awkward, so I'll just allocate both before passing them into the function.

[gvanrossum]

While all the tests pass here, when I try them at home, I get a crash in test_descr, both on Mac and Windows. I need to understand better what's going on there. Maybe I just need to merge a later version of main. But I'm out of time for now.

[gvanrossum]

Oh wait, I get the same crash on main. Something's wrong!

[gvanrossum]

So, using git bisect I've narrowed it down to GH-26595:

commit e117c0283705943189e6b1aef668a1f68f3f00a4 (HEAD)
Author: Mark Shannon <mark@hotpy.org>
Date: Thu Jun 10 00:46:01 2021
[bpo-44337](https://bugs.python.org/issue44337): Port LOAD_ATTR to PEP 659 adaptive interpreter (GH-26595)
* Specialize LOAD_ATTR with LOAD_ATTR_SLOT and LOAD_ATTR_SPLIT_KEYS
* Move dict-common.h to internal/pycore_dict.h
* Add LOAD_ATTR_WITH_HINT specialized opcode.
* Quicken in function if loopy
* Specialize LOAD_ATTR for module attributes.
* Add specialization stats

I can repro this on my Mac using the following sequence:

$ ./python.exe -m test test_unittest test_doctest test_descr
0:00:00 load avg: 2.28 Run tests sequentially
0:00:00 load avg: 2.28 [1/3] test_unittest
0:00:02 load avg: 2.28 [2/3] test_doctest
0:00:02 load avg: 2.17 [3/3] test_descr
Fatal Python error: Segmentation fault
Current thread 0x0000000111041e00 (most recent call first):
File "<frozen importlib._bootstrap>", line 306 in _module_repr
File "/Users/guido/cpython/Lib/test/test_descr.py", line 3696 in test_uninitialized_modules
File "/Users/guido/cpython/Lib/unittest/case.py", line 549 in _callTestMethod
File "/Users/guido/cpython/Lib/unittest/case.py", line 592 in run
File "/Users/guido/cpython/Lib/unittest/case.py", line 652 in __call__
File "/Users/guido/cpython/Lib/unittest/suite.py", line 122 in run
File "/Users/guido/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/Users/guido/cpython/Lib/unittest/suite.py", line 122 in run
File "/Users/guido/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/Users/guido/cpython/Lib/test/support/testresult.py", line 169 in run
File "/Users/guido/cpython/Lib/test/support/__init__.py", line 960 in _run_suite
File "/Users/guido/cpython/Lib/test/support/__init__.py", line 1083 in run_unittest
File "/Users/guido/cpython/Lib/test/test_descr.py", line 5734 in test_main
File "/Users/guido/cpython/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
File "/Users/guido/cpython/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
File "/Users/guido/cpython/Lib/test/libregrtest/runtest.py", line 154 in _runtest
File "/Users/guido/cpython/Lib/test/libregrtest/runtest.py", line 194 in runtest
File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 423 in run_tests_sequential
File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 521 in run_tests
File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 694 in _main
File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 641 in main
File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 719 in main
File "/Users/guido/cpython/Lib/test/__main__.py", line 2 in <module>
File "/Users/guido/cpython/Lib/runpy.py", line 86 in _run_code
File "/Users/guido/cpython/Lib/runpy.py", line 196 in _run_module_as_main
Extension modules: _testcapi, xxsubtype (total: 2)
Segmentation fault: 11

The same three tests also cause a crash on Windows, with roughly the same traceback.

@markshannon Can you repro this on your Linux machine?

[markshannon]

Repro what?
Your PR is green. The buildbots seem OK.

[gvanrossum]

@markshannon

Repro what?
Your PR is green. The buildbots seem OK.

I know, but when I try this at home, on both Windows and Mac, I get a crash (details in my previous comment):

$ ./python.exe -m test test_unittest test_doctest test_descr

I've painstakingly bisected this behavior down to your PR GH-26595. If I back up in git to the PR before that, that command passes. Any commit on master after that PR, it also fails (including HEAD).

And this is both on Mac and on Windows, so I can't blame it on environmental issues -- the two environments are just too different.

So I beg you to take this serious. I suspect something odd happens in the LOAD_ATTR cache.

Could it be that the CI tests set PYTHONHASHSEED to a fixed value that happens to avoid the crash?

[gvanrossum]

@markshannon
To help pin this down further, I bisected at the test level. Here is the minimal file to pass as --matchfile FILE flag to the test runner:

unittest.test.test_discovery.TestDiscovery.test_discovery_failed_discovery
unittest.test.testmock.testpatch.PatchTest.test_patch_wont_create_by_default
test.test_doctest.TestDocTestFinder.test_empty_namespace_package
test.test_doctest.TestDocTestFinder.test_issue35753
test.test_descr.ClassPropertiesAndMethods.test_uninitialized_modules

[gvanrossum]

There LOAD_ATTR issue is unrelated (discussed offline, we understand it and Mark will fix), so assuming the tests pass after the merge with origin/main I'll land this.