gh-105481: Generate the opcode lists in dis from data extracted from bytecodes.c by iritkatriel · Pull Request #106758 · python/cpython

iritkatriel · 2023-07-14T21:05:19Z

The generated lists are more correct than the (old) hard-coded ones, which were not maintained properly.

For example:

Old:

>>> [opcode.opname[op] for op in opcode.haslocal]
['LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'LOAD_FAST_CHECK', 'LOAD_FAST_AND_CLEAR', 'STORE_FAST_MAYBE_NULL', 'LOAD_CLOSURE']

New:

>>> [opcode.opname[op] for op in opcode.haslocal]
['LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'LOAD_FAST_CHECK', 'MAKE_CELL', 'LOAD_DEREF', 'STORE_DEREF', 'DELETE_DEREF', 'LOAD_FAST_AND_CLEAR', 'LOAD_FAST_LOAD_FAST', 'STORE_FAST_LOAD_FAST', 'STORE_FAST_STORE_FAST', 'LOAD_FROM_DICT_OR_DEREF', 'STORE_FAST_MAYBE_NULL', 'LOAD_CLOSURE']

Issue: Generate opcode metadata from bytecodes.c instead of opcode.py #105481

… opcode.py with the functions in _opcode.py

iritkatriel · 2023-07-14T21:11:38Z

The generated lists are more correct than the (old) hard-coded ones, which were not maintained properly.

For example:

Old:

>>> [opcode.opname[op] for op in opcode.haslocal]
['LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'LOAD_FAST_CHECK', 'LOAD_FAST_AND_CLEAR', 'STORE_FAST_MAYBE_NULL', 'LOAD_CLOSURE']

New:

>>> [opcode.opname[op] for op in opcode.haslocal]
['LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'LOAD_FAST_CHECK', 'MAKE_CELL', 'LOAD_DEREF', 'STORE_DEREF', 'DELETE_DEREF', 'LOAD_FAST_AND_CLEAR', 'LOAD_FAST_LOAD_FAST', 'STORE_FAST_LOAD_FAST', 'STORE_FAST_STORE_FAST', 'LOAD_FROM_DICT_OR_DEREF', 'STORE_FAST_MAYBE_NULL', 'LOAD_CLOSURE']

Maybe in this case actually the new list is incorrect - all the opcodes with DEREF seem to belong to hasfree rather than haslocal:

[opcode.opname[op] for op in opcode.hasfree]
['MAKE_CELL', 'LOAD_DEREF', 'STORE_DEREF', 'DELETE_DEREF', '<148>', 'LOAD_FROM_DICT_OR_DEREF']

gvanrossum · 2023-07-14T21:14:10Z

 for op, i in opmap.items():
 opname[i] = op

+# _opcode may not be ready during early stages of the build


Is that separate from Larry's limerick at the top?

It's the same thing. I'm planning to try and get rid of this issue - figure out what is needed in the build and separate it out somehow.

Bot we've gotta keep a limerick in. :-)

Is it in the stable ABI?

gvanrossum · 2023-07-14T21:22:53Z

The generated lists are more correct than the (old) hard-coded ones, which were not maintained properly.
For example:
Old:

>>> [opcode.opname[op] for op in opcode.haslocal]
['LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'LOAD_FAST_CHECK', 'LOAD_FAST_AND_CLEAR', 'STORE_FAST_MAYBE_NULL', 'LOAD_CLOSURE']

New:

>>> [opcode.opname[op] for op in opcode.haslocal]
['LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'LOAD_FAST_CHECK', 'MAKE_CELL', 'LOAD_DEREF', 'STORE_DEREF', 'DELETE_DEREF', 'LOAD_FAST_AND_CLEAR', 'LOAD_FAST_LOAD_FAST', 'STORE_FAST_LOAD_FAST', 'STORE_FAST_STORE_FAST', 'LOAD_FROM_DICT_OR_DEREF', 'STORE_FAST_MAYBE_NULL', 'LOAD_CLOSURE']

Maybe in this case actually the new list is incorrect - all the opcodes with DEREF seem to belong to hasfree rather than haslocal:

[opcode.opname[op] for op in opcode.hasfree]
['MAKE_CELL', 'LOAD_DEREF', 'STORE_DEREF', 'DELETE_DEREF', '<148>', 'LOAD_FROM_DICT_OR_DEREF']

Ugh. Worse, now LOAD_CLOSURE is gone, it’s ambiguous. Who uses this? How? (Only dis.py?)

iritkatriel · 2023-07-14T21:33:25Z

Ugh. Worse, now LOAD_CLOSURE is gone, it’s ambiguous. Who uses this? How? (Only dis.py?)

LOAD_CLOSURE is just an alias (pseudo op) for LOAD_FAST. It's in the haslocal list in the old and new version, no change there.

iritkatriel · 2023-07-14T22:14:20Z

Now it's like this:

Old:

>>> sorted([opcode.opname[op] for op in opcode.hasfree])
['<148>', 'DELETE_DEREF', 'LOAD_DEREF', 'LOAD_FROM_DICT_OR_DEREF', 'MAKE_CELL', 'STORE_DEREF']
>>> sorted([opcode.opname[op] for op in opcode.haslocal])
['DELETE_FAST', 'LOAD_CLOSURE', 'LOAD_FAST', 'LOAD_FAST_AND_CLEAR', 'LOAD_FAST_CHECK', 'STORE_FAST', 'STORE_FAST_MAYBE_NULL']

New:

>>> sorted([opcode.opname[op] for op in opcode.hasfree])
['DELETE_DEREF', 'LOAD_DEREF', 'LOAD_FROM_DICT_OR_DEREF', 'MAKE_CELL', 'STORE_DEREF']
>>> sorted([opcode.opname[op] for op in opcode.haslocal])
['DELETE_FAST', 'LOAD_CLOSURE', 'LOAD_FAST', 'LOAD_FAST_AND_CLEAR', 'LOAD_FAST_CHECK', 'LOAD_FAST_LOAD_FAST', 'STORE_FAST', 'STORE_FAST_LOAD_FAST', 'STORE_FAST_MAYBE_NULL', 'STORE_FAST_STORE_FAST']

The "<148>" is because in opcode.py in main we have hasfree.append(148) even though there is no such opcode.

In the haslocal list, we see that the old list is missing the super instructions.

iritkatriel · 2023-07-14T22:23:17Z

For the other lists:

Old:

>>> sorted([opcode.opname[op] for op in opcode.hasconst])
['KW_NAMES', 'LOAD_CONST', 'RETURN_CONST']

New:

>>> sorted([opcode.opname[op] for op in opcode.hasconst])
['INSTRUMENTED_RETURN_CONST', 'KW_NAMES', 'LOAD_CONST', 'RETURN_CONST']

old is missing the INSTRUMENTED_*.

Old:

>>> sorted([opcode.opname[op] for op in opcode.hasname])
['DELETE_ATTR', 'DELETE_GLOBAL', 'DELETE_NAME', 'IMPORT_FROM', 'IMPORT_NAME', 'LOAD_ATTR', 'LOAD_FROM_DICT_OR_GLOBALS', 'LOAD_GLOBAL', 'LOAD_METHOD', 'LOAD_NAME', 'LOAD_SUPER_ATTR', 'LOAD_SUPER_METHOD', 'LOAD_ZERO_SUPER_ATTR', 'LOAD_ZERO_SUPER_METHOD', 'STORE_ATTR', 'STORE_GLOBAL', 'STORE_NAME']

New:

>>> sorted([opcode.opname[op] for op in opcode.hasname])
['DELETE_ATTR', 'DELETE_GLOBAL', 'DELETE_NAME', 'IMPORT_FROM', 'IMPORT_NAME', 'LOAD_ATTR', 'LOAD_FROM_DICT_OR_GLOBALS', 'LOAD_GLOBAL', 'LOAD_METHOD', 'LOAD_NAME', 'LOAD_SUPER_ATTR', 'LOAD_SUPER_METHOD', 'LOAD_ZERO_SUPER_ATTR', 'LOAD_ZERO_SUPER_METHOD', 'STORE_ATTR', 'STORE_GLOBAL', 'STORE_NAME']

Old is missing LOAD_ZERO_SUPER_ATTR and LOAD_ZERO_SUPER_METHOD.

Old:

>>> sorted([opcode.opname[op] for op in opcode.hasjrel])
['FOR_ITER', 'JUMP', 'JUMP_BACKWARD', 'JUMP_BACKWARD_NO_INTERRUPT', 'JUMP_FORWARD', 'JUMP_NO_INTERRUPT', 'POP_JUMP_IF_FALSE', 'POP_JUMP_IF_NONE', 'POP_JUMP_IF_NOT_NONE', 'POP_JUMP_IF_TRUE', 'SEND']

New:

>>> sorted([opcode.opname[op] for op in opcode.hasjrel])
['ENTER_EXECUTOR', 'FOR_ITER', 'JUMP', 'JUMP_BACKWARD', 'JUMP_BACKWARD_NO_INTERRUPT', 'JUMP_FORWARD', 'JUMP_NO_INTERRUPT', 'POP_JUMP_IF_FALSE', 'POP_JUMP_IF_NONE', 'POP_JUMP_IF_NOT_NONE', 'POP_JUMP_IF_TRUE', 'SEND']

New has ENTER_EXECUTOR as a jump.

For hasarg, the old list contained a few that the new list does not:

{'INSTRUMENTED_CALL_FUNCTION_EX', 'INSTRUMENTED_RETURN_VALUE', 'INSTRUMENTED_LINE', 'INSTRUMENTED_END_SEND', 'INSTRUMENTED_INSTRUCTION', 'INSTRUMENTED_END_FOR'}

iritkatriel · 2023-07-14T22:26:40Z

It mostly looks right, except for INSTRUMENTED_CALL_FUNCTION_EX, which delegates the work to CALL_FUNCTION_EX which does have an arg.

gvanrossum · 2023-07-14T23:21:32Z

Ugh. Worse, now LOAD_CLOSURE is gone, it’s ambiguous. Who uses this? How? (Only dis.py?)
LOAD_CLOSURE is just an alias (pseudo op) for LOAD_FAST. It's in the haslocal list in the old and new version, no change there.

My point is that LOAD_CLOSURE used to be separate because it was in the hasfree list. Now some LOAD_FAST instances refer to a local and others to a free variable. Example:

>>> def foo(a):
... print(a)
... def bar(): return a
... >>> dis.dis(foo)
0 MAKE_CELL 0 (a)
1 2 RESUME 0
2 4 LOAD_GLOBAL 1 (NULL + print)
14 LOAD_DEREF 0 (a)
16 CALL 1
24 POP_TOP
3 26 LOAD_FAST 0 (a)
28 BUILD_TUPLE 1
30 LOAD_CONST 1 (<code object bar at 0x101d18400, file "<stdin>", line 3>)
32 MAKE_FUNCTION
34 SET_FUNCTION_ATTRIBUTE 8 (closure)
36 STORE_FAST 1 (bar)
38 RETURN_CONST 0 (None)
Disassembly of <code object bar at 0x101d18400, file "<stdin>", line 3>:
0 COPY_FREE_VARS 1
3 2 RESUME 0
4 LOAD_DEREF 0 (a)
6 RETURN_VALUE
>>>

The LOAD_FAST (a) at offset 26 used to be a LOAD_CLOSURE in 3.12 and before (and in 3.13 until LOAD_CLOSURE was made a pseudo-op). It's harmless, because we have co_varnames indexed by both locals and cells. So maybe hasfree is no longer useful?

I don't think

gvanrossum · 2023-07-14T23:28:16Z

The "<148>" is because in opcode.py in main we have hasfree.append(148) even though there is no such opcode.

Looks like a relic. In 3.11, 148 was LOAD_CLASSDEREF. That opcode no longer exists in 3.12 (it was removed in gh-103764 for PEP 695). So let's get rid of it.

iritkatriel · 2023-07-15T16:07:02Z

The LOAD_FAST (a) at offset 26 used to be a LOAD_CLOSURE in 3.12 and before (and in 3.13 until LOAD_CLOSURE was made a pseudo-op). It's harmless, because we have co_varnames indexed by both locals and cells. So maybe hasfree is no longer useful?

So we merge hasfree into haslocal and remove hasfree? Or leave it empty and soft deprecate it?

gvanrossum · 2023-07-15T18:35:29Z

So we merge hasfree into haslocal and remove hasfree? Or leave it empty and soft deprecate it?

Hm, I was just writing up a rationale for this when I realized that we should just keep hasfree and populate it correctly, and explain the special situation where LOAD_FAST can be used to load a cell (previously LOAD_CLOSURE) separately.

iritkatriel · 2023-07-15T20:13:04Z

Ok, then this PR is probably fine w.r.t. hasfree and haslocal.

The only list in oplists now is hascompare, which contains only COMPARE_OP:

>>> [opcode.opname[op] for op in opcode.hascompare]
['COMPARE_OP']

It is used by dis to identify that its op (>=, ==, etc) needs to be displayed.

gvanrossum · 2023-07-15T20:41:18Z

It is used by dis to identify that its op (>=, ==, etc) needs to be displayed.

Since there's only one opcode like this, maybe dis can just check whether it's that opcode? Or do you anticipate that in the future there will be multiple ones? (Were there ever?)

iritkatriel · 2023-07-15T20:46:36Z

It is used by dis to identify that its op (>=, ==, etc) needs to be displayed.
Since there's only one opcode like this, maybe dis can just check whether it's that opcode? Or do you anticipate that in the future there will be multiple ones? (Were there ever?)

It's just this one all the way back to 3.8.

dis can stop using the list, but it's documented in the dis module so can we remove it?

gvanrossum · 2023-07-15T20:51:32Z

dis can stop using the list, but it's documented in the dis module so can we remove it?

I guess deprecate it like hasjabs/hasjrel.

gvanrossum · 2023-07-17T22:39:21Z

 for op, i in opmap.items():
 opname[i] = op

+# The build uses older versions of Python which do not have _opcode.has_* functions


This is quite mysterious. IIRC Brandt (?) wrote some hack that reads opcode.py and then execs it? But I can't find it.

Yes, in Tools/build/generate_opcode_h.py. But when it execs opcode.py that tries to import _opcode and call its has_arg etc to construct the oplists (which are not used during the build).
Ideally we should just get rid of generate_opcode_h.py if we can.

I think we don't need to import _opcode into opcode at all. We can just import it directly into dis (which is not used by any build scripts). I'll try that.

gvanrossum · 2023-07-17T22:41:03Z

- self.assertIsInstance(func(op), bool)
- self.assertEqual(func(op), expected)
-
+EXPECTED_OPLISTS = {


Does this mean that each time we add or remove an opcode we need to update this test? That feels tedious. (Even though it looks like it was always done like this and you're just refactoring that code a bit.)

In the past the list was built in opcode.py so we would need to make sure we added i there. Now the lists are derived automatically from bytecodes.c via heuristics, and you will need to update the test (which you are less likely to be able to do incorrectly). I'm not sure it's safe to not have these full tests.

Hm. So the point of the heuristics is that we don't have to manually maintain it, but the tests are manually maintained to check that the heuristics are still working. If they are, and if an instruction is added (or deleted) that changes one of these lists, we'll get a bogus test failure, and hopefully the contributor who gets to debug the test failure understands they have to fix the test. I think this warrants a comment right there where we expect the test to fail (I think on line 102, self.assertEqual(res, op in expected)).

The manually maintained lists were not tested and had some errors. We could remove the tests and trust people to examine the diff in the metadata file when they make changes. If we do that we should remove it from the autogenerated list so that it's not collapsed in a PR diff.

…ible

…ode_metadata rather than via opcode

…ad of opcode for _specialized_instructions

iritkatriel · 2023-07-18T16:00:56Z

I think I need to add in whatsnew that the dis module's opcode lists are no longer also defined in the opcode module. The opcode module is not documented, but if you look on GitHub you can see some places where people import it.

iritkatriel · 2023-07-18T16:08:17Z

I think I need to add in whatsnew that the dis module's opcode lists are no longer also defined in the opcode module. The opcode module is not documented, but if you look on GitHub you can see some places where people import it.

Or maybe this is a breaking change that I can't make without deprecation? I'm not sure what the status of the opcode module is.

I can put the lists back in opcode, and instead work to remove the dependency of the build on opcode.

gvanrossum · 2023-07-18T17:01:55Z

I have a feeling that opcode is actually a better place than dis for the official public API. And that's probably why people have been importing it despite its undocumented status.

…ta instead of opcode for _specialized_instructions" This reverts commit 5e1c4a4.

…rom _opcode_metadata rather than via opcode" This reverts commit d525c53.

This reverts commit f530469.

This reverts commit 1a0a2b2.

iritkatriel · 2023-07-18T17:31:46Z

OK, I reverted the last few commits so we leave everything in opcode. We will rid the build of opcode in a future PR once the opcode.h file is generated from bytecodes.c.

pythongh-105481: add haslocal to _opcode.py. Generate most oplists in…
36a7d56
… opcode.py with the functions in _opcode.py

iritkatriel requested a review from markshannon as a code owner July 14, 2023 21:05

bedevere-bot added the awaiting core review label Jul 14, 2023

bedevere-bot mentioned this pull request Jul 14, 2023
Generate opcode metadata from bytecodes.c instead of opcode.py #105481
Closed

iritkatriel requested a review from gvanrossum July 14, 2023 21:05

iritkatriel commented Jul 14, 2023
View reviewed changes

Comment threadLib/opcode.py

gvanrossum reviewed Jul 14, 2023
View reviewed changes

iritkatriel marked this pull request as draft July 14, 2023 21:34

bedevere-bot removed the awaiting core review label Jul 14, 2023

added HAS_FREE flag. Removed HAS_FREE from the HAS_LOCAL list
75019ca

iritkatriel changed the title ~~gh-105481: add haslocal to _opcode.py. Generate most oplists in opcode.py with the functions in _opcode.py~~gh-105481: add haslocal/hasfree to _opcode.py. Generate most oplists in opcode.py with the functions in _opcode.pyJul 14, 2023

iritkatriel added 2 commits July 15, 2023 17:14

add hasjump, soft deprecate hasjrel and hasjabs. Fix build bug
494680a

generate hasexc from the C macros instead of defining it again
24786fb

iritkatriel changed the title ~~gh-105481: add haslocal/hasfree to _opcode.py. Generate most oplists in opcode.py with the functions in _opcode.py~~gh-105481: add has_local/has_free/etc to _opcode.py. Generate oplists in opcode.py with the functions in _opcode.pyJul 15, 2023

gvanrossum reviewed Jul 15, 2023
View reviewed changes

Comment threadModules/_opcode.c Outdated

typo
563b431

iritkatriel changed the title ~~gh-105481: add has_local/has_free/etc to _opcode.py. Generate oplists in opcode.py with the functions in _opcode.py~~gh-105481: Generate the opcode lists in dis from data extracted from bytecodes.cJul 17, 2023

iritkatriel commented Jul 17, 2023
View reviewed changes

Comment threadLib/opcode.py Outdated

iritkatrieland others added 3 commits July 17, 2023 22:08

remove unnecessary comment
ce76a60

make the tests more precise
4194a12

sort the lists
6b8c46b

gvanrossum reviewed Jul 17, 2023
View reviewed changes

iritkatrieland others added 8 commits July 18, 2023 10:56

address some of the code review comments
e15f7d5

remove hard-coded lists from the tests
21fe0a4

remove opcode metadata file from generated files list so diff are vis…
75795db
…ible

import _opcode directly into dis, rather than via opcode
1a0a2b2

add stack effect to dis.__all__
f530469

Merge branch 'main' into opcode_py
71ffe34

import _specializations, _specialized_instructions directly from _opc…
d525c53
…ode_metadata rather than via opcode

update Tools/scripts/summarize_stats.py to use _opcode_metadata inste…
5e1c4a4
…ad of opcode for _specialized_instructions

gvanrossum approved these changes Jul 18, 2023
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jul 18, 2023

iritkatriel added 4 commits July 18, 2023 18:28

Revert "update Tools/scripts/summarize_stats.py to use _opcode_metada…
6077263
…ta instead of opcode for _specialized_instructions" This reverts commit 5e1c4a4.

Revert "import _specializations, _specialized_instructions directly f…
98ba11f
…rom _opcode_metadata rather than via opcode" This reverts commit d525c53.

Revert "add stack effect to dis.__all__"
cc8e5e1
This reverts commit f530469.

Revert "import _opcode directly into dis, rather than via opcode"
31dbfec
This reverts commit 1a0a2b2.

Merge branch 'main' into opcode_py
1abf6ca

iritkatriel merged commit 40f3f11 into python:mainJul 18, 2023

bedevere-bot removed the awaiting merge label Jul 18, 2023

iritkatriel deleted the opcode_py branch July 25, 2023 18:02

hugovk mentioned this pull request Sep 25, 2023
Improve import time of various stdlib modules #109653
Closed

Uh oh!

Uh oh!

Conversation

iritkatriel commented Jul 14, 2023• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

iritkatriel commented Jul 14, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gvanrossumJul 14, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gvanrossum commented Jul 14, 2023

Uh oh!

iritkatriel commented Jul 14, 2023

Uh oh!

iritkatriel commented Jul 14, 2023

Uh oh!

iritkatriel commented Jul 14, 2023

Uh oh!

iritkatriel commented Jul 14, 2023

Uh oh!

gvanrossum commented Jul 14, 2023

Uh oh!

gvanrossum commented Jul 14, 2023

Uh oh!

iritkatriel commented Jul 15, 2023

Uh oh!

gvanrossum commented Jul 15, 2023

Uh oh!

iritkatriel commented Jul 15, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

gvanrossum commented Jul 15, 2023

Uh oh!

iritkatriel commented Jul 15, 2023

Uh oh!

gvanrossum commented Jul 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iritkatrielJul 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iritkatriel commented Jul 18, 2023

Uh oh!

iritkatriel commented Jul 18, 2023

Uh oh!

gvanrossum commented Jul 18, 2023

Uh oh!

iritkatriel commented Jul 18, 2023

Uh oh!

Reviewers

Assignees

iritkatriel commented Jul 14, 2023•
edited by bedevere-bot
Loading

gvanrossumJul 14, 2023•
edited
Loading

iritkatriel commented Jul 15, 2023•
edited
Loading

iritkatrielJul 18, 2023•
edited
Loading