distutils is not reproducible

[vstinner]

BPO	34033
Nosy	@vstinner, @benjaminp, @encukou, @methane, @yan12125, @bmwiedemann, @jefferyto
PRs	bpo-34033: distutils: byte_compile() sort files #8057 gh-78214: marshal: Stabilize FLAG_REF usage #8226

Dependencies:

• remove *_INTERNED opcodes from marshal #75558
• Reproducible pyc: FLAG_REF is not stable. #78274
• Non-deterministic bytecode generation for frozenset #78903
• Reproducible pyc: frozenset is not serialized in a deterministic order #81777

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-07-03.15:46:25.794>
labels = ['3.8', 'library']
title = 'distutils is not reproducible'
updated_at = <Date 2020-04-10.13:23:09.703>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-04-10.13:23:09.703>
actor = 'yan12125'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-07-03.15:46:25.794>
creator = 'vstinner'
dependencies = ['31377', '34093']
files = []
hgrepos = []
issue_num = 34033
keywords = ['patch']
message_count = 9.0
messages = ['320988', '320990', '320991', '321383', '321408', '321432', '321434', '337975', '359595']
nosy_count = 9.0
nosy_names = ['vstinner', 'benjamin.peterson', 'petr.viktorin', 'sascha_silbe', 'zbysz', 'methane', 'yan12125', 'bmwiedemann', 'jefferyto']
pr_nums = ['8057', '8226']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue34033'
versions = ['Python 3.8']

[vstinner]

Follow up of bpo-29708: OpenSUSE uses a downstream patch for distutils to fix https://bugzilla.opensuse.org/show_bug.cgi?id=1049186: distutils-reproducible-compile.patch. I converted the patch as a PR: PR 8057.

Naoki INADA wrote:
"""
Currently, marshal uses refcnt to determine using w_ref or not. Some immutable objects (especially, long and str) can be cached and reused. It may affects refcnt when byte compiling.

I think we should use more deterministic way instead of refcnt. Maybe, count all constants in the module before marshal, like we did in compiling function for co_consts and co_names.
As a bonus, it may reduce resource usage too by merging constants over functions.
(e.g. ('self',) co_varnames and (None,) co_consts)
"""
#8057 (comment)

Serhiy Storchaka added:
"""
I think we need to understand the issue better before committing changes. When found the source of unstability of file names, we can find other similar sources and make them stable too. For example if the source is listdir() or glob(), we can consider sorting results of all listdir() or glob() in distutils and related methods.

On other side, if the problem is with reference counters in marshal, we can change the marshal module instead.
"""
#8057 (comment)

[vstinner]

Copy of https://bugzilla.opensuse.org/show_bug.cgi?id=1049186 first message:
"""
e.g. python-simplejson has one-bit diffs in .pyc files
See
http://rb.zq1.de/compare.factory-20170713/python-simplejson-compare.out

in python3-simplejson.rpm we get
-00004e50 68 6f 72 5f 5f da 07 64 65 63 69 6d 61 6c 72 0c |hor__..decimalr.|
+00004e50 68 6f 72 5f 5f 5a 07 64 65 63 69 6d 61 6c 72 0c |hor__Z.decimalr.|

in python3-simplejson-test.rpm we get the opposite change
-00000580 72 13 00 00 00 5a 07 64 65 63 69 6d 61 6c 72 03 |r....Z.decimalr.|
+00000580 72 13 00 00 00 da 07 64 65 63 69 6d 61 6c 72 03 |r......decimalr.|

and it seems to be related to filesystem ordering, since it built reproducibly
when using a filesystem with sorted readdir
using disorderfs via reproducible-faketools-filesys from
https://build.opensuse.org/package/show/home:bmwiedemann:reproducible/reproducible-faketools
"""
https://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c0

[benjaminp]

I agree that we should fix the underlying issue (marshal) rather than papering over it by sorting. In fact, we should have a test that compiles a bunch of pycs in a random orders and sees if they're the same or not.

[methane]

Is this issue for only known marshal issue?
Or is this issue for all issues in distutils including unknowns?

[benjaminp]

We should probably discuss the marshal issue in the preëxisting bpo-31377.

I'm not sure if "distutils is not reproducible" is a larger issue than "pyc compilation is not reproducible". This issue could be a meta issue for either.

[vstinner]

Is this issue for only known marshal issue?

IMHO the order in which .pyc files are created on disk also matters. It changes the result of "os.listdir()": some application can rely on unsorted os.listdir(). sorted() seems simple and hardless compared to the benefit.

[methane]

OK, I created sub issue for pyc.

[bmwiedemann]

unreproducible .pyc files are still one of the major headaches for my work on openSUSE reproducible builds.

There is also one aspect where i586 builds end up with different .pyc files than x86_64 builds. And then we randomly chose one of them for our "noarch" python module packages and hope they work everywhere (including on arm and s390 architectures).

So is someone working towards a concept that makes it is possible to create the same .pyc files anywhere?
Can I help something there?
Is there an ETA?

[encukou]

There is also one aspect where i586 builds end up with different .pyc files than x86_64 builds. And then we randomly chose one of them for our "noarch" python module packages and hope they work everywhere (including on arm and s390 architectures).

They are functionally identical, despite not being bit-by-bit identical.
If they do not work everywhere, it's a very serious bug.

So is someone working towards a concept that makes it is possible to create the same .pyc files anywhere?

No, it's a known issue no one is working on.

Can I help something there?

Maybe?
The two main culprits are in the marshal serialization algorithm: https://github.com/python/cpython/blob/master/Python/marshal.c
Specifically:

• a heuristic depends on refcount (i.e. state of objects in the entire interpreter, rather than just relationships between serialized objects):

  cpython/Python/marshal.c

  Line 304 in 33b671e

  /* if it has only one reference, it definitely isn't shared */

• (frozen)sets are serialized in iteration order, which is unpredictable (and determinig a predictable order is not trivial):

  cpython/Python/marshal.c

  Line 498 in 33b671e

  else if (PyAnySet_CheckExact(v)) {

A solution will probably come with an unacceptable performance hit -- it's good to keep generating the .pyc files fast. Two options to overcome that come to mind:

• make reproducibility optional (which would make the testing more cumbersome)
• make an add-on tool to re-serialize an existing .pyc.