bpo-45950: Introduce Bootstrap Python again

[tiran]

The build system now uses a :program:_bootstrap_python interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:_bootstrap_python and :program:_freeze_module are no longer
build with LTO.

Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.

https://bugs.python.org/issue45950

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit b757d5bd526cfffeb134eb6744ac86c561e61e9d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit fa4cd7fc9f3763ff1309d8bfbc97853223f17239 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[gvanrossum]

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

[vstinner]

it's just an extra link step

When LTO is used, a link step can be quite slow. I noticed that when there was a second _bootstrap_python binary. I didn't check if _testembed has a similar issue or not. I'm mostly developing Python without --enable-shared: Python is linked "statically" in all built executables.

[vstinner]

Maybe LTO flags can be skipped on these less important binaries.

[vstinner]

FYI the revert #29717 fixed an issue with out of tree build. Please try to make sure that this bug is not reintroduced: https://bugs.python.org/issue45866#msg406764

[tiran]

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

We need some extra logic in configure.ac and Makefile anyway. The _bootstrap_python approach is incompatible with cross compiling. When you cross compile for a different architecture or platform, then any compiled code is not executable on the build host.

The approach also speeds up local development as long as you have a recent Python installation on your computer. With _bootstrap_python almost every code file gets rebuild whenever a core file changes. Scratch that, the Program/_freeze_module dependency triggers a partial rebuild.

I'll come up with a better way to test _buildbot_python.

[gvanrossum]

I am in favor of this. Thanks for doing this!

[hroncok]

I'll try to build this as RPM in Fedora to see if there are some problems with our bootstrapped OOT build.

[hroncok]

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

...
gcc -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -fno-lto -o Programs/_freeze_module Programs/_freeze_module.o Modules/getbuildinfo.o Parser/token.o Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o Python/pydtrace.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o Modules/faulthandler.o Modules/posixmodule.o Modules/signalmodule.o Modules/_tracemalloc.o Modules/_codecsmodule.o Modules/_collectionsmodule.o Modules/errnomodule.o Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o Modules/itertoolsmodule.o Modules/_sre.o Modules/_threadmodule.o Modules/timemodule.o Modules/_weakref.o Modules/_abc.o Modules/_functoolsmodule.o Modules/_localemodule.o Modules/_operator.o Modules/_stat.o Modules/symtablemodule.o Modules/pwdmodule.o Modules/xxsubtype.o -ldl -lm /usr/bin/ld: Programs/_freeze_module.o: in function `runtime_init':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:44: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o: in function `_Py_DECREF':
/builddir/build/BUILD/cpython-bootstrap_again/Include/object.h:493: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:(.data.rel.ro+0x0): undefined reference to `__gcov_indirect_call'
/usr/bin/ld: Programs/_freeze_module.o: in function `main':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:191: undefined reference to `__gcov_indirect_call'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_indirect_call_profiler_v4'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:211: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: more undefined references to `__gcov_time_profiler_counter' follow
...

[tiran]

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

[tiran]

OOT build with optimizations is passing for me on F35:

../../configure -C --enable-optimizations``
make

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit 02da340e62db33a12aad1b03adf83a488e0255fb 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[hroncok]

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

That helped.

[tiran]

Rebased on top of #29902

[bedevere-bot]

@tiran: Please replace # with GH- in the commit message next time. Thanks!