Add a new Include/cpython/ subdirectory for the "CPython API" with implementation details

[vstinner]

BPO	35134
Nosy	@brettcannon, @ncoghlan, @vstinner, @jkloth, @ericsnowcurrently, @serhiy-storchaka, @shihai1991, @erlend-aasland, @nw0
PRs	[WIP] bpo-35134: Create Include/pycapi/ subdirectory #10285 bpo-35134: Create Include/cpython/ subdirectory #10624 bpo-35134: Create Include/cpython/object.h #10679 bpo-35134: Create Include/cpython/unicodeobject.h #10680 bpo-35134: Add Include/cpython/pyerrors.h #10727 bpo-35134: Create Include/cpython/abstract.h #10728 bpo-35134: Create Include/cpython/pylifecycle.h #10731 bpo-35134: Create Include/cpython/dictobject.h #10732 bpo-35134: Create Include/cpython/pystate.h #10733 bpo-35134: Update "make tags": add Include/cpython/ #10739 bpo-35134: Don't define types twice in header files #10754 bpo-35134: Create Include/cpython/tupleobject.h #10764 bpo-35134: Add Include/cpython/pymem.h #12840 bpo-35134: Add cpython/pymem.h to build system #12842 bpo-35134: Split traceback.h header #13430 bpo-35134: Register new traceback.h header files #13431 bpo-35134: Add Include/cpython/import.h header file #14213 bpo-35134: Migrate frameobject.h contents to cpython/frameobject.h #18052 bpo-35134: Create Include/cpython/listobject.h #18395 [WIP] bpo-35134: Move header files to Include/cpython/ #18490 bpo-35134: Add Include/cpython/fileutils.h header file #18493 bpo-35134: Add Include/cpython/bytesobject.h file #18494 bpo-40421: Add Include/cpython/code.h header file #19756 bpo-35134: Add Include/cpython/pythonrun.h file #23701 bpo-35134: Add Include/cpython/pytime.h file #23988 bpo-35134: Move Include/{pyarena.h,pyctype.h} to Include/cpython #24550 bpo-35134: Move non-limited C API files to Include/cpython/ #24561 bpo-35134: move asdl.h, pystrhex.h, symtable.h, token.h, tracemalloc.h into Include/cpython #24770 bpo-35134: Move non-limited C API of include/compile.h into include/cpython. #24922 bpo-35134: Add Include/cpython/floatobject.h #28957 bpo-35134: Move Include/funcobject.h to Include/cpython/ #28958 bpo-35134: Move Include/cellobject.h to Include/cpython/ #28964 bpo-35134: Move classobject.h to Include/cpython/ #28968 bpo-35134: Split warnings.h and weakrefobject.h #29042 bpo-35134: Add Include/cpython/longobject.h #29044 bpo-35134: Add Include/cpython/descrobject.h #30923 bpo-35134: Add Include/cpython/complexobject.h header #32383 bpo-35134: Add Include/cpython/setobject.h header #32384 bpo-35134: Remove the Include/code.h header file #32385

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-30.23:33:02.262>
created_at = <Date 2018-11-01.12:46:49.299>
labels = ['interpreter-core', '3.8']
title = 'Add a new Include/cpython/ subdirectory for the "CPython API" with implementation details'
updated_at = <Date 2022-04-07.00:29:55.966>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-04-07.00:29:55.966>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-30.23:33:02.262>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-11-01.12:46:49.299>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35134
keywords = ['patch']
message_count = 61.0
messages = ['329060', '329087', '329123', '329124', '329126', '329290', '329292', '330161', '330162', '330165', '330246', '330261', '330276', '330282', '330283', '330285', '330296', '330324', '330331', '330336', '330337', '330446', '330447', '330463', '330467', '330477', '330478', '330480', '330512', '330556', '330557', '330594', '340283', '340285', '342876', '342881', '346014', '360338', '361542', '361922', '361923', '367536', '382771', '383970', '387105', '387179', '387327', '389285', '394491', '403952', '403955', '403957', '403964', '403967', '403983', '404248', '404252', '411772', '416906', '416910', '416912']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'jkloth', 'eric.snow', 'serhiy.storchaka', 'shihai1991', 'erlendaasland', 'nw0']
pr_nums = ['10285', '10624', '10679', '10680', '10727', '10728', '10731', '10732', '10733', '10739', '10754', '10764', '12840', '12842', '13430', '13431', '14213', '18052', '18395', '18490', '18493', '18494', '19756', '23701', '23988', '24550', '24561', '24770', '24922', '28957', '28958', '28964', '28968', '29042', '29044', '30923', '32383', '32384', '32385']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35134'
versions = ['Python 3.8']

Linked PRs

• gh-79315: Add Include/cpython/memoryobject.h header #99723
• gh-79315: Add Include/cpython/marshal.h header #139725
• gh-79315: Add Include/cpython/sliceobject.h header #139729
• gh-79315: Add Include/cpython/structseq.h header #139730
• gh-79315: Remove Include/pylock.h and Include/monitoring.h #139731

[vstinner]

The PEP-384 "Defining a Stable ABI" introduced Py_LIMITED_API define to exclude functions from the Python C API. The problem is when a new API is introduced, it has to explicitly be excluded using "#ifndef Py_LIMITED_API". If the author forgets it, the function is added to be stable API by mistake.

I propose to move the API which should be excluded from the stable ABI to a new subdirectory: Include/pycapi/.

To not break the backward compatibility, I propose to automatically include new header files from existing header files. For example, Include/pycapi/pyapi_objimpl.h would be automatically included by Include/pycapi/pycapi_objimpl.h.

New header files would have a "pycapi_" prefix to avoid conflict Include/ header files, if Include/pycapi/ directory is in the header search paths.

This change is a follow-up of bpo-35081 which moved Py_BUILD_CORE code to Include/internal/.

It is also part of a larger project to cleanup the C API, see:

• https://pythoncapi.readthedocs.io/split_include.html
• https://pythoncapi.readthedocs.io/

The change is backward compatible: #include <Python.h> will still provide exactly the same API.

[serhiy-storchaka]

There are not just two sides. It is common to wrap new stable C API with something like:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000

What will you do with this?

[vstinner]

There are not just two sides. It is common to wrap new stable C API with something like:
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000
What will you do with this?

objimpl.h always includes pycapi/pycapi_objimpl.h, so I don't think that we need a strong rules. I propose to always add move code using "#if ... Py_LIMITED_API" to the pycapi/ subdirectory, even if it uses "#if !defined(Py_LIMITED_API)".

[serhiy-storchaka]

Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.

[vstinner]

Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.

The raw definition could be that Include/.h is part of the stable ABI, and Include/pycapi/.h are the definitions using Py_LIMITED_API and so can be stable or not stable depending on Py_LIMITED_API value :-)

To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.

I understand that you would prefer to leave PyObject_Calloc() in Include/objimpl.h. Honestly, I have no strong opinion on that. We can leave it there if you prefer.

--

Maybe the rule "move everything using Py_LIMITED_API to pycapi" is misleading.

My intent is that API in Include/*.h should not leak implementation details. It should be the starting point to design a new C API which does not leak any implementation detail:
http://pythoncapi.readthedocs.io/

It's easier with an example:

#define _PyObject_GC_TRACK(o) do { \
 PyGC_Head *g = _Py_AS_GC(o); \
 if (g->_gc_next != 0) { \
 Py_FatalError("GC object already tracked"); \
 } \
 assert((g->_gc_prev & _PyGC_PREV_MASK_COLLECTING) == 0); \
 ...

This macro is private: it starts with "_Py", so it doesn't belong to Include/*.h. Moreover, it access private fields like PyGC_Head._gc_prev.

From my point of view, the ideal API would not access *any* structure field and PyGC_Header structure must not be used nor part of the C API.

--

After saying that, I looked again at my PR, and I still see private functions in objimpl.h. Example:

PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *);
PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t);

#define PyObject_New(type, typeobj) \
 ( (type *) _PyObject_New(typeobj) )
#define PyObject_NewVar(type, typeobj, n) \
 ( (type *) _PyObject_NewVar((typeobj), (n)) )

These functions are not excluded from Py_LIMITED_API. Since they are private, we are free to remove them whenever we want, so IMHO it's fine to exclude from Py_LIMITED_API right now if we want.

Another example:

static inline PyObject*
PyObject_INIT(PyObject *op, PyTypeObject *typeobj)
{
 assert(op != NULL);
 Py_TYPE(op) = typeobj;
 _Py_NewReference(op);
 return op;
}

It's a public function but it calls the private function _Py_NewReference(). So _Py_NewReference() must be part of Py_LIMITED_API somehow...

In release mode (if Py_TRACE_REFS is not defined), _Py_NewReference() is defined like that:

/* Without Py_TRACE_REFS, there's little enough to do that we expand code
 inline. */
static inline void _Py_NewReference(PyObject *op)
{
 if (_Py_tracemalloc_config.tracing) {
 _PyTraceMalloc_NewReference(op);
 }
 _Py_INC_TPALLOCS(op);
 _Py_INC_REFTOTAL;
 Py_REFCNT(op) = 1;
}

It does access to the private _Py_tracemalloc_config variable and private macros/functions _Py_INC_TPALLOCS(op) and _Py_INC_REFTOTAL.

We *can* always define _Py_NewReference() as a function call if Py_LIMITED_API is defined, but it would have an impact on performance.

Right now, I don't want to risk to introduce a performance slowdown.

I have a "Proof-of-concept" implementation of my proposed "new C API":
https://github.com/pythoncapi/cpython/

My implementation currently uses 3 defines:

• Py_NEWCAPI_NO_MACRO: replace macros with function calls PyTuple_GET_SIZE() becomes PyTuple_Size()
• Py_NEWCAPI_NO_STRUCT: must not use PyObject.ob_refcnt or any other field of Python object structures; structures should hide their fields: compilation error.
• Py_NEWCAPI: new C API without borrowed references, without macro, without struct

But this project is highly experimental and I don't want to make it upstream before we measured properly the impact on the performance, the API has been properly reviewed and discussed, and the overall project has been approved by core developers. For example, by writing a PEP :-)

--

In short, I'm not sure of what can or should be done right now for Include/pycapi/ :-)

I wrote the PR to open the discussion :-)

[ncoghlan]

To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.

It's described here: https://docs.python.org/3/c-api/stable.html

If a stable ABI consumer just declares "#define PY_LIMITED_API 1", then they'll get the original stable ABI as defined in Python 3.2.

If they don't care about versions prior to 3.6, they can instead declare "#define PY_LIMITED_API 0x03060000", and get access to the functions added to the stable ABI in 3.3, 3.4, 3.5, and 3.6.

For this PR though, I think it's OK to ignore that detail, as once all the internal APIs are in "Include/internal", and all the APIs that don't offer ABI stability guarantees are in "Include/TBD" (see note below), then the general rule to follow is that everything added to the headers directly in "Include/" needs a Py_LIMITED_API guard that matches the upcoming release.

Note: I wrote "TBD" rather than "pycapi" above, as "pycapi" sounds like the name of a preferred public API to me, rather than "code compiled against this API is not portable to later versions, and may not be portable to other implementations". Given the name of the macro, "Include/unlimited/.h" may make sense, especially if those header files are all written to error out at compile time if PY_LIMITED_API is defined. "Include/unstable_abi/.h" would be another self-describing name.

[ncoghlan]

On actually looking at the initial changes in the PR:

• declarations that aren't part of the stable ABI in any version (i.e. "#ifndef PY_LIMITED_API", "#if !defined(PY_LIMITED_API)") should move to the new directory

• declarations that are part of the stable ABI in *some* version should remain where they are (i.e. in "Include/*.h")

In your initial PR, the only API that subtle distinction affects is PyObject_Calloc (since that's a new addition to the stable ABI in 3.5+), and moving that back to the public header means you can add the desired "Py_LIMITED_API is not defined" check to the header in the new directory.

[vstinner]

I created a new PR 10624:

• move "#ifndef Py_LIMITED_API" code to a new unstable/objimpl.h header file
• Include/unstable/ files (Include/unstable/objimpl.h) are no longer prefixed with "unstable_".

Include/unstable/ directory must not be added to the search paths for headers (gcc -I Include/unstable/). unstable/objimpl.h must not be included directly: it fails with a compiler error.

[vstinner]

I propose the following organization:

• Include/*.h should be the "stable API"
• Include/unstable/.h is the "unstable API" (if Py_LIMITED_API is *not defined at all)
• Include/internal/pycore_*.h is the "internal" API

It should become easier to see what is exposed or not to the stable ABI just by looking at Include.*.h.

It should also become easier to spot in a review when a pull request something to the stable ABI, whereas it should be added to the unstable or internal API.

[vstinner]

Just to avoid the risk of name conflict, would it make sense to rename "unstable" to "pyunstable" or something else with "py" inside? I'm not sure if #include "unstable/objimpl.h" first looks the same directory than the header file that does the include?

Note: I tested "make install" and I get a /opt/py38/include/python3.8dm/unstable/ directory which contains a single file (yet): objimpl.h.

[ncoghlan]

I think the rules for C includes are that "path/header.h" looks next to the current file first, whereas <path/header.h> looks only in include directories.

However, given your technique of mostly hiding the new directory name from API consumers, what do you think of calling the new directory "cpython" rather than "unstable"?

The idea there would be that the "unstable ABI" eventually become known as "the CPython C API" (since it exposes a lot of CPython implementation details", while the limited API could become known as "the portable cross-implementation Python C API".

(I know, I know, you were aiming to avoid further bikeshedding on the name, but "cpython" would namespace things nicely even if a compiler does something weird with header file lookups, and helps make it clearer to CPython contributors that we still need to care about public API stability in that directory, we just don't need to care about cross-implementation portability)