Python 3.11 by filmor · Pull Request #1955 · pythonnet/pythonnet

filmor · 2022-09-29T15:18:29Z

Add Python 3.11 type offsets
Add Python 3.11 to metadata and workflows
Improve geninterop script to handle new case in 3.11
Fix offsets for 3.11

What does this implement/fix? Explain your changes.

...

Does this close any currently open issues?

...

Any other comments?

...

Checklist

Check all those that are applicable and complete.

Make sure to include one or more tests for your change
If an enhancement PR, please create docs and at best an example
Ensure you have signed the .NET Foundation CLA
Add yourself to AUTHORS
Updated the CHANGELOG

lostmsu · 2022-10-18T00:40:23Z

+ var slots = new[] {
+ new TypeSpec.Slot(TypeSlotID.tp_traverse, subtype_traverse),
+ new TypeSpec.Slot(TypeSlotID.tp_clear, subtype_clear)
+ };


This creates multiple sources of truth. Does editing post creation not work?

It doesn't. Python 3.11 rejects creating a type that has HAVE_GC set but not these slots. Rightly so :)

Can we have this array shared with the other place that used to use this function?

lostmsu · 2022-10-28T22:39:05Z

I can take a look once 3.11 hits Anaconda.

lostmsu · 2022-10-30T20:40:47Z

Re only clear on tp_dictoffset > 0. What is the scenario when it does not hold? I suspect the assert might be there to ensure something did not blow up earlier.

filmor · 2022-10-30T20:46:56Z

Re only clear on tp_dictoffset > 0. What is the scenario when it does not hold? I suspect the assert might be there to ensure something did not blow up earlier.

tp_dictoffset is only > 0 if there is actually a tp_dict, which is apparently not the case anymore for a lot of classes in Python 3.11. Fixing this fixed almost all segfaults (which were probably just unhandled .NET exceptions in the end). The only one left is the derived class in RecursiveInheritance requiring a similar setup for the slots (i.e. up-front instead of via modification) as I did in the other case. Here, it complains about tp_traverse.

lostmsu · 2022-10-30T20:48:43Z

OK, just wanted to ensure you investigated. Any specific type combinations that turn out to be an issue to make a test?

filmor · 2022-10-30T22:23:09Z

The only issue left is fixing (and reenabling tests) for Py_SetPythonHome (which is actually deprecated since 3.11). I'm not quite sure what's going on here, but something seems to fail in the encoding as it assumes in the test that the path is \0xfffd, so Unicode Replacement Character.

filmor · 2022-11-01T13:56:18Z

The remaining issue seems to be precisely calling Py_SetPythonHome with an empty string, which seems to corrupt the underlying storage. I'll investigate a bit more, but in the worst case, we should probably just forbid it.

lostmsu · 2022-11-01T17:45:45Z

LGTM, but I'll wait for the CI passing.

filmor · 2022-11-01T22:24:29Z

Wow, from my PoV this is actually a bug in Python itself:

https://github.com/python/cpython/blob/c0859743d9ad3bbd4c021200f4162cfeadc0c17a/Python/pathconfig.c#L255-L273

If Py_SetPythonHome was called with a non-empty string and is subsequently called with an empty one, it will always run PyMem_RawFree but only actually reset the pointer in .home if has_value is set, so if home && home[0] (i.e. non-empty string).

I'll try to create a proper bug-report for this, but for now I will just forbid setting PythonHome to an empty string. All of the other property setters have the same issue, AFAICT.

The issue occurs in our test-suite, because we usually start up with an empty PYTHONHOME, and then we try to update and later reset it. I don't know why this only occurs in Python 3.11, but the bug as such exists before that as well.

filmor · 2022-11-02T11:00:28Z

I have investigated the issue further: It's indeed a regression in Python 3.11, as soon as it's fixed up-stream, I'll adjust our tests to always run again.

Python 3.11

* Merge pull request pythonnet#1955 from filmor/python-3.11 Python 3.11 * Minor fix for datetime tz conversion * Version bump to 2.0.29 --------- Co-authored-by: Benedikt Reinartz <filmor@gmail.com>

Add Python 3.11 type offsets
c422abd

filmor force-pushed the python-3.11 branch from 7bf929e to 6e56a42Compare October 15, 2022 19:42

lostmsu reviewed Oct 18, 2022
View reviewed changes

Comment threadtools/geninterop/geninterop.py Outdated

milosivanovic reviewed Oct 26, 2022
View reviewed changes

Comment thread.github/workflows/main.yml Outdated

filmor mentioned this pull request Oct 28, 2022
Trying to use Python DLL results in error. #1989
Closed

filmor added 5 commits October 28, 2022 14:58

Add Python 3.11 to metadata and workflows
2b52910

Improve geninterop script to handle new case in 3.11
c0b4eb2

Fix offsets for 3.11
cc86068

Update requires-python
5636262

Define slots before initialization
8668579

filmor force-pushed the python-3.11 branch from 644c847 to 8668579Compare October 28, 2022 12:58

This comment was marked as resolved.
Sign in to view

filmor added 3 commits October 30, 2022 21:36

Only clear dict if tp_dictoffset > 0
ddf5a70

Ensure that sub-processes in tests use the same runtime settings
d3b56ff

Update MaxSupportedVersion
a6efeae

Enforce tp_traverse/clear in AllocateTypeObject
da082ac

filmor force-pushed the python-3.11 branch from 1130234 to da082acCompare October 30, 2022 22:24

Ensure that Python is initialized before probing properties
e9283e3

filmor force-pushed the python-3.11 branch from a2cb317 to e9283e3Compare November 1, 2022 12:50

Add an Action variant of TryUsingDll
cc97b8a

filmor mentioned this pull request Nov 1, 2022
Potential use-after-free in Py_SetPythonHome and its siblings python/cpython#98978
Closed

Adjust code a bit and skip PythonHome tests for empty strings
096f50a

filmor force-pushed the python-3.11 branch 2 times, most recently from a4ac4f2 to f1be5b6Compare November 2, 2022 07:30

Set PYTHONHOME for tests
15e2e95

filmor force-pushed the python-3.11 branch from f1be5b6 to 15e2e95Compare November 2, 2022 07:35

filmor marked this pull request as ready for review November 2, 2022 08:05

filmor merged commit 6968d01 into pythonnet:masterNov 2, 2022

filmor deleted the python-3.11 branch November 2, 2022 08:06

filmor referenced this pull request Jan 26, 2023
Specify supported Python version in project metadata
0119f1c

Martin-Molinero pushed a commit to Martin-Molinero/pythonnet that referenced this pull request Mar 15, 2024
Merge pull requestpythonnet#1955from filmor/python-3.11
8e48577
Python 3.11

Martin-Molinero mentioned this pull request Mar 18, 2024
Feature python 3 11 QuantConnect/pythonnet#82
Merged

Uh oh!

Conversation

filmor commented Sep 29, 2022

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Any other comments?

Checklist

Uh oh!

lostmsuOct 18, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filmorOct 18, 2022

Choose a reason for hiding this comment

Uh oh!

lostmsuOct 19, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

lostmsu commented Oct 28, 2022

Uh oh!

lostmsu commented Oct 30, 2022

Uh oh!

filmor commented Oct 30, 2022

Uh oh!

lostmsu commented Oct 30, 2022

Uh oh!

filmor commented Oct 30, 2022

Uh oh!

filmor commented Nov 1, 2022

Uh oh!

lostmsu commented Nov 1, 2022

Uh oh!

filmor commented Nov 1, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filmor commented Nov 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lostmsuOct 18, 2022•
edited
Loading

filmor commented Nov 1, 2022•
edited
Loading