Remove bundled libffi for OSX

[zware]

BPO	28491
Nosy	@ronaldoussoren, @ned-deily, @bitdancer, @zware, @koobs, @matrixise, @yan12125
Superseder	bpo-41100: Support macOS 11 and Apple Silicon Macs
Files	remove_libffi_osx.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-08.12:07:24.418>
created_at = <Date 2016-10-20.18:40:27.517>
labels = ['OS-mac', 'ctypes', 'type-feature', '3.7', 'build']
title = 'Remove bundled libffi for OSX'
updated_at = <Date 2020-11-08.12:07:24.417>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2020-11-08.12:07:24.417>
actor = 'ronaldoussoren'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-08.12:07:24.418>
closer = 'ronaldoussoren'
components = ['Build', 'macOS', 'ctypes']
creation = <Date 2016-10-20.18:40:27.517>
creator = 'zach.ware'
dependencies = []
files = ['45157']
hgrepos = []
issue_num = 28491
keywords = ['patch']
message_count = 10.0
messages = ['279063', '279064', '279065', '279067', '279069', '279122', '279123', '289707', '379419', '380548']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'r.david.murray', 'zach.ware', 'koobs', 'matrixise', 'yan12125']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '41100'
type = 'enhancement'
url = 'https://bugs.python.org/issue28491'
versions = ['Python 3.7']

[zware]

Here's a patch that at least allows _ctypes to be built against a Homebrew libffi and pass the test_ctypes. It is almost certainly not a complete patch, but may serve as a basis for something that will actually work.

[matrixise]

I will test on my OSX (El Capitan)

[yan12125]

With this change pkg-config becomes mandatory on macOS, too. As Ned has mentioned in bpo-28207, pkg-config is not pre-installed with macOS.

[ned-deily]

Yes, we shouldn't be depending on pkg-config being available. I am not at all keen on adding a dependency on a third-party library supplied by another distributor. What I would like to see is: (1) add libffi to the third-party libs built and used for the macOS installer build, which also means on all supported versions; (2) then, more generally, refactor the third-party lib builds (e.g. OpenSSL, SQLite, xz, ncurses, et al, and then libffi) out of the installer build script and provide an option to ./configure to allow the third-party libs to be built and used in a regular developer build as the absence of or lack of updates to them in macOS releases is a growing problem. At that point we can get rid of the bundled libffi source. At the moment, AFAIK, the presence of the bundled libffi source is not causing any problems so this isn't a critical problem, unlike the case with OpenSSL.

[matrixise]

On my OSX with El Capitan and HomeBrew

Configuration:

env PKG_CONFIG_PATH=/usr/local/opt/libffi/lib/pkgconfig ./configure --prefix=$PWD-build

Compilation:

make

Tests:

./python.exe -c 'import ctypes'

Results:

• No SIGSEGV
• Add a new dependency to pkg-config :/

[ronaldoussoren]

Does the upstream libffi work reliably on OSX by this time?

The bundled version of libffi was extracted from PyObjC, and that copy is itself a patched version of the system libffi on OSX. The reason I don't use the system libffi is both that I don't want to rely on older version of libffi, and because the system libffi causes crashes in PyObjC's testsuite (which contains a lot of edge cases).

I haven't looked into using the upstream version of libffi yet. PyObjC's version of libffi is a fairly old fork of upstream libffi where support for other systems was removed and support for darwin/x86 (and later darwin/x86-64) as added. The former at a time that the upstream libffi didn't support darwin/x86 at all (and at a time that darwin/x86 itself wasn't available on consumer devices).

[matrixise]

Hi Ronald,

As you can see in my message #msg279069 I have tested with the last version of libffi (installed with HomeBrew), so in this case, I think we can say "Yes" to your question, the current release of libffi works fine with OSX. Now, I can check more deeply and give you a feedback.

[yan12125]

Update: my pull request at libffi/libffi#288 is merged. Future libffi releases don't need pkg-config anymore. However, I guess the system copy won't be updated in near future.

[ronaldoussoren]

added 41100 as a dependency because that switches to the system libffi (as a side effect of larger changes)

[ronaldoussoren]

This was implemented in bpo-41100