gh-98739: Update libexpat from 2.4.9 to 2.5.0 by scdub · Pull Request #98742 · python/cpython

scdub · 2022-10-26T21:03:07Z

Based this PR on the recent earlier upgrade at #97006 by @corona10

Issue: Upgrade bundled expat to 2.5.0 #98739

bedevere-bot · 2022-10-26T21:03:12Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

scdub · 2022-10-27T02:02:46Z

@hartwork I'm not sure if there are other changes that you think would be necessary for Python to upgrade to your latest release, but here's a first attempt at to get Python synced up. I believe there is a constellation of security fixes which Python will be requiring in the near term (SQLite, OpenSSL, ...) and this could potentially tag along for one of those releases.

hartwork · 2022-10-27T14:47:47Z

@scdub I don't really understand your message. Can you maybe rephrase your question for me to better understand?

hartwork

Verified using this Dockerfile:

# Copyright (c) 2022 Sebastian Pipping <sebastian@pipping.org># Licensed under the Apache License version 2.0FROM alpine
RUN apk add --update \
diffutils \
git \
sed \
&& \
git clone --depth 1 https://github.com/python/cpython cpython-main \
&& \
( cd cpython-main && git rev-parse HEAD ) \
&& \
git clone --depth 1 --branch expat-2.5.0 https://github.com/scdub/cpython scdub-expat-2-5-0 \
&& \
( cd scdub-expat-2-5-0 && git rev-parse HEAD ) \
&& \
git config --global advice.detachedHead false \
&& \
git clone --depth 1 --branch R_2_4_9 https://github.com/libexpat/libexpat libexpat_2_4_9 \
&& \
git clone --depth 1 --branch R_2_5_0 https://github.com/libexpat/libexpat libexpat_2_5_0 \
&& \
diff -r -u libexpat_2_4_9/expat/lib/ cpython-main/Modules/expat/ | tee 2-4-9.diff \
&& \
diff -r -u libexpat_2_5_0/expat/lib/ scdub-expat-2-5-0/Modules/expat/ | tee 2-5-0.diff \
&& \
sed -e '/^Only in /d' -e '/^\(+++\|---\) /d' -e '/^diff /d' -i 2-4-9.diff 2-5-0.diff \
&& \
diff -u 2-4-9.diff 2-5-0.diff \
&& \
echo 'Diff is good.'

scdub · 2022-10-27T17:52:25Z

@scdub I don't really understand your message. Can you maybe rephrase your question for me to better understand?

I applogize, I wasn't clear. I noticed that with 2.5.0, there is a new EXPAT_CONFIG_H definition for preventing multiple inclusions and wasn't sure if there was any changes required for pyexpatns.h or they can remain independent approaches.

hartwork · 2022-10-27T18:16:18Z

@scdub thanks for elaborating. The added EXPAT_CONFIG_H protection originally came up in relation to refactoring the test suite that triggered compiler warnings on redefining existing macros. I think it can safely be ignored in context of CPython.

scdub · 2022-10-27T18:26:55Z

@gpshead I'm not sure if this PR is of interest to you, but saw that you did initiate the work for the previous expat upgrade to 2.4.9.

gpshead · 2022-10-27T20:43:25Z

Confirmed that these match the libexpat 2.5.0 upstream sources from SHA256(expat-2.5.0.tar.xz)= ef2420f0232c087801abf705e89ae65f6257df6b7931d37846a193ef2e8cdcbe

with our pyexpatns.h addition and that this adds no new C APIs that need including in that "namespace" header hack.

miss-islington · 2022-10-27T20:45:16Z

Thanks @scdub for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9, 3.10, 3.11.
🐍🍒⛏🤖

bedevere-bot · 2022-10-27T20:45:33Z

GH-98784 is a backport of this pull request to the 3.11 branch.

bedevere-bot · 2022-10-27T20:45:37Z

GH-98785 is a backport of this pull request to the 3.10 branch.

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82) Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com>

bedevere-bot · 2022-10-27T20:45:43Z

GH-98786 is a backport of this pull request to the 3.9 branch.

bedevere-bot · 2022-10-27T20:45:49Z

GH-98787 is a backport of this pull request to the 3.8 branch.

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82) Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com>

bedevere-bot · 2022-10-27T20:45:55Z

GH-98788 is a backport of this pull request to the 3.7 branch.

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82) Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com>

bedevere-bot · 2022-10-27T21:57:50Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable Clang Installed 3.11 has failed when building commit c5f3f29.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1004/builds/393) and take a look at the build logs.
Check if the failure is related to this commit (c5f3f29) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1004/builds/393

Failed tests:

test_nntplib

Failed subtests:

setUpClass - test.test_nntplib.NetworkedNNTP_SSLTests
test_with_statement - test.test_nntplib.NetworkedNNTP_SSLTests.test_with_statement

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

416 tests OK.

1 test failed:
test_nntplib

17 tests skipped:
test_asdl_parser test_check_c_globals test_clinic test_devpoll
test_gdb test_ioctl test_kqueue test_launcher test_msilib
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

1 re-run test:
test_nntplib

Total duration: 2 min 41 sec

Click to see traceback logs

Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 252, in wrapped
meth(self)
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 298, in test_with_statementif re.search(r'(?i)KEY.TOO.SMALL', ssl_err.reason):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/re/__init__.py", line 176, in searchreturn _compile(pattern, flags).search(string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^TypeError: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 286, in test_with_statement
server =self.NNTP_CLASS(self.NNTP_HOST, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1028, in __init__super().__init__(host, port, user, password, readermode,
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 337, in __init__self.sock =self._create_socket(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1034, in _create_socket
sock = _encrypt_on(sock, self.ssl_context, self.host)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 295, in _encrypt_onreturn context.wrap_socket(sock, server_hostname=hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 517, in wrap_socketreturnself.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1075, in _createself.do_handshake()
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1346, in do_handshakeself._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:992)
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
cache[rtype].remove(name)
KeyError: '/psm_d193db14'
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
cache[rtype].remove(name)
KeyError: '/psm_f4548318'
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 335, in setUpClassif re.search(r'(?i)KEY.TOO.SMALL', ssl_err.reason):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/re/__init__.py", line 176, in searchreturn _compile(pattern, flags).search(string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^TypeError: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
cache[rtype].remove(name)
KeyError: '/psm_5be85ea9'
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/test/test_nntplib.py", line 332, in setUpClasscls.server =cls.NNTP_CLASS(cls.NNTP_HOST, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1028, in __init__super().__init__(host, port, user, password, readermode,
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 337, in __init__self.sock =self._create_socket(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 1034, in _create_socket
sock = _encrypt_on(sock, self.ssl_context, self.host)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/nntplib.py", line 295, in _encrypt_onreturn context.wrap_socket(sock, server_hostname=hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 517, in wrap_socketreturnself.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1075, in _createself.do_handshake()
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.11/ssl.py", line 1346, in do_handshakeself._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:992)

Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com> (cherry picked from commit 3e07f82)

* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com>

Update libexpat from 2.4.9 to 2.5.0
a73f8ee

bedevere-bot added the awaiting review label Oct 26, 2022

📜🤖 Added by blurb_it.
3db91f1

hartwork mentioned this pull request Oct 27, 2022
Release Expat 2.5.0 libexpat/libexpat#667
Closed
27 tasks

hartwork approved these changes Oct 27, 2022
View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Oct 27, 2022

gpshead added type-security A security issue release-blocker topic-XML needs backport to 3.7 needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Oct 27, 2022

gpshead self-assigned this Oct 27, 2022

gpshead approved these changes Oct 27, 2022
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 27, 2022

gpshead merged commit 3e07f82 into python:mainOct 27, 2022

bedevere-bot removed the awaiting merge label Oct 27, 2022

bedevere-bot removed the needs backport to 3.11 only security fixes label Oct 27, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 27, 2022

bedevere-bot removed the needs backport to 3.9 label Oct 27, 2022

bedevere-bot removed the needs backport to 3.8 label Oct 27, 2022

bedevere-bot removed the needs backport to 3.7 label Oct 27, 2022

gvanrossum pushed a commit to gvanrossum/cpython that referenced this pull request Oct 28, 2022
pythongh-98739: Update libexpat from 2.4.9 to 2.5.0 (python#98742)
8f4ca21
* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680. Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com>

illia-v mentioned this pull request Dec 26, 2022
Create a GitHub Actions workflow for verifying bundled libexpat files #100538
Closed

This was referenced Feb 13, 2024
Please upgrade bundled Expat to 2.6.0 (e.g. for the fix to CVE-2023-52425) #115399
Closed
gh-115399: Upgrade bundled libexpat to 2.6.0 #115431
Merged
[3.11] Upgrade bundled libexpat to 2.6.0 (GH-115399) #115468
Merged

Uh oh!

Uh oh!

Conversation

scdub commented Oct 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Oct 26, 2022

Uh oh!

scdub commented Oct 27, 2022

Uh oh!

hartwork commented Oct 27, 2022

Uh oh!

hartwork left a comment

Choose a reason for hiding this comment

Uh oh!

scdub commented Oct 27, 2022

Uh oh!

hartwork commented Oct 27, 2022

Uh oh!

scdub commented Oct 27, 2022

Uh oh!

gpshead commented Oct 27, 2022

Uh oh!

miss-islington commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

Uh oh!

bedevere-bot commented Oct 27, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

scdub commented Oct 26, 2022•
edited
Loading