Skip to content

[3.14] gh-90949: expose Expat API to tune exponential expansion protections (GH-139368)#150496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
StanFromIreland merged 4 commits into from
Jun 9, 2026
Merged

[3.14] gh-90949: expose Expat API to tune exponential expansion protections (GH-139368) #150496

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8615892
[3.14] gh-90949: expose Expat API to tune exponential expansion prote…
picnixz Sep 28, 2025
1493ed4
Fix some extraneous whitespace
StanFromIreland May 27, 2026
6f53983
Whoops, re-indent after a previous un-indent
StanFromIreland Jun 9, 2026
a35a24a
Apply suggestion from @StanFromIreland
StanFromIreland Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 63 additions & 5 deletions Doc/library/pyexpat.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -238,16 +238,71 @@ XMLParser Objects
.. versionadded:: 3.13


:class:`!xmlparser` objects have the following methods to mitigate some
common XML vulnerabilities.
:class:`!xmlparser` objects have the following methods to tune protections
against some common XML vulnerabilities.

.. method:: xmlparser.SetBillionLaughsAttackProtectionActivationThreshold(threshold, /)

Sets the number of output bytes needed to activate protection against
`billion laughs`_ attacks.

The number of output bytes includes amplification from entity expansion
and reading DTD files.

Parser objects usually have a protection activation threshold of 8 MiB,
but the actual default value depends on the underlying Expat library.

An :exc:`ExpatError` is raised if this method is called on a
|xml-non-root-parser| parser.
The corresponding :attr:`~ExpatError.lineno` and :attr:`~ExpatError.offset`
should not be used as they may have no special meaning.

.. note::

Activation thresholds below 4 MiB are known to break support for DITA 1.3
payload and are hence not recommended.

.. versionadded:: next

.. method:: xmlparser.SetBillionLaughsAttackProtectionMaximumAmplification(max_factor, /)

Sets the maximum tolerated amplification factor for protection against
`billion laughs`_ attacks.

The amplification factor is calculated as ``(direct + indirect) / direct``
while parsing, where ``direct`` is the number of bytes read from
the primary document in parsing and ``indirect`` is the number of
bytes added by expanding entities and reading of external DTD files.

The *max_factor* value must be a non-NaN :class:`float` value greater than
or equal to 1.0. Peak amplifications of factor 15,000 for the entire payload
and of factor 30,000 in the middle of parsing have been observed with small
benign files in practice. In particular, the activation threshold should be
carefully chosen to avoid false positives.

Parser objects usually have a maximum amplification factor of 100,
but the actual default value depends on the underlying Expat library.

An :exc:`ExpatError` is raised if this method is called on a
|xml-non-root-parser| parser or if *max_factor* is outside the valid range.
The corresponding :attr:`~ExpatError.lineno` and :attr:`~ExpatError.offset`
should not be used as they may have no special meaning.

.. note::

The maximum amplification factor is only considered if the threshold
that can be adjusted by :meth:`.SetBillionLaughsAttackProtectionActivationThreshold`
is exceeded.

.. versionadded:: next

.. method:: xmlparser.SetAllocTrackerActivationThreshold(threshold, /)

Sets the number of allocated bytes of dynamic memory needed to activate
protection against disproportionate use of RAM.

By default, parser objects have an allocation activation threshold of 64 MiB,
or equivalently 67,108,864 bytes.
Parser objects usually have an allocation activation threshold of 64 MiB,
but the actual default value depends on the underlying Expat library.

An :exc:`ExpatError` is raised if this method is called on a
|xml-non-root-parser| parser.
Expand All@@ -271,7 +326,8 @@ common XML vulnerabilities.
near the start of parsing even with benign files in practice. In particular,
the activation threshold should be carefully chosen to avoid false positives.

By default, parser objects have a maximum amplification factor of 100.0.
Parser objects usually have a maximum amplification factor of 100,
but the actual default value depends on the underlying Expat library.

An :exc:`ExpatError` is raised if this method is called on a
|xml-non-root-parser| parser or if *max_factor* is outside the valid range.
Expand DownExpand Up@@ -1019,4 +1075,6 @@ The ``errors`` module has the following attributes:
not. See https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl
and https://www.iana.org/assignments/character-sets/character-sets.xhtml.


.. _billion laughs: https://en.wikipedia.org/wiki/Billion_laughs_attack
.. |xml-non-root-parser| replace:: :ref:`non-root <xmlparser-non-root>`
5 changes: 5 additions & 0 deletions Include/pyexpat.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -57,6 +57,11 @@ struct PyExpat_CAPI
XML_Parser parser, unsigned long long activationThresholdBytes);
XML_Bool (*SetAllocTrackerMaximumAmplification)(
XML_Parser parser, float maxAmplificationFactor);
/* might be NULL for expat < 2.4.0 */
XML_Bool (*SetBillionLaughsAttackProtectionActivationThreshold)(
XML_Parser parser, unsigned long long activationThresholdBytes);
XML_Bool (*SetBillionLaughsAttackProtectionMaximumAmplification)(
XML_Parser parser, float maxAmplificationFactor);
/* always add new stuff to the end! */
};

58 changes: 58 additions & 0 deletions Lib/test/test_pyexpat.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1183,6 +1183,64 @@ def test_set_maximum_amplification__fail_for_subparser(self):
self.assert_root_parser_failure(setter, 123.45)


@unittest.skipIf(expat.version_info < (2, 4, 0), "requires Expat >= 2.4.0")

@picnixzpicnixzMay 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we had a recent issue where this check was not sufficient and tests needed to be disabled in another way using hasattr() checks. Can you check how it's currently done on main please?

@StanFromIrelandStanFromIrelandMay 27, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's identical on main?

cpython/Lib/test/test_pyexpat.py

Lines 1265 to 1266 in d8ff4f8

@unittest.skipIf(expat.version_info< (2, 4, 0), "requires Expat >= 2.4.0")
classExpansionProtectionTest(AttackProtectionTestBase, unittest.TestCase):

@picnixzpicnixzMay 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah indeed, but the problem was for the other API:

@unittest.skipIf(not hasattr(expat.XMLParserType,
"SetAllocTrackerMaximumAmplification"),
"requires Python compiled with Expat >= 2.7.2")

So I think the version check for billion laughs is also wrong and should be changed to the same as for SetAllocTrackerMaximumAmplification. But let's address this separately later in a follow-up PR (so as to ease backports)

class ExpansionProtectionTest(AttackProtectionTestBase, unittest.TestCase):

def assert_rejected(self, func, /, *args, **kwargs):
"""Check that func(*args, **kwargs) hits the allocation limit."""
msg = (
r"limit on input amplification factor \(from DTD and entities\) "
r"breached: line \d+, column \d+"
)
self.assertRaisesRegex(expat.ExpatError, msg, func, *args, **kwargs)

def set_activation_threshold(self, parser, threshold):
return parser.SetBillionLaughsAttackProtectionActivationThreshold(threshold)

def set_maximum_amplification(self, parser, max_factor):
return parser.SetBillionLaughsAttackProtectionMaximumAmplification(max_factor)

def test_set_activation_threshold__threshold_reached(self):
parser = expat.ParserCreate()
# Choose a threshold expected to be always reached.
self.set_activation_threshold(parser, 3)
# Check that the threshold is reached by choosing a small factor
# and a payload whose peak amplification factor exceeds it.
self.assertIsNone(self.set_maximum_amplification(parser, 1.0))
payload = self.exponential_expansion_payload(ncols=10, nrows=4)
self.assert_rejected(parser.Parse, payload, True)

def test_set_activation_threshold__threshold_not_reached(self):
parser = expat.ParserCreate()
# Choose a threshold expected to be never reached.
self.set_activation_threshold(parser, pow(10, 5))
# Check that the threshold is reached by choosing a small factor
# and a payload whose peak amplification factor exceeds it.
self.assertIsNone(self.set_maximum_amplification(parser, 1.0))
payload = self.exponential_expansion_payload(ncols=10, nrows=4)
self.assertIsNotNone(parser.Parse(payload, True))

def test_set_maximum_amplification__amplification_exceeded(self):
parser = expat.ParserCreate()
# Unconditionally enable maximum activation factor.
self.set_activation_threshold(parser, 0)
# Choose a max amplification factor expected to always be exceeded.
self.assertIsNone(self.set_maximum_amplification(parser, 1.0))
# Craft a payload for which the peak amplification factor is > 1.0.
payload = self.exponential_expansion_payload(ncols=1, nrows=2)
self.assert_rejected(parser.Parse, payload, True)

def test_set_maximum_amplification__amplification_not_exceeded(self):
parser = expat.ParserCreate()
# Unconditionally enable maximum activation factor.
self.set_activation_threshold(parser, 0)
# Choose a max amplification factor expected to never be exceeded.
self.assertIsNone(self.set_maximum_amplification(parser, 1e4))
# Craft a payload for which the peak amplification factor is < 1e4.
payload = self.exponential_expansion_payload(ncols=1, nrows=2)
self.assertIsNotNone(parser.Parse(payload, True))


@unittest.skipIf(not hasattr(expat.XMLParserType,
"SetAllocTrackerMaximumAmplification"),
"requires Python compiled with Expat >= 2.7.2")
Expand Down
6 changes: 3 additions & 3 deletions Misc/NEWS.d/3.14.1.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -749,9 +749,9 @@ fix: paste zero-width in default repl width is wrong.

Add :meth:`~xml.parsers.expat.xmlparser.SetAllocTrackerActivationThreshold`
and :meth:`~xml.parsers.expat.xmlparser.SetAllocTrackerMaximumAmplification`
to :ref:`xmlparser <xmlparser-objects>` objects to prevent use of
disproportional amounts of dynamic memory from within an Expat parser. Patch
by Bénédikt Tran.
to :ref:`xmlparser <xmlparser-objects>` objects to tune protections against
disproportional amounts of dynamic memory usage from within an Expat parser.
Patch by Bénédikt Tran.

..

Expand Down
7 changes: 7 additions & 0 deletions Misc/NEWS.d/next/Library/2025-09-26-18-04-28.gh-issue-90949.YHjSzX.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
Add
:meth:`~xml.parsers.expat.xmlparser.SetBillionLaughsAttackProtectionActivationThreshold`
and
:meth:`~xml.parsers.expat.xmlparser.SetBillionLaughsAttackProtectionMaximumAmplification`
to :ref:`xmlparser <xmlparser-objects>` objects to tune protections against
`billion laughs <https://en.wikipedia.org/wiki/Billion_laughs_attack>`_ attacks.
Patch by Bénédikt Tran.
151 changes: 147 additions & 4 deletions Modules/clinic/pyexpat.c.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading