Skip to content

gh-133982: Use implementation-specific open in test_fileio OtherFileTests - #135364

Merged
ZeroIntensity merged 2 commits into
python:mainfrom
cmaloney:fileio_otherfiletests
Jun 30, 2025
Merged

gh-133982: Use implementation-specific open in test_fileio OtherFileTests#135364
ZeroIntensity merged 2 commits into
python:mainfrom
cmaloney:fileio_otherfiletests

Conversation

@cmaloney

@cmaloneycmaloney commented Jun 10, 2025

Copy link
Copy Markdown
Contributor

The tests around unicode filenames use bothFileIO directly and open so kept using open for them / validates that unicode filenames passed through open() match behavior of FileIO called directly with an encoded name.

For the append tests it looks like FileIO directly is closer to what was intended, there is test_io that has append tests around self.open directly, this is test_fileio so intent I think was to test FileIO directly.

@efimov-mikhail

efimov-mikhail commented Jun 16, 2025

Copy link
Copy Markdown
Member

Is it worth backporting on 3.13? cc @Yhg1s

P.S. Congratulations with becoming Core Dev, @ZeroIntensity !!!

@ZeroIntensity

Copy link
Copy Markdown
Member

We've been backporting these to 3.13 as far as I can tell.

@cmaloney

Copy link
Copy Markdown
ContributorAuthor

Not sure policy: Is a test-only change which enables more testing of _pyio, a _io implementation CPython doesn't use directly at the moment.

The work I'm doing on top of these patches is adding another _io.Buffered* implementation which is runtime switchable, starting in a copy/paste of _io, new import that swaps implementation to ease side by side performance testing/integrating while having a working/building/running CPython. Hoping to land the feature in the 3.15 feature window, but lots of steps to get there (implementation, creating I/O benchmarks in pyperformance, testing compatibility, validating performance on range of systems, PEP, ...)

@ZeroIntensityZeroIntensity added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 30, 2025
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit e69fba0 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135364%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 30, 2025

@vstinnervstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ZeroIntensity
ZeroIntensity merged commit 23caccf into python:mainJun 30, 2025
118 of 124 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @cmaloney for the PR, and @ZeroIntensity for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 30, 2025
…therFileTests` (pythonGH-135364)
(cherry picked from commit 23caccf)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
@bedevere-app

Copy link
Copy Markdown

GH-136148 is a backport of this pull request to the 3.14 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 30, 2025
…therFileTests` (pythonGH-135364)
(cherry picked from commit 23caccf)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
@bedevere-appbedevere-appBot removed the needs backport to 3.14 bugs and security fixes label Jun 30, 2025
@bedevere-app

Copy link
Copy Markdown

GH-136149 is a backport of this pull request to the 3.13 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.13 bugs and security fixes label Jun 30, 2025
ZeroIntensity pushed a commit that referenced this pull request Jun 30, 2025
…OtherFileTests` (GH-135364) (GH-136149)
gh-133982: Use implementation-specific `open` in `test_fileio.OtherFileTests` (GH-135364)
(cherry picked from commit 23caccf)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
@cmaloney
cmaloney deleted the fileio_otherfiletests branch June 30, 2025 22:19
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x (tier-1) has failed when building commit 23caccf.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/11718) and take a look at the build logs.
  4. Check if the failure is related to this commit (23caccf) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/11718

Failed tests:

  • test_statistics

Failed subtests:

  • test_large_pool - test.test_multiprocessing_forkserver.test_misc.MiscTestCase.test_large_pool
  • test_kde_random - test.test_statistics.TestKDE.test_kde_random
  • test_framing_many_objects - test.test_pickle.CDumpPickle_LoadPickle.test_framing_many_objects

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 198, in _run_module_as_mainreturn _run_code(code, main_globals, None,
"__main__", mod_spec)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 88, in _run_codeexec(code, run_globals)
~~~~^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 6, in <module>from test.support import os_helper, Py_DEBUG
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 10, in <module>import logging
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/logging/__init__.py", line 26, in <module>import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/traceback.py", line 13, in <module>import _colorize
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/_colorize.py", line 158, in <module>@dataclass(frozen=True)
~~~~~~~~~^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 1362, in wrapreturn _process_class(cls, init, repr, eq, order, unsafe_hash,
frozen, match_args, kw_only, slots,
weakref_slot)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 1084, in _process_class
_init_fn(all_init_fields,
~~~~~~~~^^^^^^^^^^^^^^^^^
std_init_fields,
^^^^^^^^^^^^^^^^...<9 lines>...
slots,
^^^^^^
)
^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 635, in _init_fnraiseTypeError(f'non-default argument {f.name!r}'f'follows default argument {seen_default.name!r}')
TypeError: non-default argument 'reset' follows default argument 'action'
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_statistics.py", line 2510, in test_kde_random
big_sample =sorted([rand() for i inrange(n)])
~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/statistics.py", line 1122, in randreturn choice(data) + h * invcdf(random())
~~~~~~^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/random.py", line 352, in choiceifnotlen(seq):
~~~^^^^^TypeError: object of type 'Random' has no len()
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1074, in _bootstrap_innerself._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1016, in runself._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_memoryview.py", line 757, in test
obj.count(1)
~~~~~~~~~^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in countreturnsum(value == entry for entry inself)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in <genexpr>returnsum(value == entry for entry inself)
^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 439, in __getitem__self._get_packing_format(position),
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 392, in _get_packing_format
fmt = v.rstrip(b'\x00')
TypeError: a bytes-like object is required, not 'str'
Warning -- Uncaught thread exception: error
Exception in thread Thread-1 (test):
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1074, in _bootstrap_innerself._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/threading.py", line 1016, in runself._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_memoryview.py", line 757, in test
obj.count(1)
~~~~~~~~~^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in countreturnsum(value == entry for entry inself)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 532, in <genexpr>returnsum(value == entry for entry inself)
^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 439, in __getitem__self._get_packing_format(position),
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/shared_memory.py", line 387, in _get_packing_format
v = struct.unpack_from(
~~~~~~~~~~~~~~~~~~^"8s",
^^^^^self.shm.buf,
^^^^^^^^^^^^^self._offset_packing_formats + position *8^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)[0]
^struct.error: bad char in struct format
k
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/pickletester.py", line 3623, in test_framing_many_objectsself.check_frame_opcodes(pickled)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/pickletester.py", line 3572, in check_frame_opcodesfor op, arg, pos in pickletools.genops(pickled):
~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/pickletools.py", line 2291, in _genops
arg = opcode.arg.reader(data)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/pickletools.py", line 243, in read_uint2raiseValueError("not enough data in stream to read uint2")
ValueError: not enough data in stream to read uint2
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_concurrent_futures/util.py", line 54, in setUpself.manager =self.get_context().Manager()
~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 57, in Manager
m.start()
~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/managers.py", line 570, in startself._address = reader.recv()
~~~~~~~~~~~^^
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/connection.py", line 256, in recv
buf =self._recv_bytes()
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/connection.py", line 447, in _recv_bytes
buf =self._recv(4)
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/connection.py", line 416, in _recvraiseEOFError
EOFError
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6874, in test_large_poolself.assertFalse(err, msg=err.decode('utf-8'))
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'Fatal Python error: Segmentation fault\n\nCurrent thread 0x00007f8b54846740 [python] (most recent call first):\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/opcode.py", line 29 in <module>\n File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dis.py", line 8 in <module>\n File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/inspect.py", line 149 in <module>\n File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/dataclasses.py", line 5 in <module>\n File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/_colorize.py", line 6 in <module>\n File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/traceback.py", line 13 in <module>\n File "<frozen importlib._bootstrap>", line 491 in _call_with_frames_removed\n File "<frozen importlib._bootstrap_external>", line 762 in exec_module\n File "<frozen importlib._bootstrap>", line 938 in _load_unlocked\n File "<frozen importlib._bootstrap>", line 1342 in _find_and_load_unlocked\n File "<frozen importlib._bootstrap>", line 1371 in _find_and_load\n File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", lin
, at +0x30a390 [0x5556d3167390]\n Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3bfd0 [0x7f8b54884fd0]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x6a2e [0x5556d30ac271]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b2f3 [0x5556d30c82f3]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b491 [0x5556d30c8491]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyEval_EvalCode+0xb6 [0x5556d30c8566]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fb7b [0x5556d309cb7b]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fc8a [0x5556d309cc8a]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x16d209 [0x5556d2fca209]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x11a69d [0x5556d2f7769d]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x11a98d [0x5556d2f7798d]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyObject_Call+0x20 [0x5556d2f779c5]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x74f2 [0x5556d30acd35]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b2f3 [0x5556d30c82f3]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b491 [0x5556d30c8491]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x118dc1 [0x5556d2f75dc1]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x119110 [0x5556d2f76110]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x119d2c [0x5556d2f76d2c]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyObject_CallMethodObjArgs+0x101 [0x5556d2f76e53]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x2ae511 [0x5556d310b511]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyImport_ImportModuleLevelObject+0x1a3 [0x5556d310ee31]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_ImportName+0xae [0x5556d30a488f]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x11ae6 [0x5556d30b7329]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b2f3 [0x5556d30c82f3]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x26b491 [0x5556d30c8491]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyEval_EvalCode+0xb6 [0x5556d30c8566]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fb7b [0x5556d309cb7b]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x23fc8a [0x5556d309cc8a]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x16d209 [0x5556d2fca209]\n Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x11a69d [0x5556d2f7769d]\n <truncated rest of calls>\n' is not false : Fatal Python error: Segmentation fault

ZeroIntensity pushed a commit that referenced this pull request Jun 30, 2025
…OtherFileTests` (GH-135364) (GH-136148)
gh-133982: Use implementation-specific `open` in `test_fileio.OtherFileTests` (GH-135364)
(cherry picked from commit 23caccf)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
@ZeroIntensity

Copy link
Copy Markdown
Member

I'm fairly sure that isn't related, especially considering I ran the buildbots beforehand and they were all green.

@cmaloney

Copy link
Copy Markdown
ContributorAuthor

Same for me. Lots of segmentation faults and failure to unpack / not enough data across many tests which feels like running out of disk space rather than anything specific around the test_fileio changes here.

@vstinner

Copy link
Copy Markdown
Member

https://buildbot.python.org/#/builders/345/builds/11718

The following build is green. That's strange :-(

AndPuQing pushed a commit to AndPuQing/cpython that referenced this pull request Jul 11, 2025
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
picnixz pushed a commit to picnixz/cpython that referenced this pull request Jul 13, 2025
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip newstestsTests in the Lib/test dir

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@cmaloney@efimov-mikhail@ZeroIntensity@bedevere-bot@vstinner