REPL shows source code from `_pyrepl.__main__.py`

[WolframAlph]

Bug report

Bug description:

When using compile built-in function and providing non-existing filename, REPL shows source lines from _pyrepl.__main__.py if exception occurs. Reproducer:

Python 3.14.0a4+ (heads/main-dirty:ed6934e71e5, Jan 20 2025, 18:59:25) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> eval(compile("val", "non-existing-file", "eval"))

Results in:

Traceback (most recent call last):
 File "<python-input-0>", line 1, in <module>
 eval(compile("val", "non-existing-file", "eval"))
 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "non-existing-file", line 1, in <module>
 # Important: don't add things to this module, as they will end up in the REPL's
NameError: name 'val' is not defined. Did you mean: 'eval'?
>>> 

traceback with following line: # Important: don't add things to this module, as they will end up in the REPL's. This is first line from Lib/_pyrepl/__main__.py.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-129098: Skip reading source lines if filename is _pyrepl.__main__.py inside REPL #129099
• gh-129098: avoid using content of _pyrepl/__main__.py when reporting tracebacks #130721
• [3.13] gh-129098: avoid using content of _pyrepl/__main__.py when reporting tracebacks (GH-130721) #132755

[picnixz]

cc @pablogsal @ambv

[WolframAlph]

@picnixz

Or we could check whether the reported traceback is correct or not (or we could hook compile itself?)

Traceback seems like good place to check for it instead of linecache, but I did not find any reference to _pyrepl.__main__ in FrameSummary (maybe it is somewhere else?). Filename there is still non-existing-file in this case. And then it passes it to linecache.getline. And from there it falls back to _py_repl.__main__ after failing to find file. FTR:

cpython/Lib/traceback.py

Lines 341 to 351 in ed6934e

	def _set_lines(self):
	if (
	self._lines is None
	and self.lineno is not None
	and self.end_lineno is not None
	):
	lines = []
	for lineno in range(self.lineno, self.end_lineno + 1):
	# treat errors (empty string) and empty lines (newline) as the same
	lines.append(linecache.getline(self.filename, lineno).rstrip())
	self._lines = "\n".join(lines) + "\n"

So I would need someone to point me in the right direction where it should be handled as I don't have much context here.

[picnixz]

I'm sorry, I wasn't explicit enough. What I meant is that we should fix _pyrepl/* and not linecache or traceback itself. Those are general-purpose utilities that should not depend on the REPL itself. The fact that the code for _pyrepl.__main__ is shown is because we're essentially executing everything here (not sure anymore). So, we need to patch the REPL files, not the other files (linecache and traceback are technically correct; it's the fact that we don't set a filename for compile that becomes problematic).

Btw, I think we have an existing issue for this one but I don't remember which one it is...

[WolframAlph]

I will try to get more context and see how it can be fixed better if that's okay.

[ambv]

The current state of things is already due to an attempt to hide _pyrepl internals. I'm not sure there's an easy way to change _pyrepl/* in a way that linecache won't cache it without a change to linecache itself.

I disagree that it's somehow invalid to patch linecache for this. What you're trying to do here is exactly to make linecache special-case _pyrepl/*, so that would be the easiest thing to do here.

[picnixz]

I'm not sure there's an easy way to change _pyrepl/* in a way that linecache won't cache it without a change to linecache itself

Wouldn't it be possible with some custom __loader__ in _pyrepl.__main__ for which get_source() returns None? (not sure if it could cause other issues though)

I disagree that it's somehow invalid to patch linecache for this. What you're trying to do here is exactly to make linecache special-case _pyrepl/*, so that would be the easiest thing to do here.

What I meant is that I'm more worried about the fact that we have some hardcoded _pyrepl.__main__ guard here. If you however think it's the easiest way to do it maybe this can motivate that choice but... it feels a bit hacky =/ OTOH, if the repl has special cases elsewhere in the stdlib, then maybe the change is fine. But if possible, it would be nicer if we could make it work in a more generic way (though, if this properly fixes the issue, it could be fine).

[ambv]

Hm, __loader__ might be a good idea to pursue. I was worried about putting any logic in _pyrepl.__main__, as anything you put there will end up in globals of the interactive shell. But since there is a default __loader__ there already, maybe this can work.

[picnixz]

By adding __spec__ = __loader__ = None to _pyrepl/__main__.py, I have the following output:

>>> eval(compile("val", "missing", "eval"))
Traceback (most recent call last):
 File "<python-input-0>", line 1, in <module>
 eval(compile("val", "missing", "eval"))
 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "missing", line 1, in <module>
NameError: name 'val' is not defined. Did you mean: 'eval'?
>>>

So it appears to work. In linecache this would do the following:

 # Try for a __loader__, if available
 if module_globals and '__name__' in module_globals: # <- ok 
 spec = module_globals.get('__spec__') # will be explicitly None
 name = getattr(spec, 'name', None) or module_globals['__name__']
 loader = getattr(spec, 'loader', None) # will be None
 if loader is None:
 loader = module_globals.get('__loader__') # will be None
 get_source = getattr(loader, 'get_source', None) # will be None

 if name and get_source: # will be skipped
 def get_lines(name=name, *args, **kwargs):
 return get_source(name, *args, **kwargs)
 cache[filename] = (get_lines,)
 return True
 return False # and we get here

[WolframAlph]

Doesn't it break other things somewhere?

[picnixz]

Doesn't it break other things somewhere?

I don't think so because we're only changing the loader and spec of a specific module. But we should investigate (I don't have time now and I'm leaving for 2 weeks tomorrow)

[StanFromIreland]

Benedkit's fix does not work for me.

The error does not occur when using a "<foo>" placeholder

>>> eval(compile("val", "<string>", "eval"))
Traceback (most recent call last):
 File "<python-input-0>", line 1, in <module>
 eval(compile("val", "<string>", "eval"))
 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "<string>", line 1, in <module>
NameError: name 'val' is not defined. Did you mean: 'eval'?