pdb fails to access variables closed over

[anntzer]

BPO	26072
Nosy	@birkenfeld, @xdegaye, @devjoe
Files	default.patch: show proper information when Pdb fails to create a closure free_variables.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-01-10.05:16:24.039>
labels = ['3.7', 'type-bug', 'library']
title = 'pdb fails to access variables closed over'
updated_at = <Date 2016-11-22.11:54:13.971>
user = 'https://github.com/anntzer'

bugs.python.org fields:

activity = <Date 2016-11-22.11:54:13.971>
actor = 'Chun-Yu Tseng'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-01-10.05:16:24.039>
creator = 'Antony.Lee'
dependencies = []
files = ['45115', '45509']
hgrepos = []
issue_num = 26072
keywords = ['patch']
message_count = 10.0
messages = ['257888', '277181', '277578', '278761', '280912', '280930', '280933', '280988', '281016', '281469']
nosy_count = 4.0
nosy_names = ['georg.brandl', 'xdegaye', 'Jes\xc3\xbas G\xc3\xb3mez', 'Chun-Yu Tseng']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26072'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[anntzer]

Consider the following example:

 def f(x=1):
 def g():
 y = 2
 raise Exception
 g()

 f()

$ python -mpdb -ccontinue example.py
<... traceback ...>
> /tmp/example.py(4)g()
-> raise Exception
(Pdb) p x
##### this can be worked around using "up"
*** NameError: name 'x' is not defined
(Pdb) p y
2
(Pdb) p (lambda: y)()
##### this is more awkward to work around, e.g. (lambda *, y=y: y)()
*** NameError: name 'y' is not defined

Use case: I wan to pass a lambda to a numerical optimizer, but the optimizer fails using the default starting point. Within pdb, I'd like to pass a different starting point but the same lambda, to see whether this helps.

[JessGmez]

Confirming.

Another use case is the use any lambda, or function definition inside the scope of a function, for checking conditions in the REPL.

Suppose two inner functions named condition1 and condition2, and a parameter X as a Collection:

(Pdb) any(condition1(x) and condition2(x) for x in X)
*** NameError: name 'condition1' is not defined

[devjoe]

What Antony Lee mentioned are two different cases.

The former is what PDB should behave because it is not reasonable to inspect a variable does not exist in the current frame. If you want to do so, you have to reference the variable x as a closure inside inner function g in your source code before running PDB.

The latter is same as what Jesús Gómez confirmed. It's a problem about creating correct closures in an interaction prompt of PDB. It can be found in almost every versions of Python. The following are several forms of the problem:

# (1) raise exception

1 def f():
2 x = 2
3 -> import pdb; pdb.set_trace();
(Pdb) (lambda: x)()
*** NameError: name 'x' is not defined

# (2) no exception, but get the wrong value

1 x = 100
2 def f():
3 x = 2
4 -> import pdb; pdb.set_trace();
5 f()
(Pdb) x
2
(Pdb) (lambda: x)()
100

# (3) similar to (1), but this one usually makes me upset if I forget the cause of the problem

(Pdb) ll
1 def f():
2 -> import pdb; pdb.set_trace();
(Pdb) x = 5
(Pdb) [i for i in range(10) if (i % x) == 0]
*** NameError: name 'x' is not defined
(Pdb) x
5

It's easy to alleviate the problem by using interact command to start an interactive interpreter.

 1 def f():
 2 -> import pdb; pdb.set_trace();
(Pdb) x = 5
(Pdb) interact
*interactive*
>>> [i for i in range(10) if (i % x) == 0]
[0, 5]

Now the behavior looks right.

However, the problem still exists for those PDB users don't know how to pass it, and it's quite inconvenient to start an interactive interpreter every time. Maybe it's worth to keep the behavior of PDB consistent and expectable just like the behavior inside interactive interpreter.

I will try to send a patch to solve the problem in these days. Please stop me if it's inappropriate.

[devjoe]

Surrender.

After digging into the implementation of how CPython handles closures, I am sure that it is impossible to solve this problem now correctly. What I can do is create a patch to let Pdb print enough information when the problem occurs. It would look like:

(Pdb) ll
3 def f():
4 x = 1
5 -> import pdb; pdb.set_trace()
(Pdb) (lambda: x)()

 Pdb can not use the local variable 'x' to create a closure in
 your evaluation. Instead, it tries to use 'x' in globals().
 This behavior is due to the limitation of dynamic
 interpretation within a debugger session.

 Hint: type 'help interact' to check 'interact' command.

*** NameError: name 'x' is not defined

I believe it would be helpful and less annoyed for those who encounters this problem. Call for maintainers review, please.

PS.

1. Ruby does not have this problem. Whether it exists depends on how a programming language to implement closures. However, I think that what CPython do (by PyCellObject) is smart and efficient.

2. I tried to solve the problem by copying local variables to globals() (just like what interact command do), but it results in more problems. The most typical one is that if I eval globals() in such an affected environment, I will always get the wrong result.

3. I also tried to patch and evaluate again what user inputs when catching a closure-related NameError. Obviously, it is not a good idea due to side effects of evaluation.

4. The last possible solution I think of is import ast and do some hacking ... it's overkill, and it also brings up other side effects.

[devjoe]

Call for review again.

Maybe xdegaye would like to take a look?
I found this related issue: bpo-21161

[xdegaye]

It seems that the last patch in bpo-21161 fixes all the problems described here, msg 149096 explains why. Can you confirm that this is a duplicate of bpo-21161.

[devjoe]

The last patch in bpo-21161 fixes some problems but also brings up critical issues:

(Pdb) list .
1 y = 2
2
3 def f():
4 y = 9
5 z = 10
6 -> import pdb; pdb.set_trace();
7 f()
[EOF]
(Pdb) globals()['y']
9
(Pdb) global y; print(y)
9
(Pdb) globals()['z']
10

I think that we should not copy local variables to globals() while doing execution. It will always bring out the wrong result of globals().

So, the patch I proposed here is focused on "Showing Friendly Error Message" to let the users be less confused.

# The patch works when a user tries to bound a free variable to a list comprehension. It will show the proper error message, too.

[xdegaye]

This patch fixes the problems raised in this issue and allows accessing the globals at the Pdb prompt with the globals() dictionary:

(Pdb) list
1 y = 2
2
3 def f():
4 y = 9
5 z = 10
6 -> import pdb; pdb.set_trace();
7 f()
[EOF]
(Pdb) globals()['y']
2

[devjoe]

Your solution is quite neat.
But it still misses use cases of the global statement:

1 y = 2
2
3 def f():
4 y = 9
5 -> import pdb; pdb.set_trace();
6
7 f()
(Pdb) global y; y
9
(Pdb) global y; y += 1; y
10
(Pdb) globals()['y']
2

[devjoe]

Hey xdegaye, have you confirmed it?