ast: Different FormattedValue expressions have same col_offset information

[WeijarZ]

BPO	37458
Nosy	@ericvsmith, @lysnikolaou, @isidentical

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = None
created_at = <Date 2019-07-01.02:10:41.834>
labels = ['interpreter-core', 'type-bug', '3.10']
title = 'ast: Different FormattedValue expressions have same col_offset information'
updated_at = <Date 2020-07-04.23:02:28.764>
user = 'https://bugs.python.org/WeijarZ'

bugs.python.org fields:

activity = <Date 2020-07-04.23:02:28.764>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2019-07-01.02:10:41.834>
creator = 'Weijar Z'
dependencies = []
files = []
hgrepos = []
issue_num = 37458
keywords = []
message_count = 5.0
messages = ['346950', '346981', '372989', '372991', '373002']
nosy_count = 4.0
nosy_names = ['eric.smith', 'lys.nikolaou', 'BTaskaya', 'Weijar Z']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37458'
versions = ['Python 3.10']

[WeijarZ]

This express

f"{x}{x}{y}"

will produce ast tree:

{
"$node": "Module",
"body": [
{
"$node": "Expr",
"value": {
"$node": "JoinedStr",
"values": [
{
"$node": "FormattedValue",
"value": {
"$node": "Name",
"id": "x",
"ctx": {
"$node": "Load"
},
"lineno": 1,
"col_offset": 3
},
"conversion": -1,
"format_spec": null,
"lineno": 1,
"col_offset": 0
},
{
"$node": "FormattedValue",
"value": {
"$node": "Name",
"id": "x",
"ctx": {
"$node": "Load"
},
"lineno": 1,
"col_offset": 3
},
"conversion": -1,
"format_spec": null,
"lineno": 1,
"col_offset": 0
},
{
"$node": "FormattedValue",
"value": {
"$node": "Name",
"id": "y",
"ctx": {
"$node": "Load"
},
"lineno": 1,
"col_offset": 9
},
"conversion": -1,
"format_spec": null,
"lineno": 1,
"col_offset": 0
}
],
"lineno": 1,
"col_offset": 0
},
"lineno": 1,
"col_offset": 0
}
]
}

These two variable 'x' has same col_offset '3', is it wrong?

[ericvsmith]

I'm working on overhauling how these are calculated. But it's complex, and is taking a while.

[ericvsmith]

I still see this problem with 3.10, which I thought might have fixed this.

@lys.nikolaou: any ideas on this?

[lysnikolaou]

I still see this problem with 3.10, which I thought might have fixed this.
Nope, that's still true in 3.10.

I'm working on overhauling how these are calculated. But it's complex, and is taking a while.
In short, the FormattedValue nodes all have the exact same lineno's and col_offset's, which are also identical to those of the enclosing JoinedStr node. These values are equal to lineno and col_offset of the first STRING token and end_lineno and end_col_offset of the last STRING token (note that the grammar accepts a STRING+ node).

any ideas on this?
Moving f-string parsing to the parser, which we've been talking about lately, would solve this. But this will probably take some time, since I currently do not have the time. It's probably going to be a good project for this coming fall.

Another alternative, in case we don't want to wait until then, would be for the handwritten f-string parser to have its own instances of a lineno and col_offset, so that they can be used when the FormattedValue nodes are created. This would probably also require some effort though, so I'm not sure we want to do it, before we really know if we're gonnna proceed with the "moving f-string parsing to PEG" project.

[ericvsmith]

I think waiting until we decide what to do with the parser makes sense. This problem has been around for a while, and while it's unfortunate I don't think it's worth heroic measures to fix.