ast.unparse is not working correctly for strings with multiple quotes

[sunmy2019]

Bug report

CPython versions tested on:

3.11

Operating systems tested on:

macOS

A clear and concise description of the bug:

During the implementation of #108553, I discovered this bug in the original implementation.

Due to the new syntax, the fix for 3.12 above cannot be reused for 3.11 and below.

import ast

s = "f\"'''{1}\\\"\\\"\\\"\" "
print(s)
eval(s)

s = ast.parse(s)
s1 = ast.unparse(s)
print(s1)
eval(s1)

output:

f"'''{1}\"\"\"" 
f''''{1}"""'
Traceback (most recent call last):
 File "/Users/xxx/work/cpython/a.py", line 10, in <module>
 eval(s1)
 File "<string>", line 1
 f''''{1}"""'
 ^
SyntaxError: unterminated triple-quoted string literal (detected at line 1)

Linked PRs

• [3.11] gh-108843: fix ast.unparse for f-string with many quotes #108980
• gh-108843: fix ast.unparse for f-string with many quotes #108981
• [3.12] gh-108843: fix ast.unparse for f-string with many quotes (GH-108981) #109541

[hauntsaninja]

Nice spot, I've opened a PR that fixes on 3.11 over here: #108980

[hauntsaninja]

@sunmy2019 oh hmm, I got a little confused by the history here, looks like this isn't actually fixed on main?

A similar patch should make it work on main, something like the following:

Details

diff --git a/Lib/ast.py b/Lib/ast.py
index 17ec7ff6f8..f706d1f282 100644
--- a/Lib/ast.py
+++ b/Lib/ast.py
@@ -1236,17 +1236,35 @@ def visit_JoinedStr(self, node):
 
 new_fstring_parts = []
 quote_types = list(_ALL_QUOTES)
+ fallback_to_repr = False
 for value, is_constant in fstring_parts:
 if is_constant:
- value, quote_types = self._str_literal_helper(
+ value, new_quote_types = self._str_literal_helper(
 value,
 quote_types=quote_types,
 escape_special_whitespace=True,
 )
+ if set(new_quote_types).isdisjoint(quote_types):
+ fallback_to_repr = True
+ break
+ quote_types = new_quote_types
 elif "\n" in value:
 quote_types = [q for q in quote_types if q in _MULTI_QUOTES]
 new_fstring_parts.append(value)
 
+ if fallback_to_repr:
+ # If we weren't able to find a quote type that works for all parts
+ # of the JoinedStr, fallback to using repr and triple single quotes.
+ quote_types = ["'''"]
+ new_fstring_parts.clear()
+ for value, is_constant in fstring_parts:
+ if is_constant:
+ value = repr("'" + '"' + value) # force repr to escape all quotes
+ expected_prefix = r"'\'" + '"'
+ assert value.startswith(expected_prefix), repr(value)
+ value = value[len(expected_prefix):-1]
+ new_fstring_parts.append(value)
+
 value = "".join(new_fstring_parts)
 quote_type = quote_types[0]
 self.write(f"{quote_type}{value}{quote_type}")
diff --git a/Lib/test/test_unparse.py b/Lib/test/test_unparse.py
index 38c59e6d43..c09865894d 100644
--- a/Lib/test/test_unparse.py
+++ b/Lib/test/test_unparse.py
@@ -635,6 +635,16 @@ def test_star_expr_assign_target_multiple(self):
 self.check_src_roundtrip("[a, b] = [c, d] = [e, f] = g")
 self.check_src_roundtrip("a, b = [c, d] = e, f = g")
 
+ def test_multiquote_joined_string(self):
+ self.check_ast_roundtrip("f\"'''{1}\\\"\\\"\\\"\" ")
+ self.check_ast_roundtrip("""f"'''{1}""\\"" """)
+ self.check_ast_roundtrip("""f'""\"{1}''' """)
+ self.check_ast_roundtrip("""f'""\"{1}""\\"' """)
+
+ self.check_ast_roundtrip("""f"'''{"\\n"}""\\"" """)
+ self.check_ast_roundtrip("""f'""\"{"\\n"}''' """)
+ self.check_ast_roundtrip("""f'""\"{"\\n"}""\\"' """)
+
 
 class ManualASTCreationTestCase(unittest.TestCase):
 """Test that AST nodes created without a type_params field unparse correctly."""

Should I open a PR?

[sunmy2019]

I got a little confused by the history here, looks like this isn't actually fixed on main?

Yes. I got it reverted as requested to separate the fix out.

Should I open a PR?

Go ahead!