gh-105017: Include CRLF lines in strings and column numbers by mgmacias95 · Pull Request #105030 · python/cpython

mgmacias95 · 2023-05-27T21:46:42Z

Issue: Tokenizer produces different output on Windows on py312 for ends of files #105017

pablogsal · 2023-05-27T23:11:12Z

There were a bunch of failing tests because we are removing CRLF handling in the regular path that expects the source to be normalized. I pushed a commit that only preserves CRLF in our tokenize extension

bedevere-bot · 2023-05-28T12:50:37Z

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 363de28 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

bedevere-bot · 2023-05-28T13:40:11Z

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit d3d4ff5 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

miss-islington · 2023-05-28T14:15:57Z

Thanks @mgmacias95 for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

…thonGH-105030) (cherry picked from commit 96fff35) Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

bedevere-bot · 2023-05-28T14:16:08Z

GH-105041 is a backport of this pull request to the 3.12 branch.

…H-105030) (#105041) gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (cherry picked from commit 96fff35) Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

pablogsal

LGTM

Great job! ✨

bedevere-bot · 2023-05-28T15:40:03Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.12 has failed when building commit 3f8d5d9.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1102/builds/33) and take a look at the build logs.
Check if the failure is related to this commit (3f8d5d9) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1102/builds/33

Failed tests:

test_threading
test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 16, done. remote: Counting objects: 6% (1/16) remote: Counting objects: 12% (2/16) remote: Counting objects: 18% (3/16) remote: Counting objects: 25% (4/16) remote: Counting objects: 31% (5/16) remote: Counting objects: 37% (6/16) remote: Counting objects: 43% (7/16) remote: Counting objects: 50% (8/16) remote: Counting objects: 56% (9/16) remote: Counting objects: 62% (10/16) remote: Counting objects: 68% (11/16) remote: Counting objects: 75% (12/16) remote: Counting objects: 81% (13/16) remote: Counting objects: 87% (14/16) remote: Counting objects: 93% (15/16) remote: Counting objects: 100% (16/16) remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 6% (1/15) remote: Compressing objects: 13% (2/15) remote: Compressing objects: 20% (3/15) remote: Compressing objects: 26% (4/15) remote: Compressing objects: 33% (5/15) remote: Compressing objects: 40% (6/15) remote: Compressing objects: 46% (7/15) remote: Compressing objects: 53% (8/15) remote: Compressing objects: 60% (9/15) remote: Compressing objects: 66% (10/15) remote: Compressing objects: 73% (11/15) remote: Compressing objects: 80% (12/15) remote: Compressing objects: 86% (13/15) remote: Compressing objects: 93% (14/15) remote: Compressing objects: 100% (15/15) remote: Compressing objects: 100% (15/15), done. remote: Total 16 (delta 1), reused 11 (delta 1), pack-reused 0 
From https://github.com/python/cpython
* branch 3.12 -> FETCH_HEAD
Note: switching to '3f8d5d9ed60ae0f70c92dc29587cc6367eda80c3'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 3f8d5d9ed6 [3.12] gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (#105041)
Switched to and reset branch '3.12'
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.
Fatal Python error: Segmentation fault
Thread 0x00007f253095a280 (most recent call first):
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 74 in _write_status
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 95 in addSuccess
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/testresult.py", line 122 in addSuccess
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 648 in run
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1115 in _run_suite
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1241 in run_unittest
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module>
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testcapi (total: 1)
Segmentation fault
make: *** [Makefile:2009: buildbottest] Error 139
Cannot open file '/root/buildarea/3.12.angelico-debian-amd64/build/test-results.xml' for upload

fix(tokenizer): Include CRLF lines in strings and column numbers
ad130da

mgmacias95 requested review from lysnikolaou and pablogsal as code owners May 27, 2023 21:46

bedevere-bot added the awaiting review label May 27, 2023

bedevere-bot mentioned this pull request May 27, 2023
Tokenizer produces different output on Windows on py312 for ends of files #105017
Closed

mgmacias95and others added 2 commits May 27, 2023 22:48

Merge branch 'main' into crlf_strings
6688c36

📜🤖 Added by blurb_it.
e5293bc

pablogsal added the needs backport to 3.12 only security fixes label May 27, 2023

Only preserve crlf in the Python-tokenize extension
37f77ad

pablogsal reviewed May 27, 2023
View reviewed changes

Comment threadPython/Python-tokenize.c

pablogsal changed the title ~~gh-105017: [Tokenizer] Include CRLF lines in strings and column numbers~~gh-105017: Include CRLF lines in strings and column numbersMay 27, 2023

pablogsal added 4 commits May 28, 2023 12:32

Handle \r\n in continuation lines
68e7b36

fixup! Handle \r\n in continuation lines
969060f

fixup! fixup! Handle \r\n in continuation lines
559487c

fixup! fixup! fixup! Handle \r\n in continuation lines
363de28

pablogsal reviewed May 28, 2023
View reviewed changes

Comment threadLib/test/test_tokenize.py Outdated

pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

mgmacias95and others added 2 commits May 28, 2023 14:55

Update Lib/test/test_tokenize.py
67b3a9c
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

Prevent leak
d3d4ff5

pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

pablogsal merged commit 96fff35 into python:mainMay 28, 2023

bedevere-bot removed the awaiting review label May 28, 2023

bedevere-bot removed the needs backport to 3.12 only security fixes label May 28, 2023

pablogsal reviewed May 28, 2023
View reviewed changes

picnixz mentioned this pull request May 28, 2023
Compatibility: pycode parser: adjust dedent-token handling to maintain py3.12+ compatibility sphinx-doc/sphinx#11440
Closed

erlend-aasland mentioned this pull request Jun 6, 2023
3.12 backport gh 105236 #105358
Closed

Navigation Menu

Uh oh!

Uh oh!

gh-105017: Include CRLF lines in strings and column numbers - #105030

gh-105017: Include CRLF lines in strings and column numbers#105030
pablogsal merged 10 commits into
python:mainfrom
mgmacias95:crlf_strings

mgmacias95 commented May 27, 2023•
edited by bedevere-bot
Loading

Uh oh!

Uh oh!

pablogsal commented May 27, 2023

Uh oh!

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

miss-islington commented May 28, 2023

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

pablogsal left a comment

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

mgmacias95 commented May 27, 2023• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pablogsal commented May 27, 2023

Uh oh!

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

miss-islington commented May 28, 2023

Uh oh!

bedevere-bot commented May 28, 2023

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented May 28, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mgmacias95 commented May 27, 2023•
edited by bedevere-bot
Loading