bpo-35879: Fix type comment leaks by gvanrossum · Pull Request #11728 · python/cpython

gvanrossum · 2019-02-01T23:03:51Z

https://bugs.python.org/issue35879

gvanrossum · 2019-02-01T23:07:38Z

Since this just fixes a bug introduced by yesterday's merge of GH-11645, I don't think we need a blurb.

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

pablogsal

LGTM!

Finally! I spent al leat 3 hours tracking this leak. How did you find that PyArena_AddPyObject was the missing piece?

pablogsal · 2019-02-01T23:27:20Z

I have run the entire test suite with ref leaks activated and it succeeds. Here is test_type_comments:

~/github/cpython remotes/upstream/fix-type-comment-leaks ✔
❯ ./python -m test test_type_comments -R 10:10
Run tests sequentially
0:00:00 load avg: 0.59 [1/1] test_type_comments
beginning 20 repetitions
12345678901234567890
....................

== Tests result: SUCCESS ==

1 test OK.

Total duration: 168 ms
Tests result: SUCCESS

gvanrossum · 2019-02-01T23:27:42Z

How did you find that PyArena_AddPyObject was the missing piece?

I kept reading the source, trying to understand how this was done for other fields. Finally I found that identifiers had the same pattern (a string == PyObject * in the struct) and then realized that those are allocated via new_identifier()/NEW_IDENTIFIER(), which was a pattern very similar to new_type_comment()/NEW_TYPE_COMMENT(), and reading through it found that call at the end. This also solved the mystery why new_type_comment() had an unused struct compiling *c argument that was passed by the NEW_TYPE_COMMENT() macro -- this was copied NEW_IDENTIFIER(), where it was needed for the arena.

bedevere-bot · 2019-02-02T05:29:08Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64 Fedora 3.x has failed when building commit d2b4c19.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/53/builds/2271) and take a look at the build logs.
Check if the failure is related to this commit (d2b4c19) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/53/builds/2271

Click to see traceback logs

Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 129, in worker
put((job, i, result))
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in putself._writer.send_bytes(obj)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytesself._send_bytes(m[offset:offset + size])
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytesself._send(header + buf)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 302, in _bootstrapself.run()
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 99, in runself._target(*self._args, **self._kwargs)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 134, in worker
put((job, i, (False, wrapped)))
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in putself._writer.send_bytes(obj)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytesself._send_bytes(m[offset:offset + size])
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytesself._send(header + buf)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process SpawnPoolWorker-229:
Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 129, in worker
put((job, i, result))
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in putself._writer.send_bytes(obj)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytesself._send_bytes(m[offset:offset + size])
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytesself._send(header + buf)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 302, in _bootstrapself.run()
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 99, in runself._target(*self._args, **self._kwargs)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 134, in worker
put((job, i, (False, wrapped)))
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in putself._writer.send_bytes(obj)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytesself._send_bytes(m[offset:offset + size])
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytesself._send(header + buf)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process SpawnPoolWorker-230:
Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 129, in worker
put((job, i, result))
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in putself._writer.send_bytes(obj)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytesself._send_bytes(m[offset:offset + size])
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytesself._send(header + buf)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 302, in _bootstrapself.run()
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 99, in runself._target(*self._args, **self._kwargs)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 134, in worker
put((job, i, (False, wrapped)))
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in putself._writer.send_bytes(obj)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytesself._send_bytes(m[offset:offset + size])
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytesself._send(header + buf)
File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/semaphore_tracker.py:164: UserWarning: semaphore_tracker: There appear to be 4 leaked semaphores to clean up at shutdown
warnings.warn('semaphore_tracker: There appear to be %d '

pablogsal · 2019-02-02T14:52:17Z

This failure is not related with this PR (is some broken pipe in multiprocessing).

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

gvanrossum added 2 commits February 1, 2019 13:31

Fix leak for # type: ignore
ef3b9e9

Fix the type comment leak
88975a8

the-knights-who-say-ni added the CLA signed label Feb 1, 2019

bedevere-bot added the awaiting merge label Feb 1, 2019

gvanrossum added the skip news label Feb 1, 2019

gvanrossum pushed a commit to python/typed_ast that referenced this pull request Feb 1, 2019
Fix memory leaks caused by type comment parsing
6f47b31
Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

pablogsal approved these changes Feb 1, 2019
View reviewed changes

gvanrossum merged commit d2b4c19 into masterFeb 1, 2019

bedevere-bot removed the awaiting merge label Feb 1, 2019

gvanrossum deleted the fix-type-comment-leaks branch February 1, 2019 23:28

gvanrossum mentioned this pull request Feb 1, 2019
bpo-35766: Merge typed_ast back into CPython #11645
Merged

gvanrossum pushed a commit to python/typed_ast that referenced this pull request Feb 5, 2019
Fix memory leaks caused by type comment parsing
b7d582a
Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

gvanrossum mentioned this pull request Feb 5, 2019
Fix memory leaks caused by type comment parsing python/typed_ast#91
Merged

gvanrossum added a commit to python/typed_ast that referenced this pull request Feb 6, 2019
Fix memory leaks caused by type comment parsing (#91)
c6d7fcb
Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

Navigation Menu

Uh oh!

Uh oh!

bpo-35879: Fix type comment leaks - #11728

bpo-35879: Fix type comment leaks#11728
gvanrossum merged 2 commits into
masterfrom
fix-type-comment-leaks

gvanrossum commented Feb 1, 2019•
edited by bedevere-bot
Loading

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

pablogsal left a comment

Uh oh!

pablogsal commented Feb 1, 2019

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

bedevere-bot commented Feb 2, 2019

Uh oh!

pablogsal commented Feb 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

gvanrossum commented Feb 1, 2019• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal commented Feb 1, 2019

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

bedevere-bot commented Feb 2, 2019

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

pablogsal commented Feb 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gvanrossum commented Feb 1, 2019•
edited by bedevere-bot
Loading