Include debugpy in debug pex when configured as debugger

[marcuscaisey]

Problem

Debugging a pex with the debugpy debugger fails because the debugpy module is not found. For example:

please.nvim (master) plz debug -o=plugin.python.debugger:debugpy //experimental/python:ping --port 8765
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 173, in <module>
result = run(explode)
File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 154, in run
return main()
File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 139, in main
start_debugger()
~~~~~~~~~~~~~~^^
File "/Users/marcus/scratch/please.nvim/plz-out/debug/experimental/python/./ping.pex/__main__.py", line 145, in start_debugger
import debugpy
ModuleNotFoundError: No module named 'debugpy'
10:43:09.028 ERROR: Failed to execute //experimental/python:ping: exit status 1

Solution

Include debugpy in the debug pex. please_pex was already trying to include it but it wasn't included in please_pex itself.

Other changes:

• Upgrade debugpy to the latest version as the version in the repo wouldn't build (there was some python 2 code that was being compiled). The wheel couldn't be found using the any of the predictable URL schemes defined by WHEEL_NAME_SCHEME in third_party/python so I just used the wheel resolver instead.
• Add the .bootstrap directory in the exploded pex directory to sys.path. debugpy is not zip safe so it needs to be imported from the non-zip-safe location.
• Add a //tools:wheel_resolver target which downloads a pre-built wheel_resolver binary. This resolves an import cycle between debugpy -> wheel_resolver -> please_pex -> debugpy.

Tested my changes by overriding the pex tool to point to the new one:

please.nvim (master) plz debug -o=plugin.python.debugger:debugpy -o=plugin.python.pextool:/Users/marcus/scratch/python-rules/plz-out/bin/tools/please_pex/please_pex //experimental/python:ping --port 8765
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
^C10:56:14.322 ERROR: Failed to execute //experimental/python:ping: signal: terminated

I also hooked up an interactive debugger as a smoke test:

[toastwaffle]

Your title says when requested, but this looks fairly unconditional to me. What am I missing?

[marcuscaisey]

The "request" I was referring to was the user specifying which debugger they want either through .plzconfig or passing the -o=plugin.python.debugger:debugpy flag (since this is not the default). Updated the title to make this more clear 🙏

[toastwaffle]

The "request" I was referring to was the user specifying which debugger they want either through .plzconfig or passing the -o=plugin.python.debugger:debugpy flag (since this is not the default). Updated the title to make this more clear 🙏

But reading your changes, aren't we going to end up including debugpy into every single pex we build, whether we're requesting it for debugging or not?

[marcuscaisey]

aren't we going to end up including debugpy into every single pex we build

Nope. My changes unconditionally include debugpy into please_pex but please_pex only includes debugpy in the pex when -d debugpy is passed to it and -d debugpy is only passed when building the pex with --config=dbg.

Example:

please.nvim (master) unzip -l $(plz build -o=plugin.python.debugger:debugpy --config=dbg //experimental/python:ping)| rg debugpy | wc -l
550
please.nvim (master) unzip -l $(plz build //experimental/python:ping)| rg debugpy | wc -l
0

[toastwaffle]

I think the FreeBSD ports in https://github.com/please-build/python-rules/blob/master/third_party/cc/cpython/BUILD will need to be updated to get the tests to pass. You should be able to look them up on https://ports.freebsd.org/cgi/ports.cgi

[toastwaffle]

#300 should fix the freebsd stuff fyi

[marcuscaisey]

#300 should fix the freebsd stuff fyi

I had to update them again (i've separated this into its own commit) 😅

I've added the pre-built wheel_resolver and updated the hashes on the debugpy target. All tests should pass now 🙏

Btw, the tests on darwin platforms (amd64 and arm64) seem flakey. Some failing jobs which passed on retry on my branch:

• https://github.com/marcuscaisey/python-rules/actions/runs/24836518712/job/72698242419
• https://github.com/marcuscaisey/python-rules/actions/runs/24836518712/job/72698242036
• https://github.com/marcuscaisey/python-rules/actions/runs/24836518712/job/72698242111
• https://github.com/marcuscaisey/python-rules/actions/runs/24836518712/job/72698242125
• https://github.com/marcuscaisey/python-rules/actions/runs/24836518712/job/72698241774
• https://github.com/marcuscaisey/python-rules/actions/runs/24836518712/job/72698241595

All looks like the same kind of failure. The libpython shared object can't be found.