gh-153804: _remote_debugging: Tachyon Oracle

[maurycy]

Please read #153804 for raison d'être.

The ultimate idea is that instead of "I generated a small script with Claude, therefore we're better than we were" we repeatedly and rigorously measure and ensure that our changes to the profiler do not make things worse or, even sometimes, improve the reported data.

The PR introduces a new script suited for Tachyon, named Tools/inspection/oracle_external_inspection.py, mostly reusing the snippets from Tools/inspection/benchmark_external_inspection.py. For each snippet it reports:

[130] 2026-07-16T10:56:25.380112000+0200 maurycy@gimel /Users/maurycy/work/cpython (tachyon-oracle e6a6696?) % sudo ./python.exe Tools/inspection/oracle_external_inspection.py --snippet flat_alternating
3.16.0a0 (heads/tachyon-oracle:e6a6696eec2, Jul 15 2026, 16:39:58) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating runs=1 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False
flat_alternating (get_stack_trace) ref_keys=6
mode samples µs errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 291259 8.72 0.00 0 0.00 ref -
blocking-cache 299140 8.45 0.00 0 0.00 0.031 0.002
live-cache 251090 6.89 16.30 1857 0.74 0.090 0.002
live-nocache 251006 6.51 16.33 2422 0.96 0.094 0.002

I was able to nicely reproduce #151424 with 63.35±2.19 impossible% rate, that matches that finding:

[130] 2026-06-21T21:36:22.319345000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ./python.exe Tools/inspection/oracle_external_inspection.py --snippet flat_alternating --rate 1 --duration 3 --runs 8
3.16.0a0 (heads/main:6679ac07d88, Jul 16 2026, 11:19:41) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating runs=8 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False
flat_alternating (get_stack_trace) ref_keys=6
mode samples µs errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 24008 19.34±0.30 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 24008 18.51±0.33 0.00±0.00 0 0.00±0.00 0.002±0.007 0.017
live-cache 21042 7.70±0.28 12.35±0.61 13333 63.35±2.19 0.319±0.035 0.025
live-nocache 20501 9.31±1.30 14.61±1.14 60 0.29±0.10 0.028±0.004 0.018

Please see Tools/inspection/oracle_external_inspection.py --help for all the flags. Not pasting it here for the brevity.

Please see Tools/inspection/oracle_external_inspection.py for a run with all the snippets.

The first two columns are relatively obvious to read. Important bits:

• for the flat_alternating we're using get_stack_trace but get_async_stack_trace is also supported for relevant snippets,
• blocking-nocache is used as the best available reference. I don't believe it's the ground truth but it's the best one that we have available (we can call it the "oracle").
• the errors% is measured similarly to how the collector does it, that is by catching the exception,
• impossible and impossible% is the number of samples that are simply impossible for a given snippet (eg: leafs simply do not match, they couldn't be called etc.)
• tvd_floor is how much TVD two samples of the same distribution would show at these sample sizes - like: what is the noise?
• tvd_excess is the TVD minus the floor, the distance that cannot be explained by the noise. The closer to the zero, the better we're here.

In short, the idea behind tvd_* is to measure the drift (bias? noise?) against the reference.

I tried to familiarize myself with the state of the literature what is the best measure of bias, drift etc. in case of statistical samplers. Maybe it's me but I couldn't find any undisputed gold standard here. I'm more than open to any other measure, and it's very likely that we'll find something better.

The PR is big but most of it is copying the snippets to a separate file. The length of the script is on par with the existing benchmarking script.

Closes#153804

• Issue: Tachyon Oracle #153804

[maurycy]

cc @pablogsal@chris-eibl@savannahostrowski

[pablogsal]

I played with this locally and I think this is very useful. The structural part is particularly convincing: with gen_alternating I consistently get 0% impossible stacks in blocking mode and around 38–61% in the live modes across different rates and both fixed and Poisson sampling. I also inspected the raw stacks and these are real impossible combinations, such as bgen being reported with drv_a as its caller and agen being reported with drv_b as its caller.

I am less convinced by the current interpretation of tvd_excess. Independent runs of the exact same blocking mode still produce positive excess, sometimes up to 0.08, so I don't think we can describe this as distance that cannot be explained by noise. Reporting the raw TVD and documenting the floor as an IID heuristic would be more precise.

I left a few comments about this and some cases that can directly affect the measurements. Overall I really like the direction, but I would fix the Poisson scheduling and adjust the TVD presentation before merging. Thanks a lot for working on this!

[pablogsal]

This does not remove the threads. mixed_workload() loops forever, so dropping these references only makes thread_count disagree with the real number of threads.

[maurycy]

Fixed. Note that it's verbatim from benchmark_external_inspection.py.

[chris-eibl]

@maurycy asked me for Windows data:

C:\_chris\cpython\PCbuild>..\python.bat ..\Tools\inspection\oracle_external_inspection.py --snippet flat_alternating
Running Release|x64 interpreter...
3.16.0a0 (heads/pr_153806:bf7006d2adf, Jul 18 2026, 20:20:10) [MSC v.1951 64 bit (AMD64)]
cases=flat_alternating runs=1 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False
flat_alternating (get_stack_trace) ref_keys=6
mode samples µs errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 90804 26.65 0.00 0 0.00 ref -
blocking-cache 83424 29.57 0.00 0 0.00 0.023 0.003
live-cache 130542 13.63 21.18 671 0.51 0.029 0.003
live-nocache 165569 10.48 19.27 869 0.52 0.009 0.003

[pablogsal]

With DEVNULL we lose the failure output, so a broken --snippet file now only reports target exited unexpectedly with code 1 and there is no way to tell why. Can we point stderr at a temp file and dump it here instead?

[maurycy]

What do you think about just inheriting stderr? It simplifies things a bit by removing a line of code, avoids the tempfile, an undrained pipe etc., at the expense of showing the RuntimeError after.

 proc = subprocess.Popen(
[sys.executable, tmp_name],
stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL,
)

I assume it will be a part of the CI, so there's always a developer reviewing.

I applied it in 813259a.

The original error is also present in the benchmark script:

cpython/Tools/inspection/benchmark_external_inspection.py

Lines 451 to 460 in b3be16d

	process=subprocess.Popen(
	[sys.executable, temp_file.name], stdout=subprocess.PIPE, stderr=subprocess.PIPE
	)
	# Give it time to start
	time.sleep(1.0)
	# Check if it's still running
	ifprocess.poll() isnotNone:
	stdout, stderr=process.communicate()

(Tbh, I initially copied some parts from the benchmark to the Oracle.)