gh-140009: Optimize JSON parsing with object_pairs_hook via PyTuple_FromArray

[andrewloux]

Summary

This PR optimizes the object_pairs_hook path in Modules/_json.c (e.g., used by json.loads(s, object_pairs_hook=list)) by replacing PyTuple_Pack with PyTuple_FromArray.

PyTuple_Pack processes arguments variadically via va_list, while PyTuple_FromArray performs a direct memcpy from a stack-allocated array. For a fixed-size-2 tuple constructed on every key-value pair in the hot path, this eliminates unnecessary overhead.

Benchmarks (PGO+LTO)

Validated using pyperf in --rigorous mode on a full production build. Results were reproduced across multiple independent sessions.

• Platform: macOS arm64 (Apple M-series)
• Build:--enable-optimizations --with-lto
• Tool: pyperf (--rigorous mode)
• Baseline:upstream/main
• Candidate:pytuple-json-objectpairs-fromarray (92d3f1a)

Benchmark	Baseline (Mean ± Std Dev)	Candidate (Mean ± Std Dev)	Speedup
json_pairs_hook_dense	111 ms ± 9 ms	108 ms ± 3 ms	1.02x faster
json_pairs_hook_control	89.2 ms ± 2.1 ms	89.5 ms ± 1.9 ms	Neutral

Geometric mean: 1.01x faster

Benchmark script and repro commands

Repro commands (using bench_json_pairs_hook.py):

# Target: parsing with object_pairs_hook=list
python -m pyperf command --rigorous --name json_pairs_hook_dense
./python.exe bench_json_pairs_hook.py pairs 320 64 64
# Control: standard parsing (no hook)
python -m pyperf command --rigorous --name json_pairs_hook_control
./python.exe bench_json_pairs_hook.py control 320 64 64

bench_json_pairs_hook.py:

importjsonimportsysdefbuild_payload(objects, fields):
obj="{"+",".join(f'"k{i}":{i}'foriinrange(fields)) +"}"return"["+",".join(objfor_inrange(objects)) +"]"defmain():
mode, loops, objects, fields=sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4])
payload=build_payload(objects, fields)
for_inrange(loops):
ifmode=="pairs":
json.loads(payload, object_pairs_hook=list)
else:
json.loads(payload)
if__name__=="__main__":
main()

Analysis

The json_pairs_hook_dense benchmark parses a large JSON array of objects using object_pairs_hook=list. In this scenario, every key-value pair requires a size-2 tuple. The switch to the array-based API yields a ~2% speedup on this specific codepath, with a conservative geometric mean of ~1% across both benchmarks.

Notably, the candidate also shows a significant reduction in variance (±9 ms → ±3 ms), suggesting more deterministic performance on the optimized path.

The json_pairs_hook_control case confirms that the standard JSON decoding path (using the default decoder) is unaffected.

• Issue: Improve performance by replacing PyTuple_Pack with PyTuple_FromArray #140009

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[caje731]

I could actually see speedups for both on my Apple M1 Pro:

Benchmark	base	candidate
json_pairs_hook_control	158 ms	155 ms: 1.01x faster
json_pairs_hook_dense	170 ms	161 ms: 1.05x faster

👍

[picnixz]

Can your benchmarks not include the time for constructing the payload please? because this benchmark is not relevant otherwise (noise from constructing the payload for instance)

[github-actions]

This PR is stale because it has been open for 30 days with no activity.