Uh oh!
There was an error while loading. Please reload this page.
Add golden-output regression tests for mPIE and qNewton - #90
Merged
Conversation
Pin the numerical output of mPIE and qNewton so that later performance work cannot silently change what the library computes. There is currently no test that runs a reconstruction and checks its result, which makes any optimisation of the engines unverifiable. Each config runs a few iterations on a deterministic synthetic dataset built in-test and compares object, probe and error against a recorded .npz baseline. The dataset is synthesized rather than loaded from example_data/ so the baselines are reproducible in CI, where simu.hdf5 is a different file than it is locally. Every source of randomness is pinned: positionOrder is forced to sequential, and the global numpy seed is set before initializeObjectProbe() (which adds noise to break mode degeneracy) and again before reconstruct() (mPIE fires its momentum update on np.random.rand). Verified that the baselines reproduce bit-exactly and that a 0.1% change to betaObject is caught. Coverage includes a mixed-state (nosm=2, npsm=3) config, which exercises the 6D mode broadcasting that single-mode tests would miss. A GPU-vs-CPU test asserts the two backends agree; it uses a Frobenius-norm relative error rather than an elementwise one, since these arrays contain near-zero elements where a 1e-7 absolute wobble reads as a large relative error. Measured divergence is 1e-8 to 2e-4, so the 1e-3 bound leaves headroom while still catching a wrong kernel. ePIE and e3PIE are deliberately absent: ePIE.reconstruct() is a generator that callers do not iterate, so it returns without doing any work, and e3PIE raises on its own betaProbe. Pinning either now would record a meaningless baseline. They are added in the changes that repair them. Re-record with PTYLAB_REGEN_GOLDENS=1 after an intended numerical change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds golden-output regression tests that pin the numerical output of mPIE and qNewton.
There is currently no test that runs a reconstruction and checks its result. That makes any
optimisation of the engines unverifiable — you cannot tell a speedup from a silent change in
what the library computes. This PR is the safety net that later performance work is checked
against.
No
PtyLab/source is touched. Tests only.How it works
Each config runs a few iterations on a deterministic synthetic dataset built in-test, then
compares
object,probeanderroragainst a recorded.npzbaseline.The dataset is synthesised rather than loaded from
example_data/, so the baselines arereproducible in CI.
Configs covered
mpie_singlempie_mixed_stateqnewton_singleThe mixed-state config is the important one: it exercises the 6D mode broadcasting that a
single-mode-only suite would let a broken change through.
Pinning the randomness
Three sources of nondeterminism, all pinned:
params.positionOrder = "sequential"—"random"shuffles via the global RNG.np.random.seed()beforeinitializeObjectProbe()— it adds noise to break mode degeneracy.np.random.seed()again beforereconstruct()— mPIE fires its momentum update onnp.random.rand().Verified that the baselines reproduce bit-exactly, and that a 0.1% change to
betaObjectis caught.
GPU-vs-CPU test
Asserts the two backends agree, using a Frobenius-norm relative error rather than an
elementwise one.
Elementwise is the wrong metric here: these arrays contain near-zero elements where a
1e-7absolute wobble reads as a huge relative error. Measured divergence across these configs is
1e-8to2e-4, so the1e-3bound leaves roughly 5x headroom while still catching agenuinely wrong kernel.
Why ePIE and e3PIE are absent
Deliberate — both would record a meaningless baseline today:
ePIE.reconstruct()is a generator that callers do not iterate, so it returns withoutdoing any work.
e3PIEraises on its ownbetaProbe.They are added in the follow-up PRs that repair them.
Re-recording
After an intended numerical change:
Then re-run to confirm the new baselines reproduce, and review the
.npzdiff beforecommitting.