Skip to content

Add golden-output regression tests for mPIE and qNewton - #90

Merged
ShantanuKodgirwar merged 1 commit into
mainfrom
regression-tests
Aug 3, 2026
Merged

Add golden-output regression tests for mPIE and qNewton#90
ShantanuKodgirwar merged 1 commit into
mainfrom
regression-tests

Conversation

@ShantanuKodgirwar

Copy link
Copy Markdown
Contributor

What this does

Adds golden-output regression tests that pin the numerical output of mPIE and qNewton.

There is currently no test that runs a reconstruction and checks its result. That makes any
optimisation of the engines unverifiable — you cannot tell a speedup from a silent change in
what the library computes. This PR is the safety net that later performance work is checked
against.

No PtyLab/ source is touched. Tests only.

How it works

Each config runs a few iterations on a deterministic synthetic dataset built in-test, then
compares object, probe and error against a recorded .npz baseline.

The dataset is synthesised rather than loaded from example_data/, so the baselines are
reproducible in CI.

Configs covered

nameenginemodes (nlambda, nosm, npsm, nslice)
mpie_singlemPIE1, 1, 1, 1
mpie_mixed_statemPIE1, 2, 3, 1
qnewton_singleqNewton1, 1, 1, 1

The mixed-state config is the important one: it exercises the 6D mode broadcasting that a
single-mode-only suite would let a broken change through.

Pinning the randomness

Three sources of nondeterminism, all pinned:

  • params.positionOrder = "sequential""random" shuffles via the global RNG.
  • np.random.seed() before initializeObjectProbe() — it adds noise to break mode degeneracy.
  • np.random.seed() again before reconstruct() — mPIE fires its momentum update on
    np.random.rand().

Verified that the baselines reproduce bit-exactly, and that a 0.1% change to betaObject
is caught.

GPU-vs-CPU test

Asserts the two backends agree, using a Frobenius-norm relative error rather than an
elementwise one.

Elementwise is the wrong metric here: these arrays contain near-zero elements where a 1e-7
absolute wobble reads as a huge relative error. Measured divergence across these configs is
1e-8 to 2e-4, so the 1e-3 bound leaves roughly 5x headroom while still catching a
genuinely wrong kernel.

Why ePIE and e3PIE are absent

Deliberate — both would record a meaningless baseline today:

  • ePIE.reconstruct() is a generator that callers do not iterate, so it returns without
    doing any work.
  • e3PIE raises on its own betaProbe.

They are added in the follow-up PRs that repair them.

Re-recording

After an intended numerical change:

PTYLAB_REGEN_GOLDENS=1 uv run pytest tests/regression -q

Then re-run to confirm the new baselines reproduce, and review the .npz diff before
committing.

Pin the numerical output of mPIE and qNewton so that later performance work
cannot silently change what the library computes. There is currently no test
that runs a reconstruction and checks its result, which makes any optimisation
of the engines unverifiable.
Each config runs a few iterations on a deterministic synthetic dataset built
in-test and compares object, probe and error against a recorded .npz baseline.
The dataset is synthesized rather than loaded from example_data/ so the
baselines are reproducible in CI, where simu.hdf5 is a different file than it is
locally.
Every source of randomness is pinned: positionOrder is forced to sequential,
and the global numpy seed is set before initializeObjectProbe() (which adds
noise to break mode degeneracy) and again before reconstruct() (mPIE fires its
momentum update on np.random.rand). Verified that the baselines reproduce
bit-exactly and that a 0.1% change to betaObject is caught.
Coverage includes a mixed-state (nosm=2, npsm=3) config, which exercises the 6D
mode broadcasting that single-mode tests would miss. A GPU-vs-CPU test asserts
the two backends agree; it uses a Frobenius-norm relative error rather than an
elementwise one, since these arrays contain near-zero elements where a 1e-7
absolute wobble reads as a large relative error. Measured divergence is 1e-8 to
2e-4, so the 1e-3 bound leaves headroom while still catching a wrong kernel.
ePIE and e3PIE are deliberately absent: ePIE.reconstruct() is a generator that
callers do not iterate, so it returns without doing any work, and e3PIE raises
on its own betaProbe. Pinning either now would record a meaningless baseline.
They are added in the changes that repair them.
Re-record with PTYLAB_REGEN_GOLDENS=1 after an intended numerical change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@ShantanuKodgirwar