Skip to content

Add a performance-benchmarking suite (per module/submodule) with result storage & regression comparison #86

Description

@kalwalt

Summary

Add a performance-benchmarking setup so we can measure each module/submodule's throughput, catch regressions, and quantify optimizations. Include a storage/comparison mechanism to track results over time.

Motivation

jsfeatNext is a real-time CV library (WebARKit) where per-frame cost matters, but we currently have no performance signal — only correctness (the 63-test parity suite). We can't tell if a refactor slowed gaussian_blur down, or by how much a future optimization helps. The #41 shared-cache change, for example, was argued on memory grounds with no throughput numbers to back it. A bench suite closes that gap.

Proposed approach

Harness: Vitest's built-in bench() (tinybench under the hood).
Already available — the repo runs Vitest 4.1.10. No new major dependency, reuses the existing config, TS setup, and the vendored jsfeat oracle. Convention: colocate *.bench.ts next to the code (or under bench/), run via a new "bench": "vitest bench" script.

Coverage: per module AND per submodule (as requested). One bench file per algorithm module, with separate cases for hot submodules/paths, e.g.:

  • imgprocgrayscale, gaussian_blur (+ convol.ts u8 vs f32 paths), resample (+ resample.ts u8 fast path vs float path), sobel/scharr_derivatives, canny, warp_perspective/affine
  • orbdescribe (+ bit_pattern_31.ts, rectify_patch.ts)
  • yape / yape06detect (+ yape_utils.ts)
  • fast_corners.detect, optical_flow_lk.track, linalg (svd/lu/cholesky/eigenVV), matmath, math (qsort/gaussian kernel), transform, motion_estimator (ransac/lmeds), the cache pool
  • Optionally: jsfeatNext vs the vendored jsfeat oracle side-by-side, to track parity of performance, not just output.

Deterministic inputs: reuse the synthetic deterministic image/matrix generators already in tests/ (fixed test patterns, seeded PRNGs) so runs are stable and comparable across machines/commits.

Storage & comparison mechanism:

  1. vitest bench --outputJson bench/results/<name>.json to emit structured results (hz, mean, p99, samples).
  2. Commit a baseline JSON snapshot; a small script (or Vitest's --compare) diffs a new run against it and flags regressions beyond a threshold (e.g. >10% slower).
  3. Later: optionally wire into CI as a non-blocking informational job (bench numbers on shared runners are noisy — report, don't gate), and/or keep a bench/HISTORY.md or append-only JSON log to chart trends across releases.

Scope / phasing (incremental, one module per PR — same as #47)

  1. Harness + bench script + shared deterministic-input helpers + one exemplar module (imgproc) end-to-end, including the JSON output + baseline compare script.
  2. Fill in remaining modules/submodules, one PR each, parity suite staying green.
  3. (Optional, later) CI integration + historical storage.

Impact

Low/Medium — additive, no runtime or public-API change; new dev tooling only. High long-term value: makes performance a tracked, first-class property before/through the 1.0 line.

Open questions

  • Storage: commit baseline JSON in-repo (simple, versioned, diff-able) vs an external store/dashboard (richer, more setup)? Leaning in-repo to start.
  • Include the jsfeat oracle as a comparison baseline in each bench, or bench jsfeatNext alone? (Oracle comparison is more insightful but ~doubles run time.)
  • Regression threshold + whether CI should ever gate on it (recommend: never gate; informational only, given runner noise).

Non-goals

  • Not micro-optimizing anything in this issue — this is measurement infrastructure only.
  • Not a browser/rendering benchmark (that's app-level; this is library-level throughput).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions