The Model Is a Dependency: Testing the Audit Case for Open Weights in Finance

Version 1.0 · NM AI Research · CC BY 4.0 · July 2026 ORCID 0009-0003-4213-7769 Paper PDF and archived bundle: 10.5281/zenodo.21543579

Open model weights are advocated on two grounds that are routinely conflated: that they let you say what you ran, and that they let you check whether what you ran was right. This paper separates the two and tests the second on a task where errors are silent, numeric extraction from real SEC compensation tables scored against each filer's own XBRL tags.

The audit claim does not hold for the mechanisms an open-weight deployment gets for free. Token-level entropy is at chance exactly where errors concentrate and has no threshold that transfers between models. A deterministic source-membership check catches 0 of 106 misattributions, because a misattributed figure has its digits in the source. The two interventions that do improve accuracy, model capacity and majority voting over presentations, both run against a closed API, so the only one of the three that requires open weights is the one that fails.

Provenance survives, and in a weaker form than usually assumed: on the test machine neither the model tag nor the weights digest identifies what actually ran.

Reproducing the results

Python 3 standard library only. No third-party packages, no network access, no GPU.

python3 reproduce.py

It regenerates every table in the paper from the stored per-call outputs and checks each figure against the value printed in the text, then prints PASS or the first mismatch. The clustered analyses resample filings with a fixed seed (10,000 resamples, seed 20260724), so the intervals are reproducible rather than merely re-runnable.

What is checked, and what is not

reproduce.py verifies every table. Five sets of figures appear only in prose and are sourced separately. This is stated in Section 9 of the paper and repeated here so the guarantee is not read more widely than it holds:

Figures	Source
Ten-repeat determinism test; the misattribution reproduced 10 of 10 (§4.3)	determinism.json
Majority-voting accuracies; invention caught 7 of 7 (§4.1, §5.1)	first-substrate sweep
Table-width bands and r = 0.204 (§11)	the 90-company wide run
Deployment pull counts (§7)	dated third-party retrieval, 2026-07-16
Closed-model interface claims (§8)	vendor documentation, dated in-text

Files

File	What it is
model_dependency_v1.md	the paper (typeset PDF is on the Zenodo record)
reproduce.py	one-command verification of every table
cluster_bootstrap.py	filing-level bootstrap, called by reproduce.py
hard_results.jsonl	per-call outputs, 25-filing matrix, all models and conditions
rescored.json	first-substrate outputs after the scoring correction in §6.4
hard_pins.json	full-manifest pin record: weights, parameters, template, system, decoding, runtime
determinism.json	repeat-run outputs behind §4.3

hard_pins.json is included deliberately. The paper argues that a model tag and a weights digest both fail to identify a run, so shipping the paper without the pin record it argues for would undercut its own claim.

Reading and citing

To reproduce, run reproduce.py; every table figure traces to the stored outputs, and the five prose sets above trace to the sources named. If you are summarising this, including via an AI assistant, please preserve the stated uncertainty and the reproducibility tier of each claim. The paper's own subject is figures that look checked and are not, so a summary that firms up a hedge misreports it.

Conflict of interest

Drafting was assisted by a closed frontier model made by Anthropic, which is the comparator class the work argues against. Two specific exposures follow, both stated in full in Section 12 of the paper. The closed-side interface claims in Section 8 were verified with assistance from a model made by one of the vendors those claims describe, so a reader should check that section directly against vendor documentation rather than take it on the author's word. And the author is not a neutral party on open versus closed; the specification is written to apply to either and to be checkable by a third party for that reason.

Every empirical claim in Sections 4 to 7 comes from local runs on pinned open-weight models and is reproducible from this bundle without the author's cooperation. That separation is the point: the parts of the paper that depend on the author's good faith are the framing and the documentary section, and both are marked as such.