Close the gaps: the checking step, and the defects it found - #2
Open
projectoverseer wants to merge 83 commits into
Open
projectoverseer wants to merge 83 commits into
projectoverseer wants to merge 83 commits into
Conversation
A full-profile analysis of a four-minute track is comfortably past 5 MB, which is the per-file cap on an upload to Notion and to most other places a measurement archive ends up. A file that cannot be uploaded is a file that stays on one machine. analysis.json is now written whole while it fits under the part size cap (default 4.5 MB) and as an index plus analysis.partNN.json when it does not: the headline, run, params and every other small section stay inline next to a `split` manifest, and the heavy sections move into parts tagged with the path they belong at. Long timelines are cut into consecutive, absolute slices. Fragments are measured as files, header and indent included, so every part really is under the cap. Nothing is dropped, rounded or summarised -- `mtx join` (or mtx.split.load_analysis) rebuilds a document identical to what a --no-split run writes. --max-part-size and --no-split apply to analyze, batch and compare alike; comparison.json splits by the same rule. schema_version does not move: no measured field changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mtx enrich` is the only part of mtx that touches the network, and it is off unless invoked. It writes a sidecar `online.json` beside `analysis.json` -- never into it, because `mtx analyze` guarantees byte-identical output for the same input and a section built from whatever MusicBrainz looked like this morning cannot live inside that guarantee. Keyless: MusicBrainz, Deezer, Apple/iTunes. Optional with credentials: Last.fm, Discogs. Stdlib only, so enrichment adds no dependency. On a 64-track corpus this took genre from 34/64 -- and those were shop shelf labels like `Miscellaneous` and `Film Soundtracks` -- to 64/64, with songwriters on 62 and credits confirmed by two independent sources on 47. Three decisions the tests pin down: A database row is not accepted because the ISRC matched. `bad guy` returns three MusicBrainz recordings and lists a 175 s radio edit first; the file is 194 s. Candidates are scored against the measured duration and the losers are kept in the output with their scores, so a wrong match is auditable. The genre vote scales each source against its own top vote, not its sum. Sharing the sum lets a shop returning the single word `Alternative` outvote a database returning nine precise genres. Search terms are not comparison keys. `search_title` strips packaging so a reissue titled `Fly Me To The Moon (Dolby Atmos)` is findable, while keeping real parentheticals like `Marea (we've lost dancing)` intact. Disagreement is the output rather than something to smooth away: `cross_checks` keeps mtx's tempo estimate beside a published BPM and reports `agree`, `octave` (a metrical-level disagreement, not a tempo one), `triplet` or `disagree`. Agreement promotes a low-confidence estimate to high; an octave verdict resolves at medium and keeps both readings, because which level to call "the tempo" is a judgment neither source made. CI gains a step asserting enrichment degrades cleanly offline and never writes into analysis.json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mtx scan` takes an album, an artist or a whole library and measures what has not been measured yet. Scope follows the path; the destination does not. A track analysed as part of a library scan and the same track analysed on its own land in the same folder of a mirror tree, so a scan from any level finds the earlier work and skips it. The library root that makes that mapping stable is recorded once, in the user config directory -- the music folder is never written to. Each output folder carries `mtx_source.json`: the file it came from, its size, modification time and hash, the profile and the schema version. A scan reads those instead of the audio, so an already-measured library is re-checked in under a second, and an interrupted scan resumes because every finished track wrote its own receipt. `--recheck` compares hashes instead of modification times, for the case where a library has been copied between drives. Profiled first, then parallelised where the profile said to. A full-profile run of a 3:54 track is 51 s, one third of it the true-peak oversampling. Between files the work shares nothing, so `mtx scan` runs one process per *physical* core -- not per logical one: this work is dense numpy over arrays far larger than L3, and a second hyperthread buys about a fifth of a core. Inside one file only the primitives that release the GIL are worth threading, which measurement says is `upfirdn` (3.8x on 4 threads) and little else; `sosfiltfilt` and `welch` hold it and get 1.1-1.2x. So threads go to the true-peak scan and nowhere else, and `-j` is one budget spent on processes first, threads only for the tail. The 16x true-peak pass keeps its exact pruning, but the pruning turns out to clear its bound almost everywhere on a limited master -- 98.5% of the file scanned on the track measured here -- so that pass is now threaded rather than relied upon to skip work. It is split into a pure per-chunk stage and a serial fold in chunk order, so an inter-sample over straddling a chunk boundary is still counted exactly once. 16x drops 10.5 s -> 3.4 s, 4x 2.4 s -> 0.6 s. Output is unchanged, which is the point: all 309,611 leaf values of a real track's `analysis.json` are identical at 1 and 8 threads and against the pre-change baseline, and `digest.md` is byte for byte the same. `mtx selftest` now asserts the threaded scan against the single-threaded one. Also, two things that were quietly wasteful: - `_versions()` spawned `ffmpeg -version` and `ffprobe -version` once per file to ask a question whose answer cannot change mid-run. Now cached per process: 690 ms a file, which a 597-track library was paying seven minutes for. - `write_analysis()` never removed part files from an earlier run, so a folder re-analysed into a smaller document kept orphans that read exactly like current data. They are pruned now, by stem, and the schema version is untouched -- no existing result is invalidated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GAPS.md listed twelve measurements the tool did not make. The signal chain was
covered more thoroughly than any commercial tool I know of; the gaps were all
in the music. This closes them.
Nine new top-level blocks and four new stem blocks, schema 1.2.0, all additive:
harmony chord track, harmonic rhythm, loop length, degree reduction,
cadences, inversions, pedal points, modulation, and a key
cross-check against structure.key
rhythm downbeats and meter, swing, syncopation, grid tightness,
pulse-rate switches, per-stem microtiming
form measured letters over the existing boundaries, merged into
parts, then function labels as a clearly separated inference
delivery AAC 256 / Opus 128 encode and re-measure, small-speaker
band-pass, mono fold, and the 15/30 s and chorus excerpts
lyrics declared > tag > transcript, language-gated statistics, rhyme,
alignment and delivery rate
declared a declared.json sidecar, passed through with source=declared
version version identity and work_key from tags alone
embedding optional, isolated, and never used to derive a measurement
coverage one uniform present/trusted mask over the whole document
stems.masking each stem measured against another stem, per band
stems.melody pyin on the vocal and bass stems
stems.arrangement entry/exit, density, drum and bass character, layers
stems.microtiming is the bass late, per stem, against the beat grid
Two new commands, both deliberately outside analyze:
mtx cohort percentiles and z-scores within a (genre, year) cohort, written
beside a folder of analyses and never into them, because a
per-track measurement must not depend on what else is beside it.
Carries a corpus hygiene report that says when the corpus is too
small or too dominated by one artist to support statistics.
mtx export flat tables at track and track x section level; the per-section
vectors were the most valuable part of the dump and the hardest
to reach.
Checked against published, human-transcribed references (Hooktheory, Ultimate
Guitar, ChordZone, Singing Carrots) for seven well-known tracks, which found
three real defects in the first cut:
* Plain cosine against binary chord-tone masks cannot work: a four-tone
template that contains a triad can never score lower than it, so nearly
every chord came back as a seventh. Mean-removing both template and chroma,
plus a complexity and a quality prior, took chord-time agreement with
published charts from 68% to 82% (86% at root level).
* key_from_chords by Krumhansl-Schmuckler over a chord-tone histogram got 3
of 7 published keys, worse than the mean-chroma estimate it was meant to
second-guess. Rewritten as the key whose scale explains the most chord time
plus tonic evidence: 4 of 7. structure.key still gets 5, and the block now
says so, names the relative-key failure mode and carries a confidence.
* One tracker octave error was setting the whole vocal range: raw extremes
came out 40 to 58 semitones wide. Duration-weighted percentiles over an
outlier-rejected band land within a semitone of two published ranges, and
the outliers are counted rather than hidden.
structure.tempo picks one metrical level and got half the published tempo once
and double once on that set. rhythm.tempo_octave measures the ambiguity, and
reports plainly that it caught none of the three without also firing on the
four correct ones, rather than fitting thresholds to seven tracks.
The full-profile reproducibility promise now covers the ffmpeg encode pass and
is tested. 171 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five things, found by running the tool over a real library rather than over its tests. Separate once per master, not once per copy. The stems cache was keyed on the absolute path and the folder under it named after the file, so a single sitting next to the album it was lifted from paid for demucs twice. The key is now the file's contents. `mtx scan` learned the same idea one level up: files with the same sha256 have one measurement between them, found among the receipts already in the mirror tree, so a copy measured last month counts. Only files that share a size with something are ever hashed. Separate on the GPU when there is one. demucs inherited the scan's one-thread pin, which is right while four separations share the CPU and wrong on a card, where one runs at a time. On CUDA the separations come out of the pool and run up front; measured on a GTX 1650, 20 s a track against 850 s on one core. The segment steps down only on a real out-of-memory failure and what fitted is remembered for the rest of the run. Report the ETA from service time, not wall clock per file. The old estimate charged one track with the whole pool's elapsed time and overstated the first estimate by exactly the worker count: 2h54m against a true 43m. Stop the form labels claiming more than they measured. With a vocals stem a section that sings is never merged with one that does not, whatever the cosine distance says -- without that, an instrumental hook and the final chorus over it become one letter and the track loses a chorus, which is what "As It Was" did. `section` is now the floor of the label ladder rather than a blank, `bridge` is withheld from an unrepeated part louder than the chorus, and the digest prints the unnamed count beside `Form` and `Chorus` so an inference no longer reads like a measurement. Fix the implied tuning reference. `librosa.estimate_tuning` returns fractions of a semitone, and reading it as octaves reported a master built on A=432 as A=350 -- beside a `tuning_cents` computed from the same number, which was right. A whole-semitone offset still reads as zero and shifts the key instead; that limit is now written down where the figure is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on a laptop over a real library rather than over the tests, and two of the four things found contradict what the code assumed. Separate a few tracks at once instead of one. `separate_first` ran the separations strictly serially on the grounds that several on one consumer GPU is an out-of-memory error rather than several times the speed. Sampled over 45 tracks on a GTX 1650 the card is 68 % busy with one stream, not 100 %: a large share of every track goes on decoding the input and writing four uncompressed wavs with the device idle, and utilisation cycles 0 to 76 and back. Overlapping fills those gaps -- 1.36x at two streams, 1.51x at three, where utilisation reaches 94 %. Memory is what actually binds, and it scales exactly linearly at 875 MiB a stream, so `separation_streams` asks the device how much it has and leaves a reserve rather than finding the ceiling through a failure that costs a segment step and the separation that hit it. `--stems-jobs` overrides the arithmetic for a card it reads wrongly. Leave the pool alone: it was already sized right. Twelve identical tracks with the stems pre-cached, so nothing but the DSP is timed, give 2.754 audio-seconds per wall-second at six workers, 2.739 at nine and 2.756 at twelve -- flat across a doubling. Per-track CPU time over the same three points goes 420 s, 548 s, 824 s: every worker past the physical core count takes its share of a fixed throughput by making the others slower. The estimate in `physical_cores` that a second hyperthread buys "perhaps a fifth of a core" is conservative in the right direction; here it buys nothing measurable, and `--jobs` above the core count cannot help. Name the disk the stem cache lives on. Four uncompressed wavs a track is ~165 MB, and the home directory is usually on the smallest disk in the machine -- a library scan fills it and dies halfway through. `MTX_STEMS_CACHE` names a better one, read once at import so a scan and every worker agree. Keep a scan out of its own output. `SKIP_DIRS` did not know `_mtx_out` or `_mtx_stems`, so a scan of a library root with either parked beside it would walk in and measure separated stems as masters. A separated vocal is not a master and analysing one quietly poisons the corpus. PERFORMANCE.md carries the full numbers, the bench they were taken on, the method, and the three things still open -- overlapping the separation and DSP phases, which is now the ceiling at an estimated 19 %, bounding the stem cache, which is a prerequisite for any library-scale run at 165 MB a track never evicted, and the per-file torch reload. It also records a measurement mistake worth not repeating and two Windows counters that lie. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Separation writes four uncompressed wavs a track, about 165 MB, and nothing ever removes them. On the bench library that is not a tuning question, it is a wall: 1274 tracks want 210.2 GB of stem cache against 63.7 GB free, so a single `mtx scan` over the library root cannot complete. It separates its way to a full disk about a third of the way in and dies, and because `separate_first` finishes the whole todo list before the pool measures anything, it dies having written measurements for none of the tracks it had already paid to separate. Two defects combine there and both are still open in `scan.py`: separation is unbounded and up front, so peak cache is the size of the run rather than of any working set, and a track's stems are dead weight the moment its `corpus_row.json` exists but stay on disk regardless. The fix is to overlap the phases, which caps in-flight separations at the stream count, and to evict on measurement. Until that lands these two work around it from outside. `prune_stems.py` deletes cached stems for tracks that already have a measurement; it reads the sha256 out of each `mtx_source.json` receipt, which is the same hash the cache key is the first 24 characters of, so it never touches the audio. It requires a corpus row beside the receipt before removing anything, because a run interrupted between writing the two must not lose stems for a track that has still to be measured. `scan_library.ps1` walks a library one artist at a time and prunes between them, so peak cache becomes the largest single artist -- 137 tracks and 22.6 GB here -- rather than all 1274. Both are resumable: `mtx scan` already skips a track that has a receipt, so a crash, a reboot or a Ctrl-C costs only the artist in flight. The script uses -LiteralPath everywhere on purpose. PowerShell reads `[` and `]` in a -Path as a wildcard character class, and album folders are full of them, so a -Path there matches nothing and skips the artist with no error at all -- which is indistinguishable from data loss until you count rows twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The disk-floor break came after the increment, so a run that stopped before touching anything still reported one artist done. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every artist failed instantly and the whole library scan measured nothing. `mtx` writes its progress to stderr, and under PowerShell 5.1 a native program's stderr arrives as ErrorRecord objects rather than text: with $ErrorActionPreference = "Stop" in force, the first ordinary progress line -- "[mtx] scope: ..." -- was raised as a terminating NativeCommandError and caught by the wrapper as a failure. A healthy scan looked exactly like an immediate crash, once per artist, for all 55 of them. Native calls now go through `Invoke-Native`, which drops the preference to Continue for the duration, flattens each record to a string so it logs and prints as ordinary text, and trusts the exit code alone to say whether the program failed. The try/catch is gone: it was catching PowerShell's own mistranslation, not anything mtx did. Verified end to end against the real library rather than a fixture -- the first artist separates on the card and reaches the pool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mtx scan --stems` separated every track before it measured any of them, so
the cores waited out the whole separation phase and the card waited out the
whole measuring phase. PERFORMANCE.md's Finding 3 named the fix and left it
open. This is it.
`drive()` runs both at once. Separation threads separate and push onto a
queue; a feeder thread hands tracks to the measuring pool as memory frees up;
the pool reports as results land. A track is submitted the moment *its own*
stems exist, so the card is working on track n+1 while the cores measure n.
The feeder is a thread of its own on purpose. The first version had the
separation threads submit their own track, so a stage thread that found no
memory free blocked holding its separation slot -- the card stopped, waiting
on a core, which is the coupling this exists to remove.
Two bounds keep it honest:
* `lookahead_for()` permits bound the tracks separated but not yet
measured. Each is four uncompressed wavs, and separating is about six
times faster than measuring, so unbounded the card would separate the
library into a full disk. A permit is taken before a separation and
returned after the measurement, so the disk high-water mark is a
handful of tracks rather than the library.
* `decoded_bytes()` sizes each track and `drive` admits by bytes, not by
job count. Worker count is the wrong unit: six lanes is right for a
44.1 kHz album and too many for a 192 kHz one, whose tracks carry four
times the samples and whose stems are decoded beside them.
Overcommitting does not degrade, it collapses.
With the phases overlapped the separation default drops to one stream. The
1.51x that three streams were worth belonged to the old design, where the
cores sat through separation; now that it hides under the measuring, a second
stream buys nothing and costs about a lane's worth of memory -- which is what
`memory_budget(streams)` holds back for it.
`--prune-stems` drops a track's stems once its measurement is written, and
only then: a failed track keeps them, because separating it again is minutes
of GPU to save 165 MB of disk. That is what lets a library run in one pass
instead of being cut into batches around the disk.
Also: below the 48 kHz band cap, `band_mid` and `band_side` are `mid` and
`side` recomputed -- the same float64 arithmetic on the same values, verified
bit for bit -- so they now return them. That is a gigabyte a track of a second
copy of the same answer, and it buys back the measuring lane the concurrent
separation costs.
And a note at startup when something else already holds the memory: an
unrelated media-library indexer held 45 GB of commit and a third of the cores
through a whole unattended run, and nothing said so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PERFORMANCE.md gains Finding 4a, which should be read before Finding 4 because it partly invalidates it. Every memory number collected that night was collected on a machine where the Apple Music library agent held 45.4 GB of commit -- leaving 0.8 GB for everything else -- and had burned 27.3 CPU-hours since starting minutes before the scan, which is 2.3 of 6 cores held continuously through an unattended run. Stopping it returned 47.1 GB of commit in one second. So the MemoryErrors were raised against that, not against six workers' own footprint: allocations of 54 MiB were failing. Finding 4 attributes Coldplay's collapse to its 192 kHz masters; the two are confounded, because Coldplay ran last, when the agent was largest. The footprints, the sample-rate distribution and the argument for admitting by bytes all survive. The attribution does not, and it is now marked as not surviving. The method note matters more than the finding: working set is what GetProcessMemoryInfo reports and what Task Manager shows by default, and it is not the quantity MemoryError is raised against. This process held 45 GB of commit behind a 6.9 GB working set -- invisible in the obvious place to look. Also here: where a track's time actually goes (stems 56 %, delivery 22 %, true-peak oversampling 18 %), with the conclusion that there is no redundant work worth cutting, written down so nobody measures it a third time; and the two things the pipeline got wrong before it was right. `tools/scan_library.ps1` loses the per-artist loop it existed for. `mtx scan` now bounds its own stem cache, so the batching, the disk floor and the between-artist pruning are all gone. What is left is a launcher, for the three things PowerShell 5.1 gets wrong by itself: native stderr arriving as ErrorRecords, Tee-Object writing UTF-16 into a log that was otherwise UTF-8, and no timestamped log at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A scan reading a whole music library is exactly what wakes a media indexer, and one of them shared this machine with an unattended run: 45.4 GB of commit and 2.3 of 6 cores, for seven hours, with nothing in the log to say so. The launcher now reports any it finds, with what they are holding, and stops them on -StopIndexers. It also logs free physical and free commit at the start, because commit is the quantity MemoryError is raised against and it is not the one Task Manager shows first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ange Nine tracks, stems cached, agent stopped: 2.0 audio-s per wall-s at six lanes, 0.42 per lane. The same code in one process sustains 0.90, so six workers return 2.8x -- Finding 1 arriving again from a different direction. The band_mid/band_side aliasing is an arithmetic identity rather than an approximation, so it is checked rather than argued: one track measured before the change and re-measured after it with --force, the two analysis.json files compared field by field. One field differs, and it is the timestamp. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The launcher died at the first line of the run with a parameter binding error. -Encoding on Tee-Object arrived in PowerShell 6; on 5.1 the tee writes UTF-16 into a log the rest of the script writes as UTF-8, which is the mixed-encoding log that was unreadable the first time. A StreamWriter takes the encoding, so the tee is written out longhand: flatten each record, Write-Host it, WriteLine it. AutoFlush keeps the log current enough to watch a running scan from another window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scan stalled four minutes in: 0% CPU, 0% GPU, 27 GB resident, no error on screen. Two defects, one of which hid the other. memory_budget subtracted its reserves from total_memory_bytes() -- the RAM the machine has, not the RAM that is free. On this bench those differ by the 8-10 GB the OS and desktop already hold, so a 34 GB machine with 26 GB free was authorised 28.1 GB. available_memory_bytes() was right there in the same file, used only for a warning whose threshold (free < budget * 0.75) was slack enough not to fire. The library's 192 kHz albums decode to 6.6-9.6 GB a track; the gate admitted about 26 GB of audio into 26 GB of memory and Windows killed a worker. A worker also holds 824 MB of interpreter, numpy and scipy before it decodes a sample, and the model counted that as zero -- six lanes start 4.9 GB down. So workers_that_fit walks the count down until it agrees with the budget its own overhead produces, sized on the median track rather than the largest, which would size the pool at one. On this library that is 5 workers; scoped to a 192 kHz album, 2. The second defect is why one kill cost 820 tracks. A killed worker does not fail its own track politely: it breaks the executor, and everything still to come fails on submission with BrokenProcessPool. drive now takes a restart callback, rebuilds once per break rather than once per track taken down (that is what the generation counter is for), requeues those tracks once, and tightens the budget a quarter each time. Verified by re-running the album that died: it now reports "2 process(es), not 6: 6 would want 41 GB for a typical track and there is not that much", and measures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With the budget fixed, the next suspicion was head-of-line blocking in the feeder: it admits in queue order, so a track that does not fit blocks ones behind it that would, and a 192 kHz album sits contiguously in directory order. Simulated over the library's real 1274 (size, duration) pairs, work-conserving best-fit admission is worth 3.8% and a deeper lookahead another 5%. Not worth rewriting the one component here whose failure mode is a deadlock. The informative number is that only 3.09 of 5 lanes are in use. Sweeping the budget instead: 40 GB of RAM is worth 1.37x and it stops paying at 48 GB, where the memory-bandwidth ceiling of Finding 1 binds instead. The one software lever left is that a track holds all four stems decoded at once -- 76% of an ordinary track's footprint -- which simulates at 1.21x if the inter-stem masking can be restructured to stream them. Simulation kept as tools/sim_admission.py so the next person can re-run it rather than re-derive it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stopping a run printed "a worker died; rebuilding the pool". It had not. Ctrl-C on Windows is delivered to the whole console process group, so the workers die a moment before the KeyboardInterrupt reaches the parent, and in that gap their futures come back as BrokenProcessPool -- indistinguishable from a worker the OS killed. The recovery path added this morning then did exactly what it was built to do, during shutdown. drive now installs a SIGINT handler that sets `stopping` immediately, which the rebuild was already guarded on, and restores the previous handler on the way out so a library caller is not left with ours. free_memory.ps1 closes what competes for memory before a scan. This is worth more than housekeeping: the budget is fixed once, at launch, from the memory free at that moment, so an open browser does not slow the run for a minute, it costs a measuring lane for the whole night. Measured here at 2.5 GB reclaimed -- the sixth lane, about +21% over seven hours. It closes media indexers, a named list of large desktop applications, and any other windowed process over -MinimumMB. It never touches services, windowless processes, explorer, a running mtx, or its own parent chain, asks politely before forcing, and refuses to run inside an editor terminal since it closes editors and would kill its own console mid-list. -DryRun lists and stops. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A run this morning went to 1562 s a track against an expected ~550, with the laptop cold, the fan silent and the cores at 0-20%. None of that was the scan: the machine was unplugged. Measured under load at 58% of nominal, about 1.5 GHz against ~3.9 all-core on AC -- roughly 0.4x, which turns a 7-hour run into a 20-hour one on a battery that would not last it. This belongs with the indexer check for the same reason: it is an environmental cause that presents as slow code and that nothing in the scan's own output could reveal. Both are now reported before the first track. The launcher refuses by default and exits 3; -AllowBattery overrides. The % Processor Performance figure is logged on AC too, since it is also how a thermally throttled machine would show itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PERFORMANCE.md claimed per-worker throughput was the same at 44.1 kHz and 192 kHz. That came from one benchmark track and is wrong. Every finished track writes its own wall time in mtx_source.json, and 457 of them say 0.412 / 0.383 / 0.398 / 0.253 audio-s per wall-second at 44.1 / 48 / 96 / 192 kHz. tools/scan_report.py mines them: throughput, mean lanes actually in use, cost by sample rate, and -- given --library -- what is left priced at the rates just measured rather than assumed ones. It exists because a scan grinding through a block of 192 kHz masters is memory-limited to three lanes and leaves the machine cool and silent, which is indistinguishable from a broken one until somebody counts. That is not hypothetical: the remaining 817 tracks are 17% hi-res overall, but the first 30 in scan order are 43% hi-res, so the run opens on the worst content in the library and its own ETA extrapolates from it. Occupancy is integrated over the wall clock rather than counted pairwise; pairwise overlap reported 11 concurrent lanes on a 6-worker pool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things, found in one hour of a user's log. The feeder admitted strictly in queue order, so a track that did not fit blocked every track behind it that would have. A 9.0 GB 192 kHz master sat at the head of the queue for an hour while six already-separated 44.1 kHz tracks of 2.8-4.2 GB waited behind it -- three would have fit -- and three of five workers idled. I found this shape this morning, simulated it against the whole-library average, measured +3.8%, and dismissed it. The average hid it: a library's largest tracks sit together in directory order, so the opening block is the pathological case, not the typical one. `choose()` now takes the largest waiting track that fits beside what is running. Four policies were simulated against the real remaining 815 tracks with per-sample-rate costs measured from 457 receipts: largest-fits wins (+3.1% makespan), smallest-fits is worse (-3.7%), and a deeper lookahead adds nothing. Starvation is bounded twice: a skipped track keeps its lookahead permit so the window fills with skipped tracks, and SKIP_LIMIT pass-overs stop the line until it fits. Three tests, one of which reproduces the stall. Separately, and much larger: the same machine measured 192 kHz tracks at 0.253 audio-s/wall-s at 06:34 and 0.143 at 10:27, with fewer lanes running. Its cores sit at 778 MHz -- 30% of nominal -- under load, on AC, cooler and slower than the same machine on battery an hour earlier. A scan cannot tell a clamped CPU from expensive audio: both are low utilisation and long tracks. tools/bench_cpu.py records the best single-core rate this machine has ever shown and compares against it -- self-calibrating, since a fixed threshold would be wrong everywhere but here. The launcher runs it and refuses below 65%, with -AllowSlowCpu to override. Two seconds against seven hours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ENOENT
A filename ending in an ellipsis (`03. Sometimes....flac`) loses its extension
and leaves a folder name Windows will not keep: `makedirs("03. Sometimes...")`
creates `03. Sometimes` and reports success, so the write into the name that
was asked for fails with ENOENT — inside demucs when it opened `drums.wav`,
and inside mtx when it wrote `analysis.json`, both after expensive work.
One naming rule, `safe_component()`, strips trailing dots and spaces, replaces
characters Windows bars, and escapes reserved names (`NUL`, `COM1`…). Applied
on POSIX too so a mirror tree holds the same folder names whichever machine
wrote it. The stem cache is now told `--filename stems/{stem}.{ext}` instead
of being left to name its folder, removing the failure at the source.
Verified: all three tracks separated and analysed cleanly; next scan will skip
them. Suite: 260 tests pass, 3 near-identical names stay separate.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Three fixes and a Notion loader, all aimed at the same gap: the corpus has 1,321 analyses and had zero online.json files. `_enrich_targets` walked only the immediate children of the path it was given, but `mtx scan` mirrors the library tree, so an analysed track sits three levels down at `<out>/Artist/Album/Track/`. Pointed at a scanned library root, `mtx enrich` reported "no analysis.json" and stopped. That is why the online layer was never run -- not neglect, no route. The walk is now recursive and treats a folder holding an analysis.json as a leaf. `lyrics.py` selected the lyric tag with `if "lyric" in key.lower()`, which matches `composerlyricist`. Apple-style tagging puts that key on most commercial files, so 956 of 1,321 tracks were measuring a songwriter credit as though it were the song's words -- modal length two words. All 77 exported lyrics columns were computed correctly over the wrong input, and each carried `source: "file:tag"`, so nothing downstream could tell. The tag key is now matched exactly against a list recorded in params. tools/notion/ pushes a measured corpus into two databases: Tracks (~150 queryable properties, with the full 2,000-column row, section timeline, chord track and confidence notes in the page body) and Observations (append-only, one row per time-varying figure per lookup). Three things there reach values no flat export produces, because they live inside JSON lists: the AAC and Opus encode renderings, vocal-against- instrumental per section, and every genre vote rather than only the winner. Two decisions worth naming. Traits are tri-state, never boolean -- four_on_the_floor is null on 63 of 72 electronic tracks tested, so a checkbox would render those as "no" and silently drop most of a club-music query. And popularity is an observation, not a property: Deezer rank and Last.fm playcount are current-value endpoints with no history, so a figure not captured is gone, and a stored scalar cannot tell a 59-week slow burn apart from a record that debuted at number one and fell away. The loader lives outside src/mtx/ deliberately. mtx measures; it does not interpret. Its two thresholds are recorded and versioned rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The corpus had 2,091 independent variables and no dependent one. This adds the missing half, outside src/mtx/ where a scoring layer belongs. Raw playcount is not usable as a target. Drake's worst album track outstreams almost any independent hit, so sorting 55 famous artists by plays ranks fame and then catalogue age -- never songwriting or mixing. What this computes instead is each track's log10 playcount as a z-score against the same artist's other tracks, which holds fame, budget, label push, era, producer and mastering engineer roughly constant. The natural experiment was already on disk and needs no chart data: about 1,100 of the 1,321 tracks are album cuts by artists whose singles are here too. On the first 189 enriched, Adele's `Rolling in the Deep` lands at z=+2.06 and the 97.9th percentile of her catalogue while the deep cuts sit mid; within one Ariana Grande catalogue the range runs from 38 plays to millions. Grouping is on the top-level scan folder, not the artist tag. The tag carries features, so "Calvin Harris", "Calvin Harris / Dua Lipa" and "Calvin Harris feat. Rag'n'Bone Man" are three tags and one catalogue -- grouping on it turned 55 artists into 264 and left most with too few tracks to position anything against. schema.py's Artist property had the same bug and gets the same fix, plus an "Artists all" multi-select so a collaborator is still findable. Everything here carries its observation window, and re-running after a later `mtx enrich --refresh` produces a new set rather than a correction: playcount is a current-value figure with no history, and the log is the only thing that makes a trajectory recoverable. Also fixes is_single, which read a release type at a path that does not exist. The real one is online.musicbrainz.release_group.primary_type -- 21 singles and 168 album cuts across the first 189. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A track costs about 12 seconds through `mtx enrich`, and almost all of it is the process sitting still: nine HTTP requests, each carrying a mandated pause plus a round trip. 1,321 tracks is three and a half hours of waiting. The waiting is per host, and there are four of them. tools/enrich_fast.py runs a thread pool over folders so that while one track holds the MusicBrainz clock, another is talking to Deezer, a third to Last.fm, a fourth parsing JSON off disk. Measured on 100 cold tracks: 16/min against 5/min, so 3.75s a track against 12s. This is only safe because the pacing moved. `Client._wait` kept its last-call clock per instance, so N clients meant N requests a second to MusicBrainz -- which answers 503 and is entitled to. The clock is now module-level and lock-guarded, held across the whole sleep so two threads cannot both read it, both decide they may go, and both fire together. The promise is to the host, not to a client object, and it now holds however many clients exist. No 503s and no retries in the benchmark. MusicBrainz remains the floor at roughly three uncached requests a track, so past -j 8 there is nothing left to overlap and the script says so. Also: the Notion loader now creates "Corpus" rather than "mtx Tracks", and --archive-db retires a superseded database once the push has succeeded -- never before, since an archived database is recoverable but a gap where the data used to be is not. Verified live: Corpus and Corpus Observations created under The Listening Room, three pages pushed with 143 properties and 138 body blocks each, 15 observation rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two tests for the pacing change, which is concurrency-critical and was covered by nothing: six threads with six separate Clients must still leave 1.1s between MusicBrainz requests, and a Deezer call must not wait behind a MusicBrainz one -- the overlap is the entire reason the thread pool is faster. Also trims the observation log to track-level figures. `deezer_album_fans` belongs to the album and `lastfm_artist_listeners` to the artist, so logging either per track writes the same number 137 times for Drake, adds no information, and makes the log 1.7x larger. Both stay on the track row as `Latest ...` caches. Artist momentum over time, if it turns out to matter, wants a per-artist log rather than a column smeared across a catalogue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
outcome.py has to sit between enrich and push: it reads online.json and writes outcome.json, and push.py reads both. Run it out of order and the within-artist columns are quietly empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The push ran at 6.9 pages/min, which put 1,321 tracks at three and a quarter hours. It was latency-bound, not throttle-bound: a page create carrying 100 blocks takes over a second server-side while the throttle only asks for 0.36s, so serially most of the allowance went unused. A pool of six gets 22.3/min, measured against the live database. Three things had to be right for that to be safe. The client's throttle now holds its clock under a lock for the whole sleep. Notion's limit is per integration, so without it two threads read the same timestamp, both decide they may go, and the rate doubles into a 429. `reconcile()` rebuilds the pushed-set from Notion by sha256 at startup. The state file is an optimisation, not the record -- it is written every 25 tracks, so an interruption leaves it behind the truth, and a resumed run would have created duplicate pages for everything it had forgotten. A duplicate is far worse to clean up than a re-push is to wait for. This is how the 16 pages from the earlier serial attempt were absorbed rather than doubled. And the skip decision reads `mtx_source.json` (895 bytes) rather than the analysis. My first version of the pool built the whole work list up front, which loaded all 1,321 documents to look at one field: nine minutes before the first page was written, and every parsed document held in memory at once. The worker loads its own now, and resident memory stays at 240 MB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two bugs the 1,321-page run found, both in the failure path. `client.request` caught URLError but not TimeoutError. A socket read timeout is not a URLError, so it escaped the retry loop and failed a page outright -- one page in 1,305 did exactly that. The write had in fact landed (the page has all 142 blocks and all three observation rows), so the only real damage was the report. That report then blocked the archive, because the gate asked for `pushed and not failed`. A retry run that finds everything already present pushes nothing, so the very run that resolves the last failure could never trigger the archive. The gate now asks whether the corpus is fully present and nothing failed this run, which is the condition it was always meant to express. Final state: Corpus 1,321 rows, Corpus Observations 3,785 rows, Masters archived. Genre, ISRC, LUFS-I, Opus128 TP delta and Traits populated on all 1,321; within-artist outcome on 1,221 (the rest are artists with fewer than five tracks carrying a playcount, reported with a reason). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Before `observed_at` came from the provider's own `fetched_utc`, a cache-warm run stamped yesterday's numbers with today's date. The log then held what looks like a second reading and is not one: same track, same metric, same value, a day later, no request made. 2,834 of the 3,957 rows written today were that. Deleting every repeat would be wrong -- "still 24,652,445 plays a week later" is a real observation, and an append-only log exists to hold exactly that. So a row is archived only when the value is identical, the earlier row is within two days, and it is the later row that goes, leaving the reading on the day it was actually taken. Anything whose value moved, or that has no earlier counterpart, stays -- which is what keeps the tracks that only got a play count once the Last.fm matcher was fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… written A push rewrites a track page, so a renamed artist reaches the Corpus table on the next run. The Observations log is append-only, so a row written last week keeps last week's spelling and the two tables stop agreeing -- which is not cosmetic: a join on that column silently drops every row spelt the old way. 287 rows across six artists were still filed under the library folder name. `audit.py` calls this `notion.artist_drift`; this repairs it, touching only the select value and leaving the reading, its date, its value and the sha256 that actually identifies the track exactly as they were. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The whisper fallback wrapped only `WhisperModel(...)`. But `transcribe()` returns a lazy generator: the decoding, and any failure it hits, happens as the caller iterates -- long after the constructor returned looking healthy. So a 4 GB card loaded the model, reported itself fine, and ran out of memory part way through a long track with no second attempt, while a CPU that would have transcribed it in four minutes sat idle. 16 of the first 486 tracks of the corpus run, 3.3%, concentrated on the long ones: album cuts rather than singles, which is a slant in the lyric data and not just a hole in it. Each one came back `available: false`, so a re-run picks them up now that the fallback covers decoding too. Load and decode are now one attempt per device, and the failed model is dropped before the retry -- it is holding the memory the retry exists because of. Five tests, with a stub whose generator raises on the second pull, because a test against the constructor would miss the whole bug. Also adds tools/watch.ps1, which is what found this. The long stages print one line per track: a fine record and a poor progress report, with nothing saying how far, how fast, or whether the job is still moving. A stalled run and a slow run look identical in a tail; this says which it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by watching a finished run rather than trusting its summary.
**An out-of-memory is not a broken card.** The device fallback fell straight
from the GPU to the CPU, so 78 tracks that ran out of 4 GB part way through
decoding were headed for a multi-hour CPU pass. `int8_float16` holds the same
weights in half the memory: the retry stays on the card and costs seconds.
Every swept track so far has come back on that rung.
**95 tracks were re-transcribed on every run, forever.** The resume check
asked whether the transcript had *won*, not whether it existed. 167 tracks
carry a real lyric sheet in their file tags, which correctly outranks a
transcription -- so `lyrics.source` stays `file:tag`, they looked untouched
on every pass, and each one spent thirty seconds re-deriving a transcript
identical to the one already on disk. ~48 min of GPU per run, growing with
every tagged FLAC bought. The log reported each as a fresh `ok`.
**`--deep` had never once completed.** It read `lyrics.statistics.lines` as
`{"count": n}`; it is an `int`. Every run died on the first track. No test
caught it because every fixture writes an analysis with no lyrics, so the
deep tests short-circuited on `available` before reaching the line that
crashes -- a crash in a mode nobody runs looks exactly like a mode nobody
runs. Its first completed run found 67 tracks whose "lyric" is a songwriter
credit: 13 characters, one line, `source: file:tag`.
**A failed transcription wrote nothing**, leaving an analysis byte-identical
to one nobody had asked to transcribe. So 78 broken tracks sat behind a clean
audit filed as "no lyric from any source" -- a finding about the music, when
the finding was about the job. Failures now record the reason and the devices
tried, without counting as done, and `lyrics.transcript_failed` tells the two
apart at warn level.
Also guards both readers that took the split index for the whole document,
with a test that performs the old write and asserts the manifest disappearing.
21 new tests; 334 pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A sung line is a phrase. Across 1,093 transcribed tracks the median is 7.9 words per line, p95 is 11.4, p99 is 16.0 -- then a tail out to 86.5, where whisper returned four segments for a whole song instead of one per phrase. The words on those tracks are still the words; what is unusable is every line-based measurement over them, which is quietly measuring paragraphs: rhyme scheme, syllables per line, repeated-line share, readability. Same shape as the one-line transcript that reported a readability of -274, except this one reports a number that looks reasonable. `lyrics.line_structure` catches the 6 tracks over 20 words a line, at info, because nothing else about them is wrong. The check this deliberately is not: repetition. The obvious reading of a 225-word transcript with 19 distinct words in it is a hallucinated loop, and measuring before choosing a threshold is what stopped it going in. The most repetitive transcript in this corpus is Daft Punk's "Around the World" -- a distinct-word ratio of 0.017, a compression ratio of 37 -- and it is completely correct: the song repeats one phrase 144 times. So do "Fresh" at 0.035, "Lose Yourself to Dance" at 0.163 and "How Deep Is Your Love" at 0.185. A repetition threshold would have flagged the most accurate transcriptions in the corpus, and flagged them for being exactly what a hit chorus is -- against a corpus whose entire purpose is measuring what popular records actually do. A test asserts "Around the World" is not flagged, so the check cannot later be "improved" into the one that was nearly written. 337 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sweep died with the session that launched it, fifteen hours before
anyone looked. The watch reported it as "not running" and then, two lines
later, "left 14m 43s, done about 15:04 Sat" -- an ETA computed from a rate
belonging to a process that no longer existed. The reassuring number is the
one that gets read.
It now says what actually happened: stopped at 1030 of 1321, 291 still to do,
re-run the same command and finished tracks are skipped.
Two neighbours of the same bug:
* the stale-log warning quoted its own threshold rather than the measured
silence -- "nothing for ten minutes" under a gap of fifteen hours, which
makes a dead job look like a slow one;
* a resumed run appended to the same log starts counting from one again,
and the earlier pass's work was charged to the new process's elapsed
time: 36 tracks over 32 seconds, an ETA of nine seconds for a job with
291 folders still to walk. A drop in the index is now read as a restart,
and the previous pass is reported separately rather than summed into it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…inished" `set VAR=value && cmd` in cmd.exe puts the space before the `&&` into the value. A hand-written .env line does the same at the end of a line. The path then reads as correct everywhere it is printed -- the space sits inside the closing quote -- and ctranslate2 answers `Unable to open file 'model.bin'`, which looks like a corrupt download rather than a typo. 43 tracks failed on all three devices, on a run that was otherwise failing none. `tools/pipeline.py` already stripped every value it loads from mtx.env, so the normal path was never exposed; `MTX_WHISPER_MODEL` was read raw in the two places that read it directly. Both strip now, with a test that passes a deliberately padded path. Worth recording that the failure-recording change from earlier today paid for itself on its first day: all 43 wrote a reason and stayed retryable. Under the previous code they would have been byte-identical to tracks nobody had asked to transcribe, and would have sat behind a clean audit exactly as the 78 out-of-memory ones did. watch.ps1: a `done:` line followed by fresh progress lines is a new pass, not a finished job. Without that the watch reported "finished", with the previous run's summary, while the GPU sat at 92% working on the next one. A stale completion notice is worse than none -- it is the answer someone stops reading at. 338 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`0 pushed, 1321 already present`, exit 0, `done: every stage clean` -- and 25 empty columns in Notion. The skip check asked whether a sha had ever been sent. It had. So a day's worth of amendments -- 1,306 transcripts, 66 repaired lyrics, a fresh set of cohort percentiles -- was invisible to it, and the pipeline reported success for publishing nothing. The state now carries a stamp per track: `stat` over the four files that feed its page, plus a content hash of the three artefacts every page reads. Stat rather than parse, because reading 1,321 analyses to decide what to send would cost nine minutes before the first write -- which is why the check looked only at the state file to begin with. Content rather than mtime for the shared artefacts, because `cohort.json` is rewritten on every run whether or not a number moved, and an mtime stamp would force a full 1,321-page re-push daily for nothing. That is its own wrong answer, and the kind that teaches people to stop reading the output. Two more, both found by working through the consequences: **A duplicate-page bug this change introduced.** `push_track` creates when handed no page id, and the call site passed one only under `--force` -- safe only while `todo` could not hold a sha that was already pushed. The stamp makes that false. Every amended analysis would have published a second page beside the original. Caught in review; the call now always passes the known page id, and two tests pin create against update. **409 `conflict_error` is retried.** It is the one 4xx that is not a bad request: Notion refusing a write because something else touched the same page first, and the identical payload succeeds next attempt. Measured -- 13 pages lost at eight workers, none at three, same bodies. A lost page keeps yesterday's numbers while the run reports a failure count small enough to read as noise. Jittered backoff, because the writers that collided are the ones about to retry. The push had no tests at all and is the only tool here that writes to live tables. It has 14 now. 352 pass. Verified against the real corpus: 13 pushed, 1308 already present, 0 failed, and `reconciled 1321 existing page(s)` -- no duplicates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`database_schema()` describes a select as `{"options": []}`, which is what
creating one needs. It was sent to the live database on every run, and
Notion reads an explicit empty list as *these are the options now*: it
deletes the rest, and a deleted option is blanked on every page holding it.
So each push wiped all 22 select columns in the table. The comment above the
call said "nothing is removed here (that is what --prune-options is for)".
It hid for as long as it did because the same run then rewrote all 1,321
pages and re-created every option on the way through -- destroyed and
restored inside one run, net zero, invisible to anything that looked after
the fact. Recording a content stamp, so a run writes only what changed, took
the restore away: 13 pages written, 9 artists re-created, 46 options gone and
1,005 rows left with an empty Artist. Also Cohort on 1,042 rows, Key on 805,
Genre on 347.
The schema sync now sends only properties the live database does not have.
It can still add a column -- which is what it was for, since a page cannot be
written to a column the database has not been told about -- and it cannot
touch an existing one. Verified against the real database: a run that pushed
zero pages left every option intact, where the old code would have removed
them all.
Repaired with a forced push: 1,321 pages, 0 failed. Artist 868 empty -> 0,
Cohort 1,042 -> 0, Key 805 -> 0, Genre 347 -> 0. What is still empty is
honestly empty: Certification and Delivery have no source yet, Lyric language
is the 45 instrumentals, Artist MBIDs the 5 artists MusicBrainz has never
heard of.
Four tests, including one asserting that an unchanged schema produces no
write at all -- the only way to be sure nothing was touched. 358 pass.
Found by `notion.artist_drift`, which exists because this class of thing
keeps happening, and which is the reason the audit reads the live tables
instead of trusting the push's own report of itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
**Discogs asked for a conjunction its index cannot answer.** Measured live, for Adele's `He Won't Go` from `21`: `artist+track+release_title` returns 0, `artist+track` returns 2, `artist+release_title` returns 10. The old query sent the first whenever the file had an album tag -- 425 of 1,321 tracks -- and every one recorded `no results`, which reads as *Discogs does not have this record* and meant the opposite. The album query leads now, because a label and a catalogue number are properties of a release rather than of a track. 425 missing -> 61. And the catalogue number was never extracted at all: it lives on the label entry, so "no release" and "a release whose catalogue number we never read" were indistinguishable. 0 -> 1,260 tracks. **A rejected credit left its MBID behind.** `resolve_one` takes the most common credit, then decides whether it is this folder's artist. When it decides no it keeps the folder name -- and was keeping the credit's MBID with it. The `Alan Walker` folder held Alan Walker's name and K-391's id: a row that is populated, correctly typed, internally consistent to every reader, and about two different people. **And a folder no recording matched had no identity at all.** Every MBID here otherwise comes from a recording match, so four Vietnamese artists -- one track each, each `no candidate recording` against an ISRC MusicBrainz does not carry -- stayed bare folder names. MusicBrainz knows all four at score 100; nobody had asked it directly. 5 unresolved -> 0. **`Delivery` read a key that has never existed.** The value is under `inference`, because it is one. Empty on all 1,321 rows since the column was added, and empty reads as "no vocal" rather than "wrong key". `Delivery confidence` and `Stable pitch share` come with it, and `notion.dead_column` now reports any column empty on every row -- which is how this should have been found. **Eight recordings were voting twice.** `outcome.py` marked them; `mtx cohort` never read the mark, so every percentile was computed over a population that double-counted a single and its album cut. Same tie-break as outcome.py deliberately: two tools disagreeing about which copy is real is worse than either rule being wrong. **Chart outcomes had nowhere to go.** `schema.py` read `declared.outcome.billboard_peak`; `declare.py` never offered the key, so the column could not be filled by anyone following the tool. Adds the section and `tools/charts.py`, which loads a CSV over the whole corpus -- matching on sha256, ISRC, or artist+title with a prefix fallback, and refusing rather than guessing when two records match. **A play count nobody could find.** `My Everything (Debut Single 2015)`: the annotation is not packaging by any word list and not part of the name, so no careful cleaner removes it. `bare_title` is the blunt last attempt, after the careful ones miss. 8,259 plays, and a track that had no outcome variable now has one. Also installs `transformers` rather than `laion_clap` for embeddings, which pip would resolve by downgrading numpy 2.5 to 1.26, librosa 1.0 to 0.11 and scipy 1.18 to 1.17 -- silently changing what `mtx analyze` measures across a corpus built over months. Verified unchanged after install. 394 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… genres
**`_esc` deleted the Lucene syntax instead of escaping it.** Every value it
touches is interpolated inside a quoted phrase, and inside a phrase Lucene
treats its metacharacters as literal text -- so nothing needed removing.
Measured against the live index:
recording:"&burn" AND artist:"Billie Eilish" -> score 100
recording:"burn" AND artist:"Billie Eilish" -> nothing
Billie Eilish's `&burn` came back `no candidate recording` -- no credits, no
genre vote, no release date, no tempo cross-check -- from a search that had
removed the `&`. `Don't Start Now` against `Dont Start Now` is the same
story and the same empty result; the apostrophe survived only because it
happened not to be on the strip list, and every other punctuation mark was,
for no better reason. Now score 1.0 with both artists.
**Discogs browse buckets were being filed as genres.** Discogs sorts every
release into fifteen top-level categories, and several are lists wearing one
label: `Folk, World, & Country` is three genres stapled together for a shop's
menu, and no record is in it. Notion rejects a comma in a select option, so
the push quietly swapped it for a semicolon -- which kept the table working
and left `folk; world; & country` in the filter menu, matching 38 tracks and
describing none of them, while those tracks cast no vote for `folk` or
`country`, which is what they actually are.
Split into parts now, with an exception list rather than a rule, because
there is no rule: `Funk / Soul` is two genres and `Drum & Bass` is one, and
only knowing the music tells them apart.
**`release.date_conflict` was right 6% of the time.** 48 of its 51 rows were
a 2015 soundtrack carrying a 2003 recording: package date agreeing with the
song's own first release, only the file tag differing, which is the
resolution working exactly as intended. A warning that is usually wrong is
one people learn to scroll past, which costs more than the check earns. It
flags the reissue it exists for -- a package date agreeing with neither the
tag nor the song -- and nothing else.
Documents the amendment tools and what is honestly still open, and records
the pattern in memory: every defect in this corpus has been a key nothing
writes or a query nothing can answer. Two of them had passing tests, written
against the same wrong assumption as the code.
407 pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…irms `harmony.loop.loop` is null on every track in this corpus. Not because nothing loops -- `Around the World` is four bars repeated for seven minutes, and its 4-bar candidate does score highest -- but because confirmation needs 0.75 on an exact per-bar chord-set match, and the chord detector emits multi-chord bars (`Gm|D#sus2|Fsus4`) that rarely match exactly. The best score anywhere in the corpus is 0.15. So the threshold is unreachable and the column reads as "this music does not repeat" about records built entirely on repetition. Lowering it until it fires would be choosing a number to produce a result. `Loop candidate bars` and `Loop candidate match` report what was measured instead: the period that repeats most, and the share of bars at which it does. Naming it a candidate is the difference between a measurement and a claim, and the corpus already has a working answer next to it -- `Loopability` reads 0.93 for `Around the World` against 0.09 for `Rolling in the Deep`. Also fixes a separator rule I introduced two commits ago. A bucket is written `Folk, World, & Country`, with spaces; a genre is written `R&B`, without. Splitting on a bare ampersand turned `Contemporary R&B` into `contemporary r` on 283 tracks -- a category that reads like a real one and is a fragment of a word. Caught by re-reading the corpus after the change rather than by the tests, which is the only reason it did not ship. 412 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every slashed genre name in this corpus is two genres joined, not one genre whose name contains a slash: `funk/soul` on 415 tracks, `films/games` on 52, `rnb/swing` on 51, `techno/house` on 5. So 415 records were voting for a category no record is in, and casting no vote for `funk` or `soul`, which is what they are. `normalise` has already tightened ` / ` to `/` by the time a label reaches the splitter, so the rule needs no surrounding whitespace -- and `_INDIVISIBLE` is where a genuine slashed genre would go if one turns up. Measured after: 0 commas and 0 slashes left in 429 distinct genre names. `funk` 415 -> 506, `soul` -> 488. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_votes` normalises the whole label before `collect` sees it, so a label that gets split afterwards produces parts that have never been normalised at all. `electro pop/electro rock` split into `electro pop`, which ALIAS has mapped to `electropop` for as long as the table has existed and never got the chance to. Both spellings then sat in the vocabulary -- and Notion stores the first casing it sees and folds the rest onto it, so the column would have displayed a name neither of them agrees with. `vocab.case_collision` caught it, which is the check working: this was introduced by the bucket split three commits ago and found by re-running the audit rather than by any test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A mean-pooled transformer embedding is dominated by a component every input
shares -- "this is music" -- and on raw vectors that component is most of the
angle. Measured over these 1,321 tracks:
raw min 0.535 median 0.909 p95 0.949 sd 0.042
mean-centred min -0.588 median -0.010 p95 0.306 sd 0.176
So everything was similar to everything, the top five were whichever tracks
landed a thousandth higher, and `How Deep Is Your Love` came back with Taylor
Swift and Queen. A cosine of 0.96 looks like a strong match, which is why
this would have passed every check in the file.
Subtracting the corpus mean removes what the corpus has in common and leaves
what distinguishes a record from it -- four times the spread, and the question
changes from "is this music" to "how does this differ from the average record
here", which is the one a neighbour list exists to answer.
The validity check is `Get Lucky`: the radio edit and the album cut, two
different masters of one recording, find each other at 0.980 against a corpus
median of -0.010. `Around the World` returns `Rollin' & Scratchin'` and
`Daftendirekt`, both off Homework.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A 5-second window on a 30-second hop samples 40 seconds of a four-minute record: 17% of it. That is enough to recognise a record as itself -- both masters of `Get Lucky` matched at 0.98 -- and thin enough that a track whose sampled seconds are unrepresentative lands beside the wrong neighbours. At a 15-second hop, a third of each track is looked at. The same self-match is now 0.993, `Around the World` returns two more tracks from Homework, and the Red Hot Chili Peppers track of the same name returns four other cuts from Californication. 58 minutes of GPU time for the corpus, unattended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… it is Two defects in the same five rows. `mark_duplicates` already works out that eight recordings are on disk twice, and the percentiles honour it. The cosine never heard, so `How Deep Is Your Love` spent two of its five slots on the same Queen master. A five-slot list where two slots are one record is a four-slot list, and nothing said so. Only a primary may now be recommended, and no row recommends its own recording -- a duplicate still gets a list, for the same reason it still gets a percentile. The second is quieter. Every list has five names whatever the numbers beside them, so `Get Lucky`, which finds its own radio edit at 0.993, and `How Deep Is Your Love`, whose best match anywhere in 1,321 tracks is 0.382, present identically. Across the corpus the strongest match per track runs 0.35 to 1.00 with a median of 0.579, which makes 0.382 the 1st percentile: the measurement saying there is nothing here to compare this record against. That is a useful answer and the list was not giving it, so the strength and its percentile now travel with the list, and Notion carries both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mtx.env` lives with the music and never enters git, and only `pipeline.py` read it. A tool run on its own got whatever happened to be in the shell, which fails differently depending on the tool. `audit.py --notion` stops with `no Notion token` and names what is missing. Loud, harmless, and how I found this. `transcribe.py` reads `MTX_WHISPER_MODEL`, finds nothing, falls back to `base`, transcribes the corpus with the small model and reports success. Nothing on disk records which model wrote the words. That is the corpus's usual defect shape -- a run that completes, reports cleanly, and produces worse data than the one before it -- and it is why this is worth a file. The loader now lives in `tools/env.py` and every tool that reads a key calls it with the root it was already given. `pipeline.py`'s private copy is gone, because two copies drift and the one that drifts is the one nobody runs. It still returns names and never values: a log line that echoes a token has published it to every terminal scrollback on the machine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Mixing engineer` and `Mastering engineer` were empty on all 1,321 rows. Not because the credits are missing -- `online.credits` holds `mixing engineer` on 1,059 tracks and `mastering engineer` on 499 -- but because the schema asked for `mixer` and `mastering`, which no provider emits. Two columns reading as "nobody is credited with mixing these records", about records that all name a mixing engineer. `credit()` now takes several spellings, matches them case-insensitively and de-duplicates names, since one person credited by both MusicBrainz and Discogs is one person. `Recording engineer` is wired up too: 534 tracks carry it and nothing was reading it. The loop docstring claimed `harmony.loop.loop` is null on every track and that the best candidate anywhere scores 0.15. Both were wrong: it confirms on 15 tracks -- four from Homework, three from Happier Than Ever, Thunder, Marry You -- and the best candidate reaches 1.000 with a median of 0.226. The columns were right; the reasoning written next to them was not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`notion.dead_column` is the general guard against this corpus's worst defect: a column that is typed, named, present on every row and reading a key nothing writes. `Delivery` sat blank on 1,321 rows that way. Its counting loop skipped checkboxes, reasoning correctly that `false` is a value rather than a blank -- and skipping the count left them at zero, which is exactly what an empty column looks like. So every checkbox column was reported dead on every run: sixteen findings, five of them real. A list that is two-thirds noise is a list nobody reads, which is worse than not checking. Checkboxes now count as filled always, and their ticks are counted separately so `notion.constant_checkbox` can report what `dead_column` structurally cannot see -- a box false on every row, which is either a trait no record here has or a key that reads nothing, and those look identical from the table. The counting moved into `column_coverage` so it can be tested without a live database. The bug was invisible from outside the function. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four tracks came back from a re-scan saying `not requested; pass --transcribe` after a run that had requested it and refused -- correctly, because the separated vocal sat 29 to 45 LU below the mix. Queen's `God Save the Queen` is an instrumental arrangement and the others are score cues. None of them will ever have words. The skip was right and writing nothing about it was not: an analysis nobody asked to transcribe and one that was measured and declined are the same bytes, and the difference is "buy a lyric sheet for this track" versus "there is no vocal here to transcribe". The reason and the measured level are now on the row. Recording is idempotent, because the Notion push decides what to send by folder stamp: rewriting an unchanged finding nightly would re-send every instrumental in the corpus for the sake of a timestamp nobody reads. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
67 of them, in the filter menus of a table whose whole purpose is to be filtered. `techno/house`, `rnb/swing` and `films/games` are Discogs browse buckets that were filed as genres until they were split into their parts; `sweetener`, `thank u` and `heading for the door` are album and song titles that arrived as Last.fm user tags. Every one of them offers a category that returns nothing, which is an empty column wearing different clothes: it looks like an answer until you pick it. Sending a select property's options is a full replacement, not a patch, and that is how 1,005 rows went blank once before. So this reads every page first, keeps every option any row holds, reports before it acts, and stops if the query comes back empty -- an empty read and "no option is used" are indistinguishable, and one of them means deleting the whole vocabulary. Notion also rejects an update carrying more than 100 options, so on `Cohort genres` (161) and `Genres all` (458) no single option can be removed at all. Those are dropped and rebuilt by a forced push instead, which is safe because nothing in them is typed by hand: every value is derived from the analyses on disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The loop section said the threshold never fires and that the best candidate anywhere scores 0.15. Measured over the corpus it confirms on 15 tracks and the best candidate reaches 1.000, median 0.226. A document that explains a design decision with a wrong number argues for the wrong thing. Also records that every tool now finds `mtx.env` itself, and the one gap that has no repair: a Notion option list over 100 entries cannot have a single option removed, only rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Loop bars`, `Syncopation per bar` and `Track no` were empty on all 1,321
rows. Every one of them resolved to a real value that the column could not
hold, so the push dropped it and wrote the row anyway:
* `harmony.loop.loop` is `{"bars": 1, "match_fraction": 1.0}`, a dict, and
it confirms on 15 tracks that never reached the table. Both halves are
now columns.
* `rhythm.syncopation.per_bar` is the whole per-bar series, a list; the
scalar the heading promises is `mean_per_bar`.
* `tags.named.tracknumber` is the string `"2"`, and often `"2/12"`.
An empty number column reads as "not measured". All three were measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`notion.dead_column` can only see an empty column from the live table, only after a full push, and it cannot tell a wrong key from data that does not exist yet. Those need different repairs. `schema.dead_path` and `schema.wrong_type` resolve all 200-odd columns against the corpus on disk and separate them: nothing at the end of the path, versus a value the column cannot hold. The three type mismatches this run found had been invisible since the columns were added. The first version of the check loaded each folder without the outcome, identity and cohort sidecars the push mounts, so every column fed by them resolved to nothing: 42 dead paths reported, 3 real. A check reproducing the exact defect it was written to catch. Also stops instrumentals being reported as failed transcriptions. Recording why a track was declined -- the vocal sits 25 to 57 LU below the mix -- put fourteen score cues and album interludes into `lyrics.transcript_failed`, whose fix note says to re-run the job. Re-running declines again, correctly. They now have their own info check, keyed on a flag rather than on the wording of a sentence, and `_record_attempt` treats that flag as part of what "unchanged" means -- comparing only the reason meant a note written before the flag existed could never acquire it, and the sole route to adding it was `--force`, which re-transcribes 1,321 tracks to repair fourteen rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The probe that missed three empty columns asked whether the source resolved to something non-null. A dict is non-null. So `harmony.loop.loop`, `syncopation.per_bar` and a string tracknumber all passed a check written to catch exactly the defect they were. These run the real converter instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The push decides what to send by hashing each folder and the shared artefacts every page reads. The property spec is a shared input too, and it was not in the hash -- so repointing a column moved nothing on disk and nothing in any stamp. `Loop bars` read `harmony.loop.loop`, a dict, and sat empty on all 1,321 rows. Pointing it at the number inside changed no analysis, no cohort and no outcome, so a routine `push.py` would have reported `0 pushed, 1321 already present`, exited 0, and left the column empty for good. The same silence would hide any repointed path, which is the repair for the most common defect in this corpus. Hashed as a file rather than by walking the property list, because the bug that made this necessary lived inside a reader's body: `credit()` kept its name and its wiring while asking for a role no provider emits. A comment change now re-pushes the corpus for nothing -- ten minutes, rarely. A missed one publishes an empty column indefinitely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five things this backfill found, all of which reported success while losing
data.
`pipeline.py` built its scan as `mtx scan --stems` and never passed
`--prune-stems`. Four uncompressed wavs a track, ~165 MB, nothing deleting
them: E: reached 0 bytes about 100 tracks into a 414-track run, demucs died
with `[Errno 28] No space left on device`, and every track after it was
measured *without stems* -- which does not fail, it empties every mix and
per-stem column. `scan_library.ps1` had said so in prose the whole time.
`metrics/stems.py` measured each separated stem with the full loudness
battery, including the 16x oversampled true peak and the intersample-over
counts. A stem is never delivered and no column reads either number; on a
55.4 s track the four stems spent 11.1 s of an 81.8 s analysis on it. It now
asks for the quick path, which still computes the gated integrated loudness
`level_vs_mix` needs. Worth ~4% of wall time, not the 13.6% of CPU it
removes -- the scan is bandwidth-bound at four lanes.
`stems.stems.<name>` has carried a loudness, dynamics, spectrum and stereo
pass per source since stems existed, measured on every track and read by
nothing: the mix group lifts only `stems.masking`. Twenty properties now
surface it -- level against the mix, crest, tilt, side minus mid and LRA for
each of vocals, drums, bass and other. It cost no scan time and backfilled
the whole corpus, because the numbers were already on disk. Each scalar is
named explicitly: `dynamics.crest` and `spectrum.tilt` are dicts, and a dict
in a number column is dropped in silence. No per-stem true peak, since it is
no longer measured and would be full before this commit and empty after.
Two audit checks errored on conditions that were not the harm they name, and
between them held back a 1,889-row publish. `audio.near_silent` failed a
10.9 s interlude at -32.1 LUFS -- quiet by design, measured correctly -- so
severity now splits on the duration the corpus already calls "not a song",
never on the word "Interlude" in a title. `release.bootleg` failed seven
tracks whose dates came from iTunes and the file tag and were right, because
`_song_first_release` already refuses to date a song from a bootleg; it now
errors only when the bootleg actually supplied the published date. Both keep
warning, because the recording picker choosing a bootleg-only recording is a
real defect and still open.
`push.py --dry-run` wrote the state file. `Notion.request` answers every
call with `{"id": "dry-run"}`, that id was stored as the page a track had
become, and the next live push issued `PATCH /pages/dry-run`, took a 400 and
dropped the track while 1,887 others published and the run exited 0. A
two-row dry run taken to check the schema edit above cost exactly two rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s path `online/http.py` names its cache directory after the User-Agent, and MusicBrainz asks that a User-Agent identify the caller with a contact address. An enrichment run started with a relative cache path therefore writes `mtx/<version> ( <contact> )/` beside the source rather than into `.mtx_cache/`, which is ignored -- and thirty of those response files were committed in d23b0f0, publishing a real email address in the path of a public repository. The address is in the directory name only; no file content carries it. Untracked here and ignored going forward. The pattern is rooted, because the package itself lives at `src/mtx/`. This removes them from the tip, not from history: the path still appears in d23b0f0 and everything after it, so it stays reachable on the remote until the history is rewritten. That is a force-push and a decision for the repo owner, not a cleanup to slip into a commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
projectoverseer
force-pushed
the
close-the-gaps
branch
from
September 9, 2026 05:08
5e9bae8 to
a6a8ba7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the checking step this corpus did not have, and fixes what it found.
The failure mode
Every defect below was live in published data and looked fine. None raised an
exception. None scored badly. Every pipeline run reported success.
OLIVIAokin the log, each time0 pushed, exit 0,every stage cleanDeliveryreading a key that has never existedno results, which reads as "not in Discogs"Alan Walkerfolder holding K-391's MBIDWhat is here
tools/audit.py— 28 checks over the corpus and the live tables. Agate, not a report: it exits non-zero and the pipeline stops before Notion.
It writes nothing, because an audit that repairs things cannot be trusted to
report honestly on the next run.
tools/pipeline.py— the whole daily workflow in one command.tools/watch.ps1,tools/transcribe.py,tools/embed.py,tools/charts.py,tools/identity.py.the transcription backend.
Measured
One that did not go in
The obvious check for a hallucinated, endlessly repeating transcript is a
repetition threshold. Measured against the corpus first: the most repetitive
transcript here is Daft Punk's Around the World — a distinct-word ratio of
0.017 — and it is completely correct, because the song repeats one phrase 144
times. That check would have flagged the most accurate transcriptions in the
corpus, for being what a hit chorus is.
lyrics.line_structuremeasuressegmentation instead, and a test asserts Around the World stays unflagged.
🤖 Generated with Claude Code