Fix stems endpoint with null original filenames by raymondjacobson · Pull Request #973 · AudiusProject/api

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

raymondjacobson · 2026-06-24T01:28:02Z

Summary

return a stable filename string from the track stems endpoint when tracks.orig_filename is null
order stems by track ID for deterministic responses
add a regression case for a stem row with a null original filename

Verification

go test ./api -run TestGetTrackStems -count=1

Live investigation

GET /v1/tracks/qJ0RAXE/stems was returning 500: cannot scan NULL into *string for Burko's contest track
read-only prod query verified parent track 1960897973 has 30 current, non-deleted stem rows, 10 distinct titles, and all 30 rows have orig_filename IS NULL

…ing (stems invisible since ~Jul 12 cutover) (#995) ## TL;DR `/v1/tracks/{id}/stems` fails the entire request when any selected column scans NULL into a non-pointer Go field, and every stem indexed by the Go ETL since the ~July 12 Python→Go cutover has NULL `orig_filename`. The result was a **silent 500** that made the stems section / ContestStemsCard blank for post-cutover uploads — most visibly Andrew Lux's "Alone" remix contest (parent `JBKl7R6`, 43 stems uploaded 7/20, all correct on-chain). #973 fixed the `orig_filename` case on main (June 23), but this PR closes the remaining NULL-scan path (`stem_of` wiped → `category`/`parent_track_id` NULL) and adds regression coverage. ## Root cause chain 1. **The Go ETL never persists `orig_filename`** — the entity-manager create handler doesn't extract it, so only legacy Python-indexed rows carry a value. Verified against the raw on-chain txs (core blocks 28480416–28480440): the metadata contains `orig_filename` and a correct `stem_of`, and replaying the exact tx through the vendored ETL persists `stem_of` fine. 2. **The endpoint scanned NULLs into non-pointer `string`/`int` fields** (`pgx.RowToStructByName`), so one ETL-shaped row 500s the whole response — every stem on the parent disappears, not just the affected one. 3. #973 (June 23) fixed the `orig_filename` scan via `COALESCE(orig_filename, title, '')`. Prod is running `915c37d` today (pods restarted 2026-07-26 08:07 UTC), which includes it — but the incident behavior on 7/20–7/25 is consistent with the pods running a **pre-#973 build** until that 7/26 restart. Worth confirming from deploy history (`kubectl --context prod -n api rollout history` / image tags). 4. **Still broken until this PR:** a `stems` join row can outlive `tracks.stem_of`. An explicit `"stem_of": null` in a client update wipes the column (clients send the full track object on edit — same failure class as the CID wipe fixed in OpenAudio/go-openaudio#410) while the `stems` row survives. `category` and `parent_track_id` then come back NULL from the jsonb and the request 500s again. ## Fix - `COALESCE(t.stem_of->>'category', '')`, `COALESCE(t.track_cid, '')` - `parent_track_id` now read from the `stems` join key (never NULL) instead of the jsonb - Regression test `TestGetTrackStemsWipedStemOf`: seeds a stems row whose track has no `stem_of`/`orig_filename` — 500 before, 200 after ## Blast radius Every stem uploaded after the ~July 12 cutover was invisible in the UI while the NULL-scan bug was live (any parent with ≥1 ETL-shaped stem row 500'd). The DB rows themselves are intact — once the endpoint tolerates NULLs, existing stems reappear with no reindex. ## Andrew Lux follow-up (support) Andrew deleted all of his stems while debugging (one on 7/25 13:38 UTC after 9 retries, a fresh batch uploaded 7/25 14:08–14:50, then a bulk on-chain `Track/Delete` of everything on 7/27 12:35 UTC). Deleted tracks can't be un-deleted on-chain, so **after this deploys he needs to re-upload the stems once** via Edit Track → stems on "Alone" — the same flow he used before; nothing was wrong with his uploads. ## Go ETL follow-ups (OpenAudio/go-openaudio — patches staged locally, separate PR) - [ ] Persist `orig_filename` on track create/update (Python parity; never-clear like the CIDs) so new rows don't rely on the title fallback - [ ] Ignore explicit `"stem_of": null` in track updates (same guard as the #410 CID fix) so client edits can't unlink stems - [ ] Upsert the `stems` join row when an update carries `stem_of` (also gives us an on-chain repair path for lost links) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

fix: handle null stem filenames
f3e02ed

raymondjacobson merged commit 0881627 into mainJun 24, 2026
5 checks passed