Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Auto-sync: DTW step constraint — riff-based songs no longer produce garbage sync points.** `librosa.sequence.dtw`'s default step pattern allows unbounded horizontal/vertical path runs, and on music with long self-similar chroma stretches (riff-driven stoner/doom, drone sections) the flat cost surface let the warping path collapse — minutes of score mapped onto a single audio frame, so the per-bar warp imported charts wildly out of sync while reporting success (observed on a real 138 BPM tab: effective displayed tempo 159 BPM, three sync points sharing one audio timestamp). `_dtw_align` now uses the standard music-sync slope-constrained step pattern (`[[1,1],[1,2],[2,1]]`, local tempo ratio bounded to 0.5x–2x), which makes the degenerate path impossible, with a fallback to unconstrained steps when the global length ratio makes the constrained pattern infeasible (e.g. a tab aligned against a full-concert video). Validated on the failing song: coarse points track the recording 1:1, refined downbeats land on onset peaks at 3.3x background energy.

### Added
- **3D Keys Highway: key layout modes, lane-color opacity & octave lines.** A new **Highway layout** settings section rebuilds how sharps/flats and lanes draw on the 3D piano highway. **Sharps & flats layout** (`keys3d_bg_sharpMode`) picks between **floating** (the original raised-sharp look), **flat** (one plane, zero-overlap piano-shaped tiled lanes with the naturals evened out), and **realistic** (one plane, bars sized like the physical keys) — default **realistic**; the geometry lives in pure, unit-tested `laneSpanFlat()`/`laneSpanReal()` helpers. **Lane color opacity** (`keys3d_bg_laneOpacity`, 0–1, default 0) fades the pitch-class lane tint from full vivid color down to a dark floor with guide lines only at the key-block boundaries (E→F and each octave); the lane strips, per-lane separators and block lines crossfade with the value. **Octave separators** (`keys3d_bg_octaveGaps`, default on) and **Octave line contrast** (`keys3d_bg_octaveContrast`, 0–1) control the B→C octave divider, which auto-shifts from a dark to a bright layer as lane opacity fades. Settings re-read on init and apply on the next chart build. `plugins/keys_highway_3d` → 0.2.0. Tests: `plugins/keys_highway_3d/tests/fx_settings.test.js` (new defaults, sharp-mode setting, lane-geometry tiling/evening for flat, uniform/overlap for realistic, and an active-range boundary case where a white key's edge stays untrimmed when its neighboring sharp falls outside the active range).
- **Handedness (left-handed) is now a first-class choice in the instrument selector — and surfaced during onboarding.** Left-handed players could already mirror the highway, but only via a buried Settings toggle they had to find *after* setup — so a lefty hit the tour, the tuner and calibration all right-handed first. The v3 instrument badge popover now has a **Handedness: Right / Left** row alongside Instrument / Strings / Tuning (all player-orientation choices), writing the same `lefty` preference (`highway.setLefty` when a live highway exists, else the `lefty` localStorage key it reads on init; the Settings checkbox stays in sync). The first-run tour's "Choose your instrument" step — which runs **before** the tuner/audio-calibration steps — now calls it out so lefties flip it up front. Frontend-only, additive: `static/v3/badges.js`, `static/v3/onboarding-tour.js`. Tests: `tests/js/badges_handedness.test.js`.
- **"Colorblind (deuteranope)" highway string-color preset.** Adds a one-click preset to the shared "Highway String Colors" picker, sitting next to the existing Okabe–Ito "Colorblind-friendly" preset — contributed by a deuteranopic player who found the Okabe–Ito set still hard to separate. It retunes the six main strings (red / yellow-green / blue / orange / teal / deep-purple) and keeps that set's 7/8-string colors, and applies to **both** the 2D and 3D highways via the shared picker. Frontend-only, additive: `static/app.js` (`HWC_PRESETS`).
- **`lib/gp_autosync.py`: piecewise time-warp helpers + a working `refine_sync()`.** `auto_sync()` has always computed per-bar sync points (DTW), but consumers could only apply the scalar bar-1 `audio_offset`, so any tempo difference between the recording and the tab's authored tempo accumulated audibly over the song. New librosa-free helpers expose the full mapping: `bar_start_times(gp_path)` (per-bar score times on the same axis as the sync points — GPIF bar-resolution map for `.gp`/`.gpx`, per-tick integration for GP3/4/5), `build_warp_anchors(points, bar_starts)` (strictly-monotonic `(score, audio)` anchor pairs), `warp_time(t, anchors)` (piecewise-linear map with edge-slope extrapolation for count-ins/tails), `warp_song_times(song, warp)` (retimes a `lib.song.Song` in place: notes + sustains, chords, beats, sections, anchors, handshapes, per-phrase difficulty levels, tone changes, tempo overrides), and `gp_has_expandable_repeats(gp_path)` (detects GP3/4/5 repeat/volta/direction markup whose playback expansion the as-written sync points cannot map — callers fall back to offset-only sync). Also implements `refine_sync()`, which the editor plugin's refine-sync endpoint has imported since the snapshot but which never existed in core (the Refine button 500'd): it densifies the coarse DTW points to every Nth bar and re-times each with a local onset phase sweep (sweep radius clamped under half a beat so periodic material can't lock a full beat off; short scoring grid + median residual snap). Synthetic click-track validation: ~13ms mean / ~40ms max error from ±180ms coarse input across 117–123 BPM recordings of a 120 BPM tab. Tests: `tests/test_gp_autosync_warp.py`.
Expand Down
25 changes: 23 additions & 2 deletions plugins/keys_highway_3d/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,34 @@
RS+-style falling-note 3D piano highway for [Slopsmith](https://github.com/got-feedback/feedback), fed by the **Sloppak Notation Format** (sloppak-spec §5.3) — part of the piano/keys first-class epic (slopsmith#828, plugin workstream slopsmith#824).

- Consumes the `notation_info` / `notation_measures` highway-WS stream over a private per-instance socket and flattens measure → staff → voice → beat → note into `{midi, t, durSec, hand}` (durations derived from written `dur`/`dot`/`tu` at the running tempo; ties extend; overlap-clamped).
- 3D perspective highway to a vanishing point with a real white/black-key keyboard; per-key **pitch-class colours** (Synthesia convention — C red, D yellow, E blue, …) with hand (rh/lh) as a secondary brightness cue. Selectable **note-colour palettes** (settings → Note colours, `keys3d_bg_palette`, default the per-octave scheme): a per-octave rainbow (each octave its own hue, darker sharps), the original per-pitch "Rainbow" table, vivid/pastel per-pitch variants, and single-hue two-tone palettes (uniform naturals, darker sharps) for players who want "black key coming" to read at a glance; notes, key glow, lane guides and hit flames all follow the pick live.
- 3D perspective highway to a vanishing point with a real white/black-key keyboard; per-key **pitch-class colors** (Synthesia convention — C red, D yellow, E blue, …) with hand (rh/lh) as a secondary brightness cue. Selectable **note-color palettes** (settings → Note colors, `keys3d_bg_palette`, default the per-octave scheme): a per-octave rainbow (each octave its own hue, darker sharps), the original per-pitch "Rainbow" table, vivid/pastel per-pitch variants, and single-hue two-tone palettes (uniform naturals, darker sharps) for players who want "black key coming" to read at a glance; notes, key glow, lane guides and hit flames all follow the pick live.
- Full RS+ visual treatment: key **letter glyphs** printed on the active-range key tops (cached CanvasTextures), **bevelled gem-style note blocks** (ExtrudeGeometry, geometry/material caches keyed by size and pitch-class×hand), **floating bar numbers** scrolling with the notes, **active-range lane dimming** so the playable span pops, and a **glowing pulsing hit-line** (layered additive gradient planes — no postprocessing).
- Performance discipline: no per-frame allocations or DOM queries in `draw()`. Chart-scoped resources — note geometries/materials, bar-number and glow textures — are cached and disposed on chart teardown; the key-letter glyph `CanvasTexture`s live in a shared module-level cache that survives teardown and is reused across instances.
- Auto-selected for arrangements with notation via `matchesArrangement(songInfo.has_notation)`; capability-native `visualization` provider declaration.
- **Camera settings**: camera-rig presets (`keys3d_bg_camera` — classic low rig / elevated / overhead; default overhead, applied live, adaptive pan-zoom preserved) with base-rig fine-tune sliders for height, distance and tilt (`keys3d_bg_camHeight` / `camDist` / `camTilt`) that nudge the vantage point the follow-motion orbits. Numeric FX keys clamp to per-key declared ranges (`FX_RANGES`, default 0–1).
- **Highway-layout options** (settings → Highway layout). **Sharps & flats**
(`keys3d_bg_sharpMode`, string; default `realistic`) picks the sharp layout:
`floating` (original raised-plane sharps, white-only lanes); `flat` (one plane,
zero-overlap piano-shaped tiled lanes — white lanes trimmed where a sharp adjoins
them, and each sharp leaned toward the edge natural beside it so the naturals come
out close to even: C/D/E/F/B equal, G/A a hair smaller since G# can't lean; pure
`laneSpanFlat()`); `realistic` (one plane, bars sized like the physical keys — full
naturals always rendered full, full black keys drawn on top and only occluding a
natural where a sharp note actually coincides in time; pure `laneSpanReal()`).
**Lane color opacity** (`keys3d_bg_laneOpacity`, 0–1, default 0) fades the
pitch-class lane tint; at 0 (default) the strips are a dark floor with guide lines
only at the key-block boundaries (E→F and each octave B→C), so each block is bounded
rather than every lane — the notes keep their colors; toward 1 it fills in full,
vivid colored lanes. The strips, per-lane separators and block lines crossfade with
this value. **Octave separators** (`keys3d_bg_octaveGaps`, default on) widens the
gap a touch at each B→C octave boundary. **Octave line contrast**
(`keys3d_bg_octaveContrast`, 0–1, default 0.5) scales how hard the B→C octave line
reads; it is drawn as a dark layer (scaled by lane opacity) plus a bright layer
(scaled by its inverse), so it auto-shifts dark→bright as the lanes fade — no mode
switch needed. All are geometry-time — applied on the next chart build via
`init()`'s re-read.
- **Web MIDI input scoring**: module-level MIDI singleton (one access per tab, focused-instance routing) with device auto-connect by saved id+name, loopback blocklist, channel filter, transpose and CC64 sustain (`keys3d_` localStorage prefix; `window.keysH3d*` settings API). Hit detection matches played MIDI against the flattened chart notes within ±0.10 s with per-note dedupe and a missed-note sweep (only while a device is connected — never retroactive across a mid-song connect).
- **Live hit feedback on the MIDI path** (not the chart): key depress (~4° back-edge pivot, ~120 ms spring; the key letter rides along), wrong-note red key flash, and a vertical flame flare on hits (pooled additive sprites, white-hot base fading into the pitch-class colour, ~400 ms).
- **Live hit feedback on the MIDI path** (not the chart): key depress (~4° back-edge pivot, ~120 ms spring; the key letter rides along), wrong-note red key flash, and a vertical flame flare on hits (pooled additive sprites, white-hot base fading into the pitch-class color, ~400 ms).
- **End-of-run stats**: POSTs `/api/stats` `{filename, arrangement, score, accuracy}` exactly once per run with the same formula as the guitar notedetect path (`accuracy = hits / max(1, hits+misses)`, `score = round(hits·100·accuracy)`), then notifies the progression core when present.
- **Capability wiring** (all guarded for servers without the hosts): registers as a note-detection `midi` provider (`keys-midi`, `verify.target`), opens a per-song binding scoped to the chart's keys range, reports hit/miss observability events, and exposes Web MIDI inputs to the audio-input domain with pseudonymized labels (`midi-input-1`, …) via `source.enumerate/describe/open/close`.
- Headless test hook: `window.__keysHwTest = { injectNoteOn(midi, when), getScore() }`.
Expand Down
2 changes: 1 addition & 1 deletion plugins/keys_highway_3d/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "keys_highway_3d",
"name": "Keys Highway 3D",
"version": "0.1.2",
"version": "0.2.0",
"description": "RS+-style 3D falling-note piano highway fed by the Sloppak Notation Format, with Web MIDI input scoring.",
"type": "visualization",
"bundled": true,
Expand Down
Loading
Loading