Proposal: in-browser tone engine (live guitar input + NAM amp sim)

[mogul]

Restored from slopsmith/slopsmith#8 — original issue, opened by @vulngraph-ai on 2026-04-12.
[restored-from: slopsmith/slopsmith#8]

Summary

Proposal to add a software tone path to Slopsmith so users can plug a guitar into any USB audio interface (Real Tone Cable, Focusrite, etc.) and hear it through the song's tone chain in-browser — no external hardware modeler required.

This is intended to complement, not replace, the existing tone ecosystem:

• slopsmith-plugin-tones already parses the [redacted] tone_*.json manifests and renders the signal chain. Reusable as-is.
• slopsmith-plugin-midi already drives external hardware modelers via MIDI program change on tone switches. Users with a Helix/Kemper/Axe-FX are covered.
• Gap: users without external hardware currently have no way to hear their guitar through the song's tone. This proposal fills that gap entirely in the browser, keeping Slopsmith Linux-native and dependency-light.

Scope (plugin-first)

I'd build this as a standalone plugin (slopsmith-plugin-tone-engine or similar), following the same pattern as slopsmith-plugin-3dhighway — opt-in, no core bloat, no new server-side dependencies beyond what the plugin ships itself. If it proves itself, core integration can be discussed later.

Technical approach

All browser-side, all WASM, no native code:

1. Input capture — getUserMedia({ audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false, latency: 0 } }) → MediaStreamAudioSourceNode into the existing AudioContext.
2. DSP in an AudioWorklet so processing runs on the audio thread (~5–10 ms round-trip on Linux/PipeWire with a decent interface).
3. Amp stage: Neural Amp Modeler (NAM) via its WASM runtime. Users load .nam capture files (thousands available free on ToneHunt, including high-gain captures that match common CDLC tones). Optionally also support AIDA-X / Proteus for lighter models.
4. Cab stage: ConvolverNode with user-supplied IR .wav files. Standard, cheap, sounds great.
5. Pedals / utility DSP (noise gate, TS-style boost, EQ, delay, plate reverb) reimplemented as small Worklet nodes — well-trodden open-source DSP.
6. Tone auto-selection: read the parsed tone_*.json (via the existing tones plugin's parser, or a shared lib) and map each song's amp/cab/pedal IDs to a user-configured NAM capture + IR + pedal chain. Falls back to a default ''clean / crunch / lead'' set when no mapping exists.
7. Guitar stem ducking: when the [redacted] has multitrack stems, mute the guitar bus so the user's live signal sits in the mix. When it doesn't, support an optional preprocessed stem-separated backing track (Demucs one-shot) as a per-song asset.
8. Latency compensation: expose the measured input→output latency and offset the note highway timing so hits line up visually with what the user hears.

Why NAM specifically

Modern neural amp captures are effectively indistinguishable from the source amp/plugin in a null test at a fixed knob setting. The ''tone quality'' problem is solved; what's left is integration work. This keeps the project fully open-source and legally clean — users bring their own captures (their own files, their own license to make them).

What I'm asking

I'm willing to build this end-to-end — plugin scaffolding, WASM NAM integration, AudioWorklet DSP chain, tone-mapping UI, latency calibration, the lot. I don't need help with the implementation; I just want to confirm direction before investing the time:

1. Does a plugin-first approach sound right to you, or would you prefer this land somewhere else (core, or a separate fork)?
2. Any constraints I should know about — e.g., plans for .sloppak (feature: integrate per-stem volume sliders into the mixer popover #7) that might affect where tone metadata lives, or conventions for plugins that need audio-graph access rather than just UI hooks?
3. Would you be open to a small core PR later that exposes a stable hook for plugins to tap into the AudioContext / insert nodes into the playback graph? That's the one piece I can't cleanly do from a plugin today without patching static/app.js, and it'd benefit any future audio-processing plugin, not just this one.

Happy to open a draft PR with the plugin scaffolding as soon as there's a green light on direction. Thanks for building Slopsmith — this project is genuinely exciting and I'd love to contribute something substantial to it.

[mogul]

@byrongamatos commented on 2026-04-13:

Love this proposal — an in-browser tone engine was actually one of the very first things I wanted to build into Slopsmith. It's exciting to see someone else independently arrive at the same idea with a solid plan.

What we tried and why we went native

We hit the same wall you're about to hit and ended up building the tone engine as a native C++ addon for Slopsmith Desktop (Electron + JUCE). Here's what we learned:

The desktop implementation uses JUCE's audio engine with a full signal chain: Guitar Input → [Input Gain] → [Processor Slots: NAM/VST/IR] → [Backing Track Mix] → [Output Gain] → Speakers. NAM models are loaded via the NAM Core library with RTNeural (XSIMD-vectorized Eigen backend) for real-time neural network inference. Cabinet IRs use JUCE's FFT-based dsp::Convolution. VST3/AU/LV2 hosting is handled by JUCE's AudioPluginFormatManager. The whole thing is bridged to Electron via a 700+ line N-API addon with ~40 exported functions, and the audio thread uses lock-free atomics for metering and pitch detection.

The challenges that pushed us to native:

1. Latency — JUCE on ASIO/JACK gives us <5ms round-trip. Web Audio API's AudioWorklet is realistically 10-20ms minimum (128-512 sample buffers), and on many setups it's worse. For a rhythm game where note hits need to line up visually with what you hear, that extra latency compounds with the display latency and starts to feel wrong.

2. NAM model performance — RTNeural in C++ with vectorized SIMD achieves ~1-2ms per buffer block. WASM doesn't have access to the same SIMD intrinsics (WebAssembly SIMD is more limited than AVX2/NEON), so we estimated 5-10x slower inference. Whether that's fast enough for real-time at your target buffer size is the big question mark.

3. Device control — no Web API for ASIO device enumeration, JACK port routing, or buffer size negotiation. getUserMedia gives you a device with whatever settings the browser picks.

4. VST hosting — obviously impossible in-browser, which is fine for your scope (NAM + IR only), but it limited our ambitions at the time.

That said — our constraints were different. We were targeting a full production signal chain (VSTs, multiple NAM models, complex routing). Your proposal is deliberately scoped to NAM + ConvolverNode + simple utility DSP, which is a much more realistic target for WASM. And since the audio processing runs on the user's client machine (not the Docker host), the performance concern is about their browser, not the NAS — which is actually fine for most users.

Green light

To answer your three questions:

1. Plugin-first is exactly right. Same pattern as every other plugin. If it proves itself, we can discuss core integration later.

2. Sloppak constraints — .sloppak songs already have per-stem audio and the stems mixer plugin already manages a Web Audio graph with GainNode per stem. Your guitar stem ducking idea (mute the guitar bus when the user is playing live) would hook into that naturally. Tone metadata in sloppak lives in the YAML manifest under each arrangement's tones key, same structure as [redacted] tone_*.json.

3. AudioContext hook PR — yes, please. This is the right move. A stable plugin hook to tap into the audio graph would benefit stems, note detection, metronome, and your tone engine. Happy to review a small core PR for that. The stems plugin currently creates its own AudioContext and uses the core <audio> as a silent timing master — a shared hook would clean that up too.

One suggestion: start with the AudioContext hook PR first, since that unblocks everything else and is the smallest reviewable unit. Then the tone engine plugin can build on top of it.

Keen to see a draft PR. Welcome aboard.

[mogul]

@byrongamatos commented on 2026-04-15:

We went ahead and built this ourselves. The plugin is live:

slopsmith-plugin-nam-tone

What's in it

• NAM via WASM AudioWorklet - uses the neural-amp-modeler-wasm package (TONE3000's build of NAM Core + RTNeural compiled to WebAssembly). Runs in an AudioWorklet in single-threaded mode, so no COOP/COEP headers needed.
• Cabinet IR loading - native ConvolverNode, swap IRs at runtime
• Preset system - SQLite backend, create named presets mapping a NAM model + IR + gain/gate settings
• Tone auto-switching - reads highway.getToneChanges() at 100ms intervals, swaps presets automatically as tones change during a song. Pre-caches all mapped models at song load.
• Guitar stem ducking - when the tone engine is active on a sloppak song, the guitar stem is automatically muted so you play "for real" in the mix. Uses window._stemsState exposed by the stems plugin.
• Latency compensation - configurable offset for highway timing

Signal chain

Guitar (USB interface) -> getUserMedia
  -> MediaStreamSource
  -> InputGainNode (sensitivity)
  -> Noise Gate (threshold-based)
  -> NAM AudioWorkletNode (WASM inference)
  -> ConvolverNode (cabinet IR)
  -> OutputGainNode (master volume)
  -> ctx.destination

Performance

Latency is 20-50ms depending on the system - acceptable for practice, not for professional recording. Requires a modern CPU (Apple Silicon or recent Intel/AMD). The WASM module runs in single-threaded mode which avoids SharedArrayBuffer requirements.

The same functionality is also available natively in Slopsmith Desktop with JUCE, where latency drops to <5ms via ASIO/JACK and VST3/AU/LV2 hosting is available.

Thanks to @vulngraph-ai for the original proposal that kicked this off - the architecture closely follows what you outlined.

[mogul]

@vulngraph-ai commented on 2026-04-16:

You guys are LEGENDS!!! and move way too fast for me!!

I was going to take a stab at this over the weekend!!