MuScriptor

MuScriptor is a multi-instrument music transcription model developed by Kyutai and Mirelo. MuScriptor is the first music transcription model that has been trained on a large scale dataset of 170k songs from classical music to heavy metal.

Online Demo | Paper | HuggingFace

Try it locally

You can try it locally with the web UI with:

uvx muscriptor serve

or with the CLI:

uvx muscriptor transcribe

Installation

with uv (recommended):

uv add muscriptor

pip install muscriptor

Models

Three variants are published under the MuScriptor HuggingFace organization. Everywhere a model is selected (load_model(), the CLI's --model, serve --model) you can pass the bare size keyword and the weights are downloaded and cached automatically. The architecture is a transformer decoder only. Here are the detailed model sizes:

Variant	Parameters	Layers	Dim	HuggingFace repo
small	103M	14	768	muscriptor-small
medium (default)	307M	24	1024	muscriptor-medium
large	1.4B	48	1536	muscriptor-large

small is the practical choice on CPU-only machines, medium is the default speed/accuracy trade-off, and large is the most accurate but really wants a GPU.

Usage

frompathlibimportPathfrommuscriptorimportTranscriptionModel# Downloads the default "medium" variant from HuggingFace (cached under# ~/.cache/muscriptor/). Also accepts "small"/"large", a local safetensors# path, or an hf:// / http(s):// URL.model=TranscriptionModel.load_model()
# Stream events as they're transcribed. Optionally tell the model which# instruments to expect — run `muscriptor list-instruments` for the names.foreventinmodel.transcribe("audio.wav", instruments=["acoustic_piano", "drums"]):
print(event)
# Or get a MIDI file directlymidi_bytes=model.transcribe_to_midi("audio.wav")
Path("out.mid").write_bytes(midi_bytes)

fromdataclassesimportdataclassfromtypingimportGenerator@dataclassclassNoteStartEvent:
pitch: int# Start of the note in seconds, from the beginning of the audio.start_time: float# A unique index for this note, used to match the corresponding# NoteEndEvent.index: intinstrument: str@dataclassclassNoteEndEvent:
end_time: float# The NoteStartEvent this end matches. Convenient for consumers —# when serializing (e.g. to JSON) drop this field and rely on# `start_event_index` to refer back to the start by id.start_event: NoteStartEvent@propertydefstart_event_index(self) ->int:
returnself.start_event.index@dataclassclassProgressEvent:
# A coarse progress anchor woven into the event stream: `completed` of# `total` 5-second chunks have been transcribed. One is emitted up front# with completed == 0 (so consumers learn `total`), then one per finished# chunk. Advisory only — consumers that just build notes can ignore them.completed: inttotal: intclassTranscriptionModel:
...
deftranscribe(
self,
audio: str|Path|tuple[torch.Tensor, int],
use_sampling: bool=False,
temperature: float=1.0,
cfg_coef: float=1.0,
instruments: list[str] |None=None,
batch_size: int|None=None,
no_eos_is_ok: bool=True,
beam_size: int=1,
) ->Generator[NoteStartEvent|NoteEndEvent|ProgressEvent, None, None]:
"""Transcribe audio into a stream of note events. Args: audio: Path to an audio file, or a tuple `(tensor, sample_rate)` with a float tensor of shape [T] or [1, T] at `sample_rate` Hz. The tuple form is useful when the audio is already loaded in memory. use_sampling: Use temperature sampling instead of greedy decoding. temperature: Sampling temperature (only used when use_sampling=True). cfg_coef: Classifier-free guidance coefficient. Keep to 1 for the released models (they are post-RL) instruments: Optional list of instrument group names to condition the model on (exact names, e.g. ["acoustic_piano", "drums"]). Run `muscriptor list-instruments` (or GET /instruments on the server) for the full list of valid names. batch_size: Number of 5-second chunks processed per forward pass. `None` (default) picks a value based on the device: 1 on CPU, 4 on GPU. Use `batch_size=1` for the lowest streaming latency — larger batches process several chunks together, so events belonging to later chunks of a batch won't arrive until the whole batch finishes. Within a batch, events are always yielded in temporal order; all events from chunk N are emitted before any event from chunk N+1. no_eos_is_ok: If True, a chunk that doesn't emit EOS within the generation budget produces a warning instead of raising. beam_size: Beam search width. 1 (default) uses greedy decoding (or sampling, with use_sampling=True); >= 2 enables beam search, which is slower but can be more accurate. Returns: Generator of NoteStartEvent, NoteEndEvent and ProgressEvent objects. Every NoteStartEvent is guaranteed to be followed by exactly one matching NoteEndEvent later in the stream (with the same `index`). Drum hits appear as a NoteStartEvent immediately followed by its matching NoteEndEvent at the same start time plus a tiny duration. Note: this tokenizer does not preserve velocity (loudness) — only onset/offset timing, pitch, and instrument are recovered. """deftranscribe_to_midi(
self,
audio: str|Path|tuple[torch.Tensor, int],
use_sampling: bool=False,
temperature: float=1.0,
cfg_coef: float=1.0,
instruments: list[str] |None=None,
batch_size: int|None=None,
no_eos_is_ok: bool=True,
beam_size: int=1,
) ->bytes:
"""Same as `transcribe`, but returns a MIDI file as bytes instead of a generator of events. Useful when you want to save the MIDI to disk or send it over a network without going through the event stream. """

CLI

# Transcribe to MIDI (defaults to <audio_file>.mid next to the input)
muscriptor transcribe audio.wav -o out.mid
# Pick a model variant: small / medium / large (default: medium),# a local safetensors path, or an hf:// / http(s):// URL
muscriptor transcribe audio.wav --model large
# Tell the model which instruments to expect (comma-separated names;# run `muscriptor list-instruments` for the full list). Case-insensitive,# and unambiguous abbreviations work: 'timp,dist' = timpani + distorted# electric guitar
muscriptor transcribe audio.wav --instruments acoustic_piano,drums
# Get the event stream instead of MIDI: json (single array) or# jsonl (one event per line, streamed while transcribing); -o - = stdout
muscriptor transcribe audio.wav --format jsonl -o -
# Decoding options: temperature sampling, or beam search (slower, can be# more accurate)
muscriptor transcribe audio.wav --sampling -t 0.8
muscriptor transcribe audio.wav --beam-size 4
# Render a stereo check-mix of the result (left channel = original audio,# right channel = synthesized MIDI; requires fluidsynth on PATH)
muscriptor transcribe audio.wav -o out.mid --auralize check.wav

See muscriptor transcribe --help for the full list of options.

Web UI

A browser client is included under web/. The FastAPI server serves both the UI and a POST /transcribe endpoint that streams NoteStart/NoteEnd events back as Server-Sent Events. The UI accepts an audio file (WAV, or any format soundfile/libsndfile can read — mp3, flac, ogg, m4a, …) via drag-and-drop, renders a live piano roll while events arrive, auto-plays once enough notes are available, and crossfades between the original WAV and the synthesized MIDI playback.

One-time setup

uv sync
cd web && pnpm install && pnpm run build &&cd ..

pnpm run build is required once — the FastAPI server auto-mounts web/dist/ if it exists. The first build downloads two soundfonts: MuseScore_General.sf2 (215 MB, to the repo root, used by /auralize) and MuseScore_General.sf3 (38 MB, the compressed build the UI plays); later builds skip files already on disk.

Run

uv run muscriptor serve \
--model medium \
--device cuda \
--host 0.0.0.0 \
--port 8222

--model accepts a size keyword (small, medium, large) that downloads the matching variant from HuggingFace (cached under ~/.cache/muscriptor/), a local safetensors path, or an hf:// / http(s):// URL. It defaults to medium when omitted.

Then open http://127.0.0.1:8222/ (or the LAN address of the host) and drop a WAV onto the page.

• Drop --device cuda if running CPU-only.
• --host 0.0.0.0 makes it reachable on the LAN; the default 127.0.0.1 is local-only.
• Playback runs a full SoundFont synthesizer (SpessaSynth) in the browser, fed with MuseScore_General.sf3 — the same soundfont the /auralize endpoint uses, served by the app itself from /soundfonts/, no third-party CDN.

License

The code in this repository is released under the MIT license.

The model weights, published on HuggingFace, are released under the CC BY-NC 4.0 license (non-commercial use).

The MuseScore General SoundFont downloaded for playback / auralization is distributed under its own (MIT) license.

Citation

@misc{rouard2026muscriptoropenmodelmultiinstrument,
title={MuScriptor: An Open Model for Multi-Instrument Music Transcription}, author={Simon Rouard and Michael Krause and Axel Roebel and Carl-Johann Simon-Gabriel and Alexandre Défossez},
year={2026},
eprint={2607.08168},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2607.08168}, }