Kyutai's MuScriptor Turns Any Band Recording Into Clean Per-Instrument MIDI

Automatic music transcription has been a stubborn open problem for decades. Drop a full band recording into any existing tool, and you'll get back a muddy piano roll that blends all instruments into one indecipherable blob. MuScriptor, a new open-weight model from Kyutai and Mirelo, is the first serious attempt to crack this at scale: give it any recording, in any genre, and it returns clean, per-instrument MIDI.

The wall that stopped everyone else

Multi-instrument music transcription aims to convert polyphonic music recordings into musical scores assigned to each instrument. The task is challenging because it requires simultaneously identifying multiple instruments and transcribing their pitch and precise timing, and the lack of fully annotated data adds to the training difficulties.

The 2026 paper introduces MuScriptor as an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres. The prior art, Google's MT3 from 2022, was a landmark in the field but hit a ceiling almost immediately: existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings.

A three-stage recipe that actually works

The core insight from the Kyutai team is that data quality, not architecture, was the real bottleneck. They built a three-stage training pipeline to address it:

1. Synthetic pre-training: The model first trains on 1.5 million MIDI files, synthesized in many variations to build broad musical coverage cheaply.
2. Real-data fine-tuning: The team collected a dataset of 170,000 music recordings (11,000 hours) paired with MIDI transcriptions spanning many genres. This is the single biggest driver of quality.
3. RL post-training: Finally, the model is post-trained using reinforcement learning on 300 tracks with manually verified, accurate transcriptions, sharpening its output on the hardest cases.

The RL post-training step is particularly interesting. Rather than just maximizing likelihood on noisy labels, the model is rewarded for getting things right on a small, high-quality gold set. This is the same intuition behind RLHF in language models, applied here to musical note prediction.

Under the hood

The architecture is a decoder-only transformer that autoregressively predicts a stream of MT3-like tokens from a mel-spectrogram of a 5-second audio excerpt. Longer recordings are processed chunk-by-chunk. Three model sizes are available:

One notable feature is instrument conditioning: you can tell the model which instruments to expect, and it will focus its transcription accordingly. This is useful when you already know the lineup of a recording and want cleaner separation.

One current limitation worth flagging: the tokenizer does not preserve velocity (note loudness). You get precise onset/offset timing, pitch, and instrument identity, but dynamics are not recovered. For many production workflows this is fine, but it matters if you're trying to recreate an expressive performance.

Getting started in minutes

The library is available on PyPI and HuggingFace. The simplest path is via uvx:

uvx muscriptor serve        # launches local web UI
uvx muscriptor transcribe   # CLI transcription

Or use the Python API directly:

from muscriptor import TranscriptionModel
from pathlib import Path
model = TranscriptionModel.load_model()  # downloads "medium" by default
# Transcribe to MIDI bytes
midi_bytes = model.transcribe_to_midi("song.wav")
Path("out.mid").write_bytes(midi_bytes)
# Or stream note events as they arrive
for event in model.transcribe("song.wav", instruments=["acoustic_piano", "drums"]):
    print(event)

The web UI includes a live piano roll that renders as events stream in, and a crossfade between the original audio and synthesized MIDI playback, powered by a full SoundFont synthesizer running in the browser.

Who should care

The practical use-cases are wide:

• Music producers who want to pull stems or riffs from reference tracks into their DAW as editable MIDI
• Musicians learning by ear who want an accurate transcription of a solo or chord progression
• Music information retrieval researchers who need a strong open baseline for downstream tasks like chord recognition, structure analysis, or cover detection
• Game and film composers who want to adapt existing arrangements into new instrument configurations
• Educators building interactive sheet music tools

The bigger shift

What MuScriptor signals is a change in assumptions about what's possible with open models in music AI. Prior AMT research has been hindered by low-resource datasets and task-specific architectures, preventing comprehensive multi-instrument transcription across diverse musical contexts. The Kyutai team's answer was not a cleverer architecture but a much larger and more carefully curated dataset, combined with RL fine-tuning on verified ground truth.

The model weights are released under CC BY-NC 4.0 (non-commercial use), and the code is MIT licensed. The paper and full codebase are both public. For anyone building music tools, this is now the open baseline to beat.