JIT-MPE Beats Offline AI Music Models With Just 86,000 Parameters
A tiny 86k-parameter causal model estimates which notes are sounding within 50 ms of audio arriving, matching or beating much larger offline systems.
- JIT-MPE is a causal 86k-parameter multi-pitch estimation model built for real-time use with explicit output deadlines.
- Five delay heads let each 20 ms frame be revised with 0, 20, 40, 80 or 160 ms of lookahead.
- At a 50 ms deadline it beats 5 of 6 offline MPE systems in 38 of 45 dev comparisons despite being 6 to 170x smaller.
- Recall in the first 20 ms after a note onset climbs from 9.8% to 38.4% once the 50 ms refinement arrives.
- Slowest frame runs in 4.15 ms on one CPU thread, roughly 2.4 ms typical; mobile-ready.
- Code and weights are MIT-licensed at github.com/Frieve-A/jit-mpe; a variant ships in EffeTune.
Most multi-pitch estimators assume offline processing: they receive a complete recording and return a piano roll. That workflow cannot support live score followers, on-stage tuners, automatic accompaniment, or other applications that need the currently sounding notes. The JIT-MPE paper addresses that gap with an 86,000-parameter causal model and an evaluation protocol based on output deadlines.
Deadlines change the benchmark
Instrument-agnostic multi-pitch estimation, or MPE, determines which of the 88 piano-key pitches are active in each audio frame, whether the source contains voice, guitar, violin, or a mixed ensemble. Many leading systems process complete recordings, allowing them to use audio that occurs long after the frame being labeled. Live software can use only the audio received so far.
JIT-MPE evaluates each estimate against a deadline D. An answer for query time q must be published by q + D, using only audio available by then. The protocol lets researchers compare streaming systems according to both accuracy and publication time.
Five heads revise each frame
JIT-MPE processes 44.1 kHz audio in 20 ms hops. For every frame, it predicts active pitches from MIDI 21 through 108 and estimates a fine frequency offset around each pitch center. Its causal architecture uses no future audio beyond the selected deadline.
Five output heads estimate the frame that ended 0, 20, 40, 80, or 160 ms earlier. A live application can publish the zero-delay estimate immediately, then replace it with estimates that incorporate later audio. Those revisions help near note onsets, where the first few milliseconds contain limited evidence about periodicity and timbre.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
