ElevenLabs API scores video with music and sound effects

ElevenLabs has released a Video to Music API that accepts one or more video files, composes an original score around the footage, and can add sound effects synchronized to on-screen events. The endpoint powers Instant Soundtracks in ElevenCreative and is available through ElevenAPI for enterprise customers and developers.

The system analyzes cuts, action, and changes in mood before generating the music. When sound effects are enabled, it identifies suitable cues, creates the effects, and places them at matching timestamps.

One request, three controls

A single POST /v1/music/video-to-music request accepts multipart form data. Multiple videos are concatenated in upload order into one timeline for scoring.

Three Boolean parameters determine how the endpoint handles effects, output files, and source audio:

• add_sfx: Detects opportunities for sound effects and includes generated effects in the output.
• separate_tracks: Returns a ZIP archive containing separate music and sound-effect files, allowing downstream mixing in an editor or automated pipeline.
• use_audio_as_reference: Analyzes the source audio for genre, tempo, instrumentation, mood, and musical arc, then uses those characteristics to guide the new score.

The endpoint supports the music_v1, music_v2, and music_v2_5 models. Available formats range from low-bitrate MP3 to 44.1 kHz PCM. MP3 output can also carry C2PA metadata that records its generated-media provenance.

Run it with cURL

curl -X POST https://api.elevenlabs.io/v1/music/video-to-music \
  -H "xi-api-key: $ELEVEN_KEY" \
  -F "videos[]=@scene.mp4" \
  -F "add_sfx=true" \
  -F "separate_tracks=true" \
  -F "use_audio_as_reference=true" \
  -F 'tags=["cinematic","tense"]' \
  -o soundtrack.zip

With both effects and separate tracks enabled, the response body is a ZIP archive containing the music and sound-effects files. Requests without separate tracks return an audio file. The response also includes a song-id header for tracking the generation.

Synchronization removes manual edit steps

ElevenLabs’ existing Music API composes tracks from text prompts or audio references without analyzing video. Scoring footage previously required generating music, cutting it to picture, creating effects separately, and placing each cue on an editing timeline. The new endpoint combines those operations in one request and returns audio already aligned to the supplied footage.

The use_audio_as_reference option allows a rough cut or scratch track to shape the score. A fast section can guide the model toward electronic or R&B instrumentation, while a slower scene can prompt a transition toward acoustic textures. Teams that need dialogue, ambience, or production sound should retain those source tracks in their editing pipeline rather than assume the reference audio will be preserved in the generated output.

Best fits and production checks

The endpoint best suits footage where synchronization to cuts and visible action matters, including advertisements, trailers, social clips, game cinematics, and product demonstrations. Separate tracks let an editor replace the music while retaining the effects, adjust levels independently, or remove an unwanted cue without regenerating the entire soundtrack.

• Each request is limited to 10 videos, 200 MB of combined input, and 600 seconds of total footage.
• Server-side concatenation can increase request duration, so ElevenLabs recommends joining clips before upload when practical.
• The Music API requires a paid account, while access to this endpoint depends on the applicable developer or enterprise plan.
• Commercial rights vary by plan. Uses including film and television require an Enterprise plan.
• Production evaluations should measure cut detection, effect selection, cue timing, music transitions, output levels, latency, pricing, and account quotas against representative footage.

How it joins the ElevenLabs stack

ElevenLabs also exposes the workflow through the ElevenCreative Music Video Editor, where effects appear as a separate layer; Creative Engine, for adding audio to advertisements; the ElevenMusic platform; and the ElevenCreative Studio API through the same add_sfx, separate_tracks, and use_audio_as_reference controls.

Teams already using ElevenLabs for text-to-speech or dubbing can now source speech, music, and effects from the same platform. Their existing editing or mixing pipeline remains responsible for source dialogue, loudness, approvals, and any manual corrections to the model’s timing.