Readme
avdesc: automatic audio description (v0.4)
Upload a video and get it back with audio description: short spoken descriptions of what is on screen, placed in the pauses between dialogue, for blind and low-vision viewers. The descriptions are also drawn into the picture, with the dialogue captions, so sighted reviewers can check them, and you get the text files for your own player or editor.
What you get
| Output | |
|---|---|
described_video.mp4 |
your video with the descriptions spoken (the film’s sound is dipped under them) and shown: a blue AD box at the top while each description is spoken, dialogue captions at the bottom. Audio track 1 = described soundtrack, track 2 = original |
described_audio.m4a |
the described soundtrack on its own |
audio_description.zip |
all the text files below |
descriptions.vtt |
WebVTT description track (for HTML5 players) |
captions.vtt, captions.srt |
dialogue captions |
combined.srt |
dialogue + [AD] descriptions, for players without description tracks |
script.md |
timed script for a voice artist |
project.json |
everything (timings, text, evidence), editable |
Inputs
| Input | Default | |
|---|---|---|
video |
required | MP4, MOV, MKV, WebM, … with English dialogue |
subtitles |
none | optional .srt / .vtt: exact dialogue timing and names (if the form refuses .srt, rename it to .txt) |
subtitles_text |
empty | or paste the subtitle text here |
cast |
empty | optional, e.g. Lisbeth (young woman, black hair); Mikael (man, glasses): lets the descriptions use names |
video_context |
empty | optional, e.g. crime drama film |
density |
standard | sparse, standard (as evaluated), film (close to professional film AD), dense |
voice |
af_heart | 28 English voices: a American, b British; f female, m male |
speed |
1.0 | narration speed, 0.7-1.4 |
show_descriptions |
on | draw the description text into the picture |
show_captions |
on | draw the dialogue captions into the picture |
Subtitles and a cast list give the best results: correct names and exact pauses.
How it works
- Speech: Demucs separates the voices from music and effects; Whisper small.en transcribes; voice-activity detection marks every sound of speech; subtitles, if given, give exact timing and names.
- Pauses: the gaps between dialogue are found and split at shot cuts.
- Seeing: Qwen3-VL-8B looks at frames from each pause and reports what is visible.
- Writing: Qwen3-32B writes a short candidate description for each pause, with an importance score.
- Choosing (code rules, not the AI): never over dialogue, must fit its pause, important on-screen text first, no repeats, spacing set by the density.
- Voice and video: a Kokoro-82M voice reads each description (a little faster if needed to fit), the film is dipped under it, and the text is drawn into the picture.
Speed and hardware
Runs on one GPU (L40S 48 GB or A100). The first request after a cold start waits several minutes while the 32B writer model (~20 GB) downloads; later requests start immediately. As a guide, the pipeline described a 92-second clip in about 2 minutes on an A100; re-encoding the picture adds roughly the video’s length or less.
Limitations and intended use
- English dialogue and English descriptions only.
- The descriptions are machine-written: they can miss things, misread a scene or confuse characters. Have a person
review them before publishing (
project.jsonandscript.mdare made for editing). Extended description (pausing the video) is not supported: descriptions only go where there is a pause. - Intended for accessibility work, research and drafting AD scripts, not as a replacement for professional describers.
- Uploaded videos are processed on Replicate’s GPU servers. Only upload videos you have the right to process.
Versions
- v0.4 (latest): described video (voice + on-screen text), on the v0.3 pipeline.
- v0.1: text outputs only. Still available under Versions and can be run by its version ID.
Models and licences
Qwen3-VL-8B and Qwen3-32B (Apache-2.0) via Ollama (MIT); Whisper (MIT) via faster-whisper; Demucs (MIT); Kokoro-82M (Apache-2.0) via kokoro-onnx (MIT); DejaVu fonts. Code: GPL-3.0, see the GitHub repository.
Authors: Deep Learning Lab KMUTT