ahmad-tijjani-garba/avdsec

Automatic English audio description: returns your video with spoken, captioned descriptions of what is on screen for blind and low-vision viewers.

Public
5 runs

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

avdesc: automatic audio description (v0.4)

Upload a video and get it back with audio description: short spoken descriptions of what is on screen, placed in the pauses between dialogue, for blind and low-vision viewers. The descriptions are also drawn into the picture, with the dialogue captions, so sighted reviewers can check them, and you get the text files for your own player or editor.

What you get

Output
described_video.mp4 your video with the descriptions spoken (the film’s sound is dipped under them) and shown: a blue AD box at the top while each description is spoken, dialogue captions at the bottom. Audio track 1 = described soundtrack, track 2 = original
described_audio.m4a the described soundtrack on its own
audio_description.zip all the text files below
descriptions.vtt WebVTT description track (for HTML5 players)
captions.vtt, captions.srt dialogue captions
combined.srt dialogue + [AD] descriptions, for players without description tracks
script.md timed script for a voice artist
project.json everything (timings, text, evidence), editable

Inputs

Input Default
video required MP4, MOV, MKV, WebM, … with English dialogue
subtitles none optional .srt / .vtt: exact dialogue timing and names (if the form refuses .srt, rename it to .txt)
subtitles_text empty or paste the subtitle text here
cast empty optional, e.g. Lisbeth (young woman, black hair); Mikael (man, glasses): lets the descriptions use names
video_context empty optional, e.g. crime drama film
density standard sparse, standard (as evaluated), film (close to professional film AD), dense
voice af_heart 28 English voices: a American, b British; f female, m male
speed 1.0 narration speed, 0.7-1.4
show_descriptions on draw the description text into the picture
show_captions on draw the dialogue captions into the picture

Subtitles and a cast list give the best results: correct names and exact pauses.

How it works

  1. Speech: Demucs separates the voices from music and effects; Whisper small.en transcribes; voice-activity detection marks every sound of speech; subtitles, if given, give exact timing and names.
  2. Pauses: the gaps between dialogue are found and split at shot cuts.
  3. Seeing: Qwen3-VL-8B looks at frames from each pause and reports what is visible.
  4. Writing: Qwen3-32B writes a short candidate description for each pause, with an importance score.
  5. Choosing (code rules, not the AI): never over dialogue, must fit its pause, important on-screen text first, no repeats, spacing set by the density.
  6. Voice and video: a Kokoro-82M voice reads each description (a little faster if needed to fit), the film is dipped under it, and the text is drawn into the picture.

Speed and hardware

Runs on one GPU (L40S 48 GB or A100). The first request after a cold start waits several minutes while the 32B writer model (~20 GB) downloads; later requests start immediately. As a guide, the pipeline described a 92-second clip in about 2 minutes on an A100; re-encoding the picture adds roughly the video’s length or less.

Limitations and intended use

  • English dialogue and English descriptions only.
  • The descriptions are machine-written: they can miss things, misread a scene or confuse characters. Have a person review them before publishing (project.json and script.md are made for editing). Extended description (pausing the video) is not supported: descriptions only go where there is a pause.
  • Intended for accessibility work, research and drafting AD scripts, not as a replacement for professional describers.
  • Uploaded videos are processed on Replicate’s GPU servers. Only upload videos you have the right to process.

Versions

  • v0.4 (latest): described video (voice + on-screen text), on the v0.3 pipeline.
  • v0.1: text outputs only. Still available under Versions and can be run by its version ID.

Models and licences

Qwen3-VL-8B and Qwen3-32B (Apache-2.0) via Ollama (MIT); Whisper (MIT) via faster-whisper; Demucs (MIT); Kokoro-82M (Apache-2.0) via kokoro-onnx (MIT); DejaVu fonts. Code: GPL-3.0, see the GitHub repository.

Authors: Deep Learning Lab KMUTT

Model created
Model updated