collectiveai-team / crisperwhisper

Unofficial implementation of Verbatim Automatic Speech Recognition with improved word-level timestamps and filler detection

Cold

Public
2.3K runs
A100 (80GB)
Paper

Run with an API

Playground API Examples README Versions

Input

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

hf_token

*string

Shift + Return to add a new line

HuggingFace token to access CrispWhisper models.

audio

file

Audio file

Default: "https://replicate.delivery/pbxt/IZjTvet2ZGiyiYaMEEPrzn0xY1UDNsh0NfcO9qeTlpwCo7ig/lex-levin-4min.mp3"

language

string

Language spoken in the audio, specify 'None' to perform language detection.

Default: "None"

batch_size

integer

Number of parallel batches you want to compute. Reduce if you face OOMs.

Default: 8

Run this model in Node.js with one line of code:

npx create-replicate --model=collectiveai-team/crisperwhisper

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run collectiveai-team/crisperwhisper using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "collectiveai-team/crisperwhisper:7296cb091e24d7ff0e774bd289f6a7653e7079661e5c42c8a15d2bfb37fc24a1",
  {
    input: {
      audio: "https://replicate.delivery/pbxt/IZjTvet2ZGiyiYaMEEPrzn0xY1UDNsh0NfcO9qeTlpwCo7ig/lex-levin-4min.mp3",
      language: "None",
      batch_size: 8
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run collectiveai-team/crisperwhisper using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "collectiveai-team/crisperwhisper:7296cb091e24d7ff0e774bd289f6a7653e7079661e5c42c8a15d2bfb37fc24a1",
    input={
        "audio": "https://replicate.delivery/pbxt/IZjTvet2ZGiyiYaMEEPrzn0xY1UDNsh0NfcO9qeTlpwCo7ig/lex-levin-4min.mp3",
        "language": "None",
        "batch_size": 8
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run collectiveai-team/crisperwhisper using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "collectiveai-team/crisperwhisper:7296cb091e24d7ff0e774bd289f6a7653e7079661e5c42c8a15d2bfb37fc24a1",
    "input": {
      "audio": "https://replicate.delivery/pbxt/IZjTvet2ZGiyiYaMEEPrzn0xY1UDNsh0NfcO9qeTlpwCo7ig/lex-levin-4min.mp3",
      "language": "None",
      "batch_size": 8
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

No output yet! Press "Submit" to start a prediction.

Run time and cost

This model costs approximately $0.73 to run on Replicate, or 1 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 9 minutes. The predict time for this model varies significantly based on the inputs.

Readme

CrisperWhisper

CrisperWhisper is an advanced variant of OpenAI’s Whisper, designed for fast, precise, and verbatim speech recognition with accurate (crisp) word-level timestamps. Unlike the original Whisper, which tends to omit disfluencies and follows more of a intended transcription style, CrisperWhisper aims to transcribe every spoken word exactly as it is, including fillers, pauses, stutters and false starts.

Key Features

🎯 Accurate Word-Level Timestamps: Provides precise timestamps, even around disfluencies and pauses, by utilizing an adjusted tokenizer and a custom attention loss during training.
📝 Verbatim Transcription: Transcribes every spoken word exactly as it is, including and differentiating fillers like “um” and “uh”.
🔍 Filler Detection: Detects and accurately transcribes fillers.
🛡️ Hallucination Mitigation: Minimizes transcription hallucinations to enhance accuracy.