zsxkib / hibiki

Hibiki: High-Fidelity Simultaneous Speech-To-Speech Translation

Cold

Public
12 runs
L40S
GitHub
Weights
Paper
License

Run with an API

Playground API Examples README Versions

Input

audio_input

file

Preview

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

Input audio file to translate

video_input

file

Optional input video file

Including volume_reduction_db and 2 more...

Run this model in Node.js with one line of code:

npx create-replicate --model=zsxkib/hibiki

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run zsxkib/hibiki using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "zsxkib/hibiki:c72042d58e88a0dbcf77942f58b3b033b23f0d02d78cad22c4960afc3b8cce86",
  {
    input: {
      audio_input: "https://replicate.delivery/pbxt/MTZNHE3PVfheTWoSVFVA3vvKxUA0is9pgtRFYnfRnRPgQz9K/sample_fr_hibiki_monologue_otis.mp3",
      max_duration: 0,
      cut_start_seconds: 2,
      volume_reduction_db: 30
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run zsxkib/hibiki using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "zsxkib/hibiki:c72042d58e88a0dbcf77942f58b3b033b23f0d02d78cad22c4960afc3b8cce86",
    input={
        "audio_input": "https://replicate.delivery/pbxt/MTZNHE3PVfheTWoSVFVA3vvKxUA0is9pgtRFYnfRnRPgQz9K/sample_fr_hibiki_monologue_otis.mp3",
        "max_duration": 0,
        "cut_start_seconds": 2,
        "volume_reduction_db": 30
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run zsxkib/hibiki using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "c72042d58e88a0dbcf77942f58b3b033b23f0d02d78cad22c4960afc3b8cce86",
    "input": {
      "audio_input": "https://replicate.delivery/pbxt/MTZNHE3PVfheTWoSVFVA3vvKxUA0is9pgtRFYnfRnRPgQz9K/sample_fr_hibiki_monologue_otis.mp3",
      "max_duration": 0,
      "cut_start_seconds": 2,
      "volume_reduction_db": 30
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/zsxkib/hibiki@sha256:c72042d58e88a0dbcf77942f58b3b033b23f0d02d78cad22c4960afc3b8cce86 \
  -i 'audio_input="https://replicate.delivery/pbxt/MTZNHE3PVfheTWoSVFVA3vvKxUA0is9pgtRFYnfRnRPgQz9K/sample_fr_hibiki_monologue_otis.mp3"' \
  -i 'max_duration=0' \
  -i 'cut_start_seconds=2' \
  -i 'volume_reduction_db=30'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/zsxkib/hibiki@sha256:c72042d58e88a0dbcf77942f58b3b033b23f0d02d78cad22c4960afc3b8cce86
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "audio_input": "https://replicate.delivery/pbxt/MTZNHE3PVfheTWoSVFVA3vvKxUA0is9pgtRFYnfRnRPgQz9K/sample_fr_hibiki_monologue_otis.mp3",
      "max_duration": 0,
      "cut_start_seconds": 2,
      "volume_reduction_db": 30
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

{
  "completed_at": "2025-02-10T17:43:21.702520Z",
  "created_at": "2025-02-10T17:42:15.751000Z",
  "data_removed": false,
  "error": null,
  "id": "d1a2bzv3rxrme0cmy3ta4hx6p8",
  "input": {
    "audio_input": "https://replicate.delivery/pbxt/MTZNHE3PVfheTWoSVFVA3vvKxUA0is9pgtRFYnfRnRPgQz9K/sample_fr_hibiki_monologue_otis.mp3",
    "max_duration": 0,
    "cut_start_seconds": 2,
    "volume_reduction_db": 30
  },
  "logs": "\u001b[1;34m[Info]\u001b[0m retrieving checkpoint\n\u001b[1;34m[Info]\u001b[0m loading mimi\n\u001b[1;34m[Info]\u001b[0m mimi loaded\n\u001b[1;34m[Info]\u001b[0m loading moshi\n\u001b[1;34m[Info]\u001b[0m moshi loaded\n\u001b[1;34m[Info]\u001b[0m loading input file /tmp/tmpd90y11yasample_fr_hibiki_monologue_otis.mp3\nInfo: starting the inference loop\nInfo: processed 795 steps in 17s, 21.72ms/step\n<s> You know, I don't think there are any good or bad situations. For me, if I had to summarize my life today with you, I would say that it is first of all encounters, people who have reached out to me, perhaps at a time when I could not, where I was alone at home. And it is quite curious to think that chance encounters forge a destiny, because when we have the taste of the thing, when we have the taste of the thing done well, the beautiful gesture, sometimes we do not find the interlocutor in front of us. I would say the mirror that helps you move forward. So this is not my case, as I said there. Since I, on the contrary, I was able, and I say thank you to life, I say thank you, I sing life, I dance life, I am only love. And finally, when many people today tell me, but how do it to have this humanity? Well, I answered them very simply. I told them that this taste of love, this taste that pushed me today to undertake a mechanical construction with both hands, who knows? Maybe simply to put myself at the service of the community, to make the gift, the gift of this. That's it I'm</s>\u001b[1;34m[Info]\u001b[0m writing /tmp/tmpd6e64nai/out_en-0 with duration 61.9 sec.\n\u001b[1;34m[Info]\u001b[0m writing /tmp/tmpd6e64nai/out_en-1 with duration 63.6 sec.",
  "metrics": {
    "predict_time": 22.223840168,
    "total_time": 65.95152
  },
  "output": "https://replicate.delivery/xezq/9VfQ7P8yqAXjJSLwv3dixITJsrGLvGjz0jtiBXmghvxcZfNUA/final_mix.wav",
  "started_at": "2025-02-10T17:42:59.478680Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://stream.replicate.com/v1/files/bcwr-6rrdnnqi3v5bqbargngco2cxchjgztnwph4w2w3zptyfuqbdm67a",
    "get": "https://api.replicate.com/v1/predictions/d1a2bzv3rxrme0cmy3ta4hx6p8",
    "cancel": "https://api.replicate.com/v1/predictions/d1a2bzv3rxrme0cmy3ta4hx6p8/cancel"
  },
  "version": "c72042d58e88a0dbcf77942f58b3b033b23f0d02d78cad22c4960afc3b8cce86"
}

Generated in

22.2 seconds

Tweak it ShareReport

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Hibiki: Real-Time Voice-Preserving Translation

[Paper] | [Hear Samples] | [Model Weights]

Hibiki delivers real-time speech translation while preserving the speaker’s voice characteristics. Designed for seamless French→English conversion, it operates locally on consumer hardware with natural-sounding results.

Why Hibiki?

🎭 Voice Preservation - Maintains speaker’s vocal identity using advanced guidance techniques
⚡ Instant Translation - Processes audio at 12.5 frames/sec for real-time conversion
🔊 Natural Output - Generates fluent target speech with human-like prosody
📝 Dual Output - Produces both translated speech and text simultaneously

Quick Translation

Run with Cog using our sample file:

sudo cog predict -i audio_input=@examples/sample_fr_hibiki_crepes.mp3

Use your own .mp3 file for custom translations.

Supported Languages

Currently supports French → English translation. More languages coming soon.

📄 Citation
If using Hibiki in research, please cite our paper.

Model weights licensed under CC-BY 4.0
Inference code MIT licensed

Maintained by @zsxkib for Replicate integration (follow me on X/Twitter for updates)!