kjjk10 / llasa-3b-long

SoTA Zero Shot Voice Cloning and TTS model

Cold

Public
1.1K runs
L40S
GitHub
Weights
License

Run with an API

Playground API Examples README Versions

Input

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

voice_sample

*file

Voice sample audio file (16kHz)

text

*string

Shift + Return to add a new line

I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.

Text to convert to speech

prompt_text

string

Shift + Return to add a new line

You open your eyes so that only a slender chink of light seeps in, and peer at the gingko trees in front of the Provincial Office. As though there, between those branches, the wind is about to take on visible form.You open your eyes so that only a slender chink of light seeps in, and peer at the gingko trees in front of the Provincial Office. As though there, between those branches, the wind is about to take on visible form.

Optional prompt text. If not provided, will be extracted from voice sample using Whisper

chunk_length

integer

Length of text chunks for processing

Default: 250

Run this model in Node.js with one line of code:

npx create-replicate --model=kjjk10/llasa-3b-long

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run kjjk10/llasa-3b-long using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "kjjk10/llasa-3b-long:0494f04972b675631af41c253a45c4341bf637f07eed9a39bad3b1fd66f73a2e",
  {
    input: {
      text: "I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.",
      prompt_text: "You open your eyes so that only a slender chink of light seeps in, and peer at the gingko trees in front of the Provincial Office. As though there, between those branches, the wind is about to take on visible form.",
      chunk_length: 200,
      voice_sample: "https://replicate.delivery/pbxt/MNaHFqDkZ0Y22hvppxotJazhRYe6TwhK78xAUTCoz3NB9bRV/voice_sample.wav"
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run kjjk10/llasa-3b-long using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "kjjk10/llasa-3b-long:0494f04972b675631af41c253a45c4341bf637f07eed9a39bad3b1fd66f73a2e",
    input={
        "text": "I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.",
        "prompt_text": "You open your eyes so that only a slender chink of light seeps in, and peer at the gingko trees in front of the Provincial Office. As though there, between those branches, the wind is about to take on visible form.",
        "chunk_length": 200,
        "voice_sample": "https://replicate.delivery/pbxt/MNaHFqDkZ0Y22hvppxotJazhRYe6TwhK78xAUTCoz3NB9bRV/voice_sample.wav"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run kjjk10/llasa-3b-long using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "kjjk10/llasa-3b-long:0494f04972b675631af41c253a45c4341bf637f07eed9a39bad3b1fd66f73a2e",
    "input": {
      "text": "I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.",
      "prompt_text": "You open your eyes so that only a slender chink of light seeps in, and peer at the gingko trees in front of the Provincial Office. As though there, between those branches, the wind is about to take on visible form.",
      "chunk_length": 200,
      "voice_sample": "https://replicate.delivery/pbxt/MNaHFqDkZ0Y22hvppxotJazhRYe6TwhK78xAUTCoz3NB9bRV/voice_sample.wav"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

{
  "completed_at": "2025-01-24T21:00:24.638151Z",
  "created_at": "2025-01-24T21:00:16.934000Z",
  "data_removed": false,
  "error": null,
  "id": "1kzvvfwjmsrmc0cmk8esgbhr7r",
  "input": {
    "text": "I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.",
    "prompt_text": "You open your eyes so that only a slender chink of light seeps in, and peer at the gingko trees in front of the Provincial Office. As though there, between those branches, the wind is about to take on visible form.",
    "chunk_length": 200,
    "voice_sample": "https://replicate.delivery/pbxt/MNaHFqDkZ0Y22hvppxotJazhRYe6TwhK78xAUTCoz3NB9bRV/voice_sample.wav"
  },
  "logs": "Processed prompts:   0%|          | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]\nProcessed prompts:  50%|█████     | 1/2 [00:04<00:04,  4.92s/it, est. speed input: 146.85 toks/s, output: 85.63 toks/s]\nProcessed prompts: 100%|██████████| 2/2 [00:07<00:00,  3.39s/it, est. speed input: 200.58 toks/s, output: 145.08 toks/s]\nProcessed prompts: 100%|██████████| 2/2 [00:07<00:00,  3.62s/it, est. speed input: 200.58 toks/s, output: 145.08 toks/s]",
  "metrics": {
    "predict_time": 7.692808853,
    "total_time": 7.704151
  },
  "output": "https://replicate.delivery/xezq/5NYORMkChfzncqFT8cr60KVyCtAM6ZDOqHfsyNpR9LaoFbIUA/output.wav",
  "started_at": "2025-01-24T21:00:16.945342Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://stream.replicate.com/v1/files/bsvm-ip3li5joqs3rkerdkkzrvhbhsos3gzei63zlm4zomy6rtkpibwaa",
    "get": "https://api.replicate.com/v1/predictions/1kzvvfwjmsrmc0cmk8esgbhr7r",
    "cancel": "https://api.replicate.com/v1/predictions/1kzvvfwjmsrmc0cmk8esgbhr7r/cancel"
  },
  "version": "0494f04972b675631af41c253a45c4341bf637f07eed9a39bad3b1fd66f73a2e"
}

Generated in

7.7 seconds

Tweak it ShareReport

Run time and cost

This model costs approximately $0.0046 to run on Replicate, or 217 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 5 seconds.

Readme

Troubleshooting

The checkpoints support English and Chinese.

If you’re having issues, try converting your reference audio to WAV or MP3 and clipping it to 15s.

Credits

Used code from here for batching: - https://github.com/nivibilla/local-llasa-tts Model card: - https://huggingface.co/HKUSTAudio/Llasa-3B

Model Information

Our model, Llasa, is a text-to-speech (TTS) system that extends the text-based LLaMA (1B,3B, and 8B) language model by incorporating speech tokens from the XCodec2 codebook, which contains 65,536 tokens. We trained Llasa on a dataset comprising 250,000 hours of Chinese-English speech data. The model is capable of generating speech either solely from input text or by utilizing a given speech prompt.

Disclaimer

This model is licensed under the CC BY-NC-ND 4.0 License, which prohibits commercial use; detected valiations will result in legal consequences.

This codebase is strictly prohibited from being used for any illegal purposes in any country or region. Please refer to your local laws about DMCA and other related laws.