adirik / hierspeechpp

Zero-shot speech synthesizer for text-to-speech and voice conversion

Cold

Public
4.6K runs
L40S
GitHub
Paper
License

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

input_text

string

Shift + Return to add a new line

And lay me down in my cold bed and leave my shining lot.And lay me down in my cold bed and leave my shining lot.

Text input to the model. If provided, it will be used for the speech content of the output.

input_sound

file

Sound input to the model in .wav format. If provided, it will be used for the speech content of the output.

target_voice

*file

Preview

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

A voice clip in .wav format containing the speaker to synthesize.

denoise_ratio

number

(minimum: 0, maximum: 1)

Noise control. 0 means no noise reduction, 1 means maximum noise reduction. If noise reduction is desired, it is recommended to set this value to 0.6~0.8

Default: 0

text_to_vector_temperature

number

(minimum: 0, maximum: 1)

Temperature for text-to-vector model. Larger value corresponds to slightly more random output.

Default: 0.33

voice_conversion_temperature

number

(minimum: 0, maximum: 1)

Temperature for the voice conversion model. Larger value corresponds to slightly more random output.

Default: 0.33

output_sample_rate

integer

Sample rate of the output audio file.

Default: 16000

scale_output_volume

boolean

Scale normalization. If set to true, the output audio will be scaled according to the input sound if provided.

Default: false

seed

integer

Random seed to use for reproducibility.

Run this model in Node.js with one line of code:

npx create-replicate --model=adirik/hierspeechpp

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";
import fs from "node:fs";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run adirik/hierspeechpp using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "adirik/hierspeechpp:ff5bcc71dc2c44662291fc348b9ca2eb40107c9f4b377b169fc0dea950c388c8",
  {
    input: {
      input_text: "And lay me down in my cold bed and leave my shining lot.",
      target_voice: "https://replicate.delivery/pbxt/K30ke0FQUcGCa4gQdyhEPaEeGvEwDZmEK3SMtXaoujJSMlSE/reference_1.wav",
      denoise_ratio: 0,
      output_sample_rate: 16000,
      scale_output_volume: false,
      text_to_vector_temperature: 0.33,
      voice_conversion_temperature: 0.33
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run adirik/hierspeechpp using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "adirik/hierspeechpp:ff5bcc71dc2c44662291fc348b9ca2eb40107c9f4b377b169fc0dea950c388c8",
    input={
        "input_text": "And lay me down in my cold bed and leave my shining lot.",
        "target_voice": "https://replicate.delivery/pbxt/K30ke0FQUcGCa4gQdyhEPaEeGvEwDZmEK3SMtXaoujJSMlSE/reference_1.wav",
        "denoise_ratio": 0,
        "output_sample_rate": 16000,
        "scale_output_volume": False,
        "text_to_vector_temperature": 0.33,
        "voice_conversion_temperature": 0.33
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run adirik/hierspeechpp using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "adirik/hierspeechpp:ff5bcc71dc2c44662291fc348b9ca2eb40107c9f4b377b169fc0dea950c388c8",
    "input": {
      "input_text": "And lay me down in my cold bed and leave my shining lot.",
      "target_voice": "https://replicate.delivery/pbxt/K30ke0FQUcGCa4gQdyhEPaEeGvEwDZmEK3SMtXaoujJSMlSE/reference_1.wav",
      "denoise_ratio": 0,
      "output_sample_rate": 16000,
      "scale_output_volume": false,
      "text_to_vector_temperature": 0.33,
      "voice_conversion_temperature": 0.33
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Video Player is loading.

Current Time 00:00:000

Duration 00:00:000

Loaded: 0%

Stream Type LIVE

Remaining Time 00:00:000

Generated in

3.2 seconds

Tweak itReport View full prediction

This output was created using a different version of the model, adirik/hierspeechpp:4e41ecdd.

Run time and cost

This model costs approximately $0.058 to run on Replicate, or 17 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 60 seconds. The predict time for this model varies significantly based on the inputs.

Readme

HierSpeech++

HierSpeech++ is a text-to-speech model that can generate speech from text and a target voice for zero-shot speech synthesis. See the original repository and paper for details.

API Usage

To use the model, simply provide the text you would like to generate speech and a sound file of your target voice as input. Optionally provide a reference speech (.mp3 or .wav) instead of text to parse speech content. The API returns an .mp3 file with generated speech.

Input parameters are as follows:
- input_text: (optional) text input to the model. If provided, it will be used for the speech content of the output.
- input_sound: (optional) sound input to the model. If provided, it will be used for the speech content of the output.
- target_voice: a voice clip containing the speaker to synthesize.
- denoise_ratio: noise control. 0 means no noise reduction, 1 means maximum noise reduction. If noise reduction is desired, it is recommended to set this value to 0.6~0.8.
- text_to_vector_temperature: temperature for text-to-vector model. Larger value corresponds to slightly more random output.
- output_sample_rate: sample rate of the output audio file.
- scale_output_volume: scale normalization. If set to true, the output audio will be scaled according to the input sound if provided.
- seed: random seed to use for reproducibility.

References

@article{Lee2023HierSpeechBT,
  title={HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis},
  author={Sang-Hoon Lee and Haram Choi and Seung-Bin Kim and Seong-Whan Lee},
  journal={ArXiv},
  year={2023},
  volume={abs/2311.12454},
  url={https://api.semanticscholar.org/CorpusID:265308903}
}