pollinations
/
stable-diffusion-dance
Audio Reactive Stable Diffusion
Run pollinations/stable-diffusion-dance with an API
Authentication
Whenever you make an API request, you need to authenticate using a token. A token is like a password that uniquely identifies your account and grants you access.
The following examples all expect your Replicate access token to be available from the command line. Because tokens are secrets, they should not be in your code. They should instead be stored in environment variables. Replicate clients look for the REPLICATE_API_TOKEN environment variable and use it if available.
To set this up you can use:
export REPLICATE_API_TOKEN=<paste-your-token-here>Some application frameworks and tools also support a text file named .env which you can edit to include the same token:
REPLICATE_API_TOKEN=<paste-your-token-here>The Replicate API uses the Authorization HTTP header to authenticate requests. If you’re using a client library this is handled for you.
You can test that your access token is setup correctly by using our account.get endpoint:
curl https://api.replicate.com/v1/account -H "Authorization: Bearer $REPLICATE_API_TOKEN"
# {"type":"user","username":"aron","name":"Aron Carroll","github_url":"https://github.com/aron"}If it is working correctly you will see a JSON object returned containing some information about your account, otherwise ensure that your token is available:
echo "$REPLICATE_API_TOKEN"
# "r8_xyz"Setup
NodeJS supports two module formats ESM and CommonJS. Below details the setup for each environment. After setup, the code is identical regardless of module format.
ESM
First you’ll need to ensure you have a NodeJS project:
npm create esm -yThen install the replicate JavaScript library using npm:
npm install replicateTo use the library, first import and create an instance of it:
import Replicate from "replicate";
const replicate = new Replicate();This will use the REPLICATE_API_TOKEN API token you’ve setup in your environment for authorization.
CommonJS
First you’ll need to ensure you have a NodeJS project:
npm create -yThen install the replicate JavaScript library using npm:
npm install replicateTo use the library, first import and create an instance of it:
const Replicate = require("replicate");
const replicate = new Replicate();This will use the REPLICATE_API_TOKEN API token you’ve setup in your environment for authorization.
Run the model
Use the replicate.run() method to run the model:
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3",
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
const output = await replicate.run("pollinations/stable-diffusion-dance:dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280", { input });
import { writeFile } from "node:fs/promises";
for (const [index, item] of Object.entries(output)) {
await writeFile(`output_${index}.png`, item);
}
//=> output_0.png, output_1.png, output_2.png, output_3.png, o...You can learn about pricing for this model on the model page.
The run() function returns the output directly, which you can then use or pass as the input to another model. If you want to access the full prediction object (not just the output), use the replicate.predictions.create() method instead. This will include the prediction id, status, logs, etc.
File inputs
This model accepts files as input. You can provide a file as input using a URL, a local file on your computer, or a base64 encoded object:
Option 1: Hosted file
Use a URL as in the earlier example:
const audio_file = "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3";This is useful if you already have an image hosted somewhere on the internet.
Option 2: Local file
You can provide Replicate with a Blob, File or Buffer object and the library will handle the upload for you:
import { readFile } from "node:fs/promises";
const audio_file = await readFile("./path/to/my/audio_file.mp3");Option 3: Data URI
You can create a data URI consisting of the base64 encoded data for your file, but this is only recommended if the file is < 1mb
import { readFile } from "node:fs/promises";
const data = (await readFile("./audio_file.mp3")).toString("base64");
const audio_file = `data:application/octet-stream;base64,${data}`;Then, pass audio_file as part of the input:
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: audio_file,
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
const output = await replicate.run("pollinations/stable-diffusion-dance:dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280", { input });
import { writeFile } from "node:fs/promises";
for (const [index, item] of Object.entries(output)) {
await writeFile(`output_${index}.png`, item);
}
//=> output_0.png, output_1.png, output_2.png, output_3.png, o...Streaming
This model supports streaming. This allows you to receive output as the model is running:
const Replicate = require("replicate")
const replicate = new Replicate()
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3",
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
for await (const event of replicate.stream("pollinations/stable-diffusion-dance:dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280", { input })) {
// event: { event: string; data: string; id: string }
process.stdout.write(`${event}
`)
//=> "https://replicate.delivery/pbxt/qeDC7uZ1nwSECK5VFOIOcFRVJgkSRaX6VD3Yye0v4dk0e50gA/00019.png"
};
process.stdout.write("\n");The replicate.stream() method returns a ReadableStream which can be iterated to transform the events into any data structure needed.
For example, to stream just the output content back:
function handler(request) {
const stream = new ReadableStream({
async start(controller) {
for await (const event of replicate.stream( "pollinations/stable-diffusion-dance:dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280", { input })) {
controller.enqueue(new TextEncoder().encode(`${event}`));
//=> "https://replicate.delivery/pbxt/qeDC7uZ1nwSECK5VFOIOcFRVJgkSRaX6VD3Yye0v4dk0e50gA/00019.png"
}
controller.close();
},
});
return new Response(stream);
}Or, stream a list of JSON objects back to the client instead of server sent events:
function handler(request) {
const iterator = replicate.stream( "pollinations/stable-diffusion-dance:dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280", { input });
const stream = new ReadableStream({
async pull(controller) {
const { value, done } = await iterator.next();
const encoder = new TextEncoder();
if (done) {
controller.close();
} else if (value.event === "output" && value.data.length > 0) {
controller.enqueue(encoder.encode(JSON.stringify({ data: value.data }) + "\n"));
} else {
controller.enqueue(encoder.encode(""));
}
},
});
return new Response(stream);
}Streaming in the browser
The JavaScript library is intended to be run on the server. Once the prediction has been created it's output can be streamed directly from the browser.
The streaming URL uses a standard format called Server Sent Events (or text/event-stream) built into all web browsers.
A common implementation is to use a web server to create the prediction using replicate.predictions.create, passing the stream property set to true. Then the urls.stream property of the response contains a URL that can be returned to your frontend application:
// POST /run_prediction
handler(req, res) {
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3",
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
const prediction = await replicate.predictions.create({
version: "dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280",
input,
stream: true,
});
return Response.json({ url: prediction.urls.stream });
// Returns {"url": "https://replicate-stream..."}
}Make a request to the server to create the prediction and use the built-in EventSource object to read the returned url.
const response = await fetch("/run_prediction", { method: "POST" });
const { url } = await response.json();
const source = new EventSource(url);
source.addEventListener("output", (evt) => {
console.log(evt.data) //=> "https://replicate.delivery/pbxt/qeDC7uZ1nwSECK5VFOIOcFRVJgkSRaX6VD3Yye0v4dk0e50gA/00019.png"
});
source.addEventListener("done", (evt) => {
console.log("stream is complete");
});Prediction lifecycle
Running predictions and trainings can often take significant time to complete, beyond what is reasonable for an HTTP request/response.
When you run a model on Replicate, the prediction is created with a “starting” state, then instantly returned. This will then move to "processing" and eventual one of “successful”, "failed" or "canceled".
You can explore the prediction lifecycle by using the predictions.get() method to retrieve the latest version of the prediction until completed.
Show example
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3",
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
const prediction = await replicate.predictions.create({
version: "dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280",
input
});
// { "id": "xyz...", "status": "starting", ... }
const latest = await replicate.predictions.get(prediction.id);
// { "id": "xyz...", "status": "processing", ... }
let completed;
for (let i = 0; i < 5; i++) {
const latest = await replicate.predictions.get(prediction.id);
if (latest.status !== "starting" && latest.status !== "processing") {
completed = latest;
break;
}
// Wait for 2 seconds and then try again.
await new Promise((resolve) => setTimeout(resolve, 2000));
}
console.log(completed.output);
//=> output_0.png, output_1.png, output_2.png, output_3.png, o...Webhooks
Webhooks provide real-time updates about your prediction. Specify an endpoint when you create a prediction, and Replicate will send HTTP POST requests to that URL when the prediction is created, updated, and finished.
It is possible to provide a URL to the predictions.create() function that will be requested by Replicate when the prediction status changes. This is an alternative to polling.
To receive webhooks you’ll need a web server. The following example uses Hono, a web standards based server, but this pattern applies to most frameworks.
Show example
import { serve } from '@hono/node-server';
import { Hono } from 'hono';
const app = new Hono();
app.get('/webhooks/replicate', async (c) => {
// Get the prediction from the request.
const prediction = await c.req.json();
console.log(prediction);
//=> {"id": "xyz", "status": "successful", ... }
// Acknowledge the webhook.
c.status(200);
c.json({ok: true});
}));
serve(app, (info) => {
console.log(`Listening on http://localhost:${info.port}`)
//=> Listening on http://localhost:3000
});Then create the prediction passing in the webhook URL and specify which events you want to receive out of "start", "output", ”logs” and "completed".
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3",
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
const callbackURL = `https://my.app/webhooks/replicate`;
await replicate.predictions.create({
version: "dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280",
input: input,
webhook: callbackURL,
webhook_events_filter: ["completed"],
});
// The server will now handle the event and log:
// => {"id": "xyz", "status": "successful", ... }Co-ordinating between a prediction request and a webhook response will require some glue. A simple implementation for a single JavaScript server could use an event emitter to manage this.
Show example
import { EventEmitter } from "node:events";
const webhooks = new EventEmitter();
// In server code, emit the prediction on the event emitter.
app.get('/webhooks/replicate', async (c) => {
const prediction = await c.req.json();
// Emit the prediction on the EventEmitter.
webhooks.emit(prediction.id, prediction)
// ...
}));
// In request code
await replicate.predictions.create({
model: "yorickvp/llava-13b",
version: "a0fdc44e4f2e1f20f2bb4e27846899953ac8e66c5886c5878fa1d6b73ce009e5",
input: input,
webhook: callbackURL,
webhook_events_filter: ["completed"],
});
// Wait for prediction to be emitted on the EventEmitter.
const prediction = await new Promise(resolve => webhooks.addEventListener(prediction.id, resolve));
// {"id": "xyz", "status": "successful", ... }From a security perspective it is also possible to verify that the webhook came from Replicate. Check out our documentation on verifying webhooks for more information.
Access a prediction
You may wish to access the prediction object. In these cases it’s easier to use the replicate.predictions.create() or replicate.deployments.predictions.create() functions which will return the prediction object.
Though note that these functions will only return the created prediction, and it will not wait for that prediction to be completed before returning. Use replicate.predictions.get() to fetch the latest prediction.
const input = {
width: 512,
prompts: "A painting of a moth\nA painting of a killer dragonfly\nTwo fishes talking to each other in deep sea\nA painting of a moth",
audio_file: "https://replicate.delivery/pbxt/IEUhQkLT4CaUs08buoPAyXgVlvf0157xNiHcyukEfDDfWxB5/ganze_boot.mp3",
batch_size: 20,
style_suffix: "by paul klee, intricate detail",
audio_smoothing: 0.85,
diffusion_steps: 15
};
const prediction = replicate.predictions.create({
version: "dfb636aa9c04fe5b7d9897e6159ef88e3ecb3e1eb274c3f072dca7b495823280",
input
});
// { "id": "xyz123", "status": "starting", ... }Cancel a prediction
You may need to cancel a prediction. Perhaps the user has navigated away from the browser or canceled your application. To prevent unnecessary work and reduce runtime costs you can use the replicate.predictions.cancel function and pass it a prediction id.
await replicate.predictions.cancel(prediction.id);