replicate/vllm:6cd734b2 | Run with an API on Replicate

Version

You're looking at a specific version of this model. Jump to the model overview.

replicate /vllm:6cd734b2

Playground API

Input

prompt

string

Shift + Return to add a new line

Prompt

Default: ""

system_prompt

string

Shift + Return to add a new line

System prompt to send to the model. This is prepended to the prompt and helps guide system behavior. Ignored for non-chat models.

Default: "You are a helpful assistant."

min_tokens

integer

The minimum number of tokens the model should generate as output.

Default: 0

max_tokens

integer

The maximum number of tokens the model should generate as output.

Default: 512

temperature

number

The value used to modulate the next token probabilities.

Default: 0.6

top_p

number

A probability threshold for generating the output. If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751).

Default: 0.9

top_k

integer

The number of highest probability tokens to consider for generating the output. If > 0, only keep the top k tokens with highest probability (top-k filtering).

Default: 50

presence_penalty

number

Presence penalty

Default: 0

frequency_penalty

number

Frequency penalty

Default: 0

stop_sequences

string

Shift + Return to add a new line

A comma-separated list of sequences to stop generation at. For example, '<end>,<stop>' will stop generation at the first instance of 'end' or '<stop>'.

prompt_template

string

Shift + Return to add a new line

A template to format the prompt with. If not provided, the default prompt template will be used.

seed

integer

Random seed. Leave blank to randomize the seed.

Run this model in Node.js with one line of code:

npx create-replicate --model=replicate/vllm

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run replicate/vllm using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "replicate/vllm:6cd734b2b0b832294919954069bfdc2b1d435bf98376f8e454f420a3f8adc23e",
  {
    input: {
      top_k: 50,
      top_p: 0.9,
      prompt: "",
      max_tokens: 512,
      min_tokens: 0,
      temperature: 0.6,
      system_prompt: "You are a helpful assistant.",
      presence_penalty: 0,
      frequency_penalty: 0
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run replicate/vllm using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "replicate/vllm:6cd734b2b0b832294919954069bfdc2b1d435bf98376f8e454f420a3f8adc23e",
    input={
        "top_k": 50,
        "top_p": 0.9,
        "prompt": "",
        "max_tokens": 512,
        "min_tokens": 0,
        "temperature": 0.6,
        "system_prompt": "You are a helpful assistant.",
        "presence_penalty": 0,
        "frequency_penalty": 0
    }
)

# The replicate/vllm model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/replicate/vllm/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run replicate/vllm using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "replicate/vllm:6cd734b2b0b832294919954069bfdc2b1d435bf98376f8e454f420a3f8adc23e",
    "input": {
      "top_k": 50,
      "top_p": 0.9,
      "prompt": "",
      "max_tokens": 512,
      "min_tokens": 0,
      "temperature": 0.6,
      "system_prompt": "You are a helpful assistant.",
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

No output yet! Press "Submit" to start a prediction.