top_p: 0.9
prompt: Story title: 3 llamas go for a walk Summary: The 3 llamas crossed a bridge and something unexpected happened Once upon a time
max_tokens: 512
min_tokens: 0
temperature: 0.6
prompt_template: {prompt}
presence_penalty: 1.15
frequency_penalty: 0.2

{
  "top_p": 0.9,
  "prompt": "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time",
  "max_tokens": 512,
  "min_tokens": 0,
  "temperature": 0.6,
  "prompt_template": "{prompt}",
  "presence_penalty": 1.15,
  "frequency_penalty": 0.2
}

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run meta/meta-llama-3-8b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const input = {
  top_p: 0.9,
  prompt: "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time",
  max_tokens: 512,
  min_tokens: 0,
  temperature: 0.6,
  prompt_template: "{prompt}",
  presence_penalty: 1.15
};

for await (const event of replicate.stream("meta/meta-llama-3-8b", { input })) {
  process.stdout.write(event.toString());
};

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run meta/meta-llama-3-8b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

# The meta/meta-llama-3-8b model can stream output as it's running.
for event in replicate.stream(
    "meta/meta-llama-3-8b",
    input={
        "top_p": 0.9,
        "prompt": "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time",
        "max_tokens": 512,
        "min_tokens": 0,
        "temperature": 0.6,
        "prompt_template": "{prompt}",
        "presence_penalty": 1.15
    },
):
    print(str(event), end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run meta/meta-llama-3-8b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "input": {
      "top_p": 0.9,
      "prompt": "Story title: 3 llamas go for a walk\\nSummary: The 3 llamas crossed a bridge and something unexpected happened\\n\\nOnce upon a time",
      "max_tokens": 512,
      "min_tokens": 0,
      "temperature": 0.6,
      "prompt_template": "{prompt}",
      "presence_penalty": 1.15
    }
  }' \
  https://api.replicate.com/v1/models/meta/meta-llama-3-8b/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

there were 3 llamas. They were very friendly and they decided to go for a walk. They crossed a bridge, but something unexpected happened. The first llama was walking on the left side of the bridge, when suddenly he saw a snake. He got scared and ran away as fast as he could. The second llama was walking in the middle of the bridge, when suddenly he saw a dog. He got scared and ran away as fast as he could. The third llama was walking on the right side of the bridge, when suddenly he saw a cat. He got scared and ran away as fast as he could. All 3 llamas ran away from the bridge and never came back. The end.

{
  "completed_at": "2024-04-18T17:21:55Z",
  "created_at": "2024-04-18T17:21:53.234000Z",
  "data_removed": false,
  "error": "",
  "id": "pe0am4n4a9rgp0cey8x9x7zqzr",
  "input": {
    "top_p": 0.9,
    "prompt": "Story title: 3 llamas go for a walk\nSummary: The 3 llamas crossed a bridge and something unexpected happened\n\nOnce upon a time",
    "max_tokens": 512,
    "min_tokens": 0,
    "temperature": 0.6,
    "prompt_template": "{prompt}",
    "presence_penalty": 1.15,
    "frequency_penalty": 0.2
  },
  "logs": "INFO 04-18 17:21:53 async_llm_engine.py:508] Received request 4020a08dfa1e411785b84ec42c00c6c7: prompt: 'Story title: 3 llamas go for a walk\\nSummary: The 3 llamas crossed a bridge and something unexpected happened\\n\\nOnce upon a time', sampling_params: SamplingParams(n=1, best_of=1, presence_penalty=1.15, frequency_penalty=0.2, repetition_penalty=1.0, temperature=0.6, top_p=0.9, top_k=50, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['<|end_of_text|>', '<|eot_id|>'], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=512, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True), prompt_token_ids: None, lora_request: None.\r\n stdoutINFO 04-18 17:21:54 metrics.py:218] Avg prompt throughput: 6.2 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%\r\n stdoutINFO 04-18 17:21:55 async_llm_engine.py:120] Finished request 4020a08dfa1e411785b84ec42c00c6c7.\r\n stdoutTokens/second: 79.92277125263657\r\n stdout",
  "metrics": {
    "total_time": 1.766,
    "input_token_count": 32,
    "tokens_per_second": 75.58125292186872,
    "output_token_count": 141,
    "predict_time": 1.86949
  },
  "output": [
    " there",
    " were",
    " ",
    "3",
    " ll",
    "amas",
    ".",
    " They",
    " were",
    " very",
    " friendly",
    " and",
    " they",
    " decided",
    " to",
    " go",
    " for",
    " a",
    " walk",
    ".\n",
    "They",
    " crossed",
    " a",
    " bridge",
    ",",
    " but",
    " something",
    " unexpected",
    " happened",
    ".\n\n",
    "The",
    " first",
    " llama",
    " was",
    " walking",
    " on",
    " the",
    " left",
    " side",
    " of",
    " the",
    " bridge",
    ",",
    " when",
    " suddenly",
    " he",
    " saw",
    " a",
    " snake",
    ".\n",
    "He",
    " got",
    " scared",
    " and",
    " ran",
    " away",
    " as",
    " fast",
    " as",
    " he",
    " could",
    ".\n\n",
    "The",
    " second",
    " llama",
    " was",
    " walking",
    " in",
    " the",
    " middle",
    " of",
    " the",
    " bridge",
    ",",
    " when",
    " suddenly",
    " he",
    " saw",
    " a",
    " dog",
    ".\n",
    "He",
    " got",
    " scared",
    " and",
    " ran",
    " away",
    " as",
    " fast",
    " as",
    " he",
    " could",
    ".\n\n",
    "The",
    " third",
    " llama",
    " was",
    " walking",
    " on",
    " the",
    " right",
    " side",
    " of",
    " the",
    " bridge",
    ",",
    " when",
    " suddenly",
    " he",
    " saw",
    " a",
    " cat",
    ".\n",
    "He",
    " got",
    " scared",
    " and",
    " ran",
    " away",
    " as",
    " fast",
    " as",
    " he",
    " could",
    ".\n\n",
    "All",
    " ",
    "3",
    " ll",
    "amas",
    " ran",
    " away",
    " from",
    " the",
    " bridge",
    " and",
    " never",
    " came",
    " back",
    ".",
    " The",
    " end",
    ".\n",
    ""
  ],
  "started_at": "2024-04-18T17:21:53Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://streaming-api.svc.us.c.replicate.net/v1/streams/hm77rvogjqvwhjz7e2wbhvp7gzx43n32iuhucornzhr5vsaxhdhq",
    "get": "https://api.replicate.com/v1/predictions/pe0am4n4a9rgp0cey8x9x7zqzr",
    "cancel": "https://api.replicate.com/v1/predictions/pe0am4n4a9rgp0cey8x9x7zqzr/cancel"
  },
  "version": "hidden"
}

Generated in

1.9 seconds

Input tokens

Output tokens

140

Tokens per second

75.58 tokens / second

Tweak it Share

Want to make some of these yourself?

Run this model