meta/llama-2-70b-chat – Run with an API on Replicate

Official

meta / llama-2-70b-chat

A 70 billion parameter language model from Meta, fine tuned for chat completions

Cold

Public
9.4M runs
Priced per token
GitHub
Paper
License

Run with an API

Playground API Examples README

Input

prompt

*string

Shift + Return to add a new line

Can you write a poem about open source machine learning? Let's make it in the style of E. E. Cummings.Can you write a poem about open source machine learning? Let's make it in the style of E. E. Cummings.

Prompt to send to the model.

system_prompt

string

Shift + Return to add a new line

You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.

System prompt to send to the model. This is prepended to the prompt and helps guide system behavior.

Default: "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."

max_tokens

integer

(minimum: 1)

Maximum number of tokens to generate. A word is generally 2-3 tokens.

Default: 512

min_tokens

integer

(minimum: -1)

Minimum number of tokens to generate. To disable, set to -1. A word is generally 2-3 tokens.

temperature

number

(minimum: 0, maximum: 5)

Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic, 0.75 is a good starting value.

Default: 0.7

top_p

number

(minimum: 0, maximum: 1)

When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens.

Default: 0.95

top_k

integer

(minimum: -1)

When decoding text, samples from the top k most likely tokens; lower to ignore less likely tokens.

Default: 0

stop_sequences

string

Shift + Return to add a new line

A comma-separated list of sequences to stop generation at. For example, '<end>,<stop>' will stop generation at the first instance of 'end' or '<stop>'.

length_penalty

number

(minimum: 0, maximum: 5)

A parameter that controls how long the outputs are. If < 1, the model will tend to generate shorter outputs, and > 1 will tend to generate longer outputs.

Default: 1

presence_penalty

number

A parameter that penalizes repeated tokens regardless of the number of appearances. As the value increases, the model will be less likely to repeat tokens in the output.

Default: 0

seed

integer

Random seed. Leave blank to randomize the seed.

prompt_template

string

Shift + Return to add a new line

<s>[INST] <<SYS>>
{system_prompt}
<</SYS>>

{prompt} [/INST]<s>[INST] <<SYS>>
{system_prompt}
<</SYS>>

{prompt} [/INST]

Template for formatting the prompt. Can be an arbitrary string, but must contain the substring `{prompt}`.

Default: "<s>[INST] <<SYS>>\n{system_prompt}\n<</SYS>>\n\n{prompt} [/INST]"

log_performance_metrics

boolean

Default: false

max_new_tokens

integer

(minimum: 1)

This parameter has been renamed to max_tokens. max_new_tokens only exists for backwards compatibility purposes. We recommend you use max_tokens instead. Both may not be specified.

min_new_tokens

integer

(minimum: -1)

This parameter has been renamed to min_tokens. min_new_tokens only exists for backwards compatibility purposes. We recommend you use min_tokens instead. Both may not be specified.

Run this model in Node.js with one line of code:

npx create-replicate --model=meta/llama-2-70b-chat

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run meta/llama-2-70b-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const input = {
  top_k: 0,
  top_p: 1,
  prompt: "Can you write a poem about open source machine learning? Let's make it in the style of E. E. Cummings.",
  max_tokens: 512,
  temperature: 0.5,
  system_prompt: "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.",
  length_penalty: 1,
  max_new_tokens: 500,
  min_new_tokens: -1,
  prompt_template: "<s>[INST] <<SYS>>\n{system_prompt}\n<</SYS>>\n\n{prompt} [/INST]",
  presence_penalty: 0,
  log_performance_metrics: false
};

for await (const event of replicate.stream("meta/llama-2-70b-chat", { input })) {
  process.stdout.write(event.toString());
};

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run meta/llama-2-70b-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

# The meta/llama-2-70b-chat model can stream output as it's running.
for event in replicate.stream(
    "meta/llama-2-70b-chat",
    input={
        "top_k": 0,
        "top_p": 1,
        "prompt": "Can you write a poem about open source machine learning? Let's make it in the style of E. E. Cummings.",
        "max_tokens": 512,
        "temperature": 0.5,
        "system_prompt": "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.",
        "length_penalty": 1,
        "max_new_tokens": 500,
        "min_new_tokens": -1,
        "prompt_template": "<s>[INST] <<SYS>>\n{system_prompt}\n<</SYS>>\n\n{prompt} [/INST]",
        "presence_penalty": 0,
        "log_performance_metrics": False
    },
):
    print(str(event), end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run meta/llama-2-70b-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "input": {
      "top_k": 0,
      "top_p": 1,
      "prompt": "Can you write a poem about open source machine learning? Let\'s make it in the style of E. E. Cummings.",
      "max_tokens": 512,
      "temperature": 0.5,
      "system_prompt": "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\\n\\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don\'t know the answer to a question, please don\'t share false information.",
      "length_penalty": 1,
      "max_new_tokens": 500,
      "min_new_tokens": -1,
      "prompt_template": "<s>[INST] <<SYS>>\\n{system_prompt}\\n<</SYS>>\\n\\n{prompt} [/INST]",
      "presence_penalty": 0,
      "log_performance_metrics": false
    }
  }' \
  https://api.replicate.com/v1/models/meta/llama-2-70b-chat/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Sure, here's a poem about open source machine learning in the style of E. E. Cummings: open source machine learning a gift to the world, a treasure trove of knowledge, free for all to use and learn from, like a digital muse with code that's open and transparent anyone can see, anyone can tinker and improve upon, a collaborative dance of innovation, without a single anchor the community shares and shapes it together they build and refine a tool that's greater than the sum of its parts a true democracy of mind and design from deep learning to natural language it's all here, a boundless expanse of possibility, a playground for thought where anyone can explore and advance so come and join this movement embrace the power, the beauty, the grace of open source machine learning and watch the future unfold with ease.

{
  "completed_at": "2023-12-12T10:05:07Z",
  "created_at": "2023-12-12T10:05:02.766571Z",
  "data_removed": false,
  "error": "",
  "id": "ssetyuzew3nbe5ndybmon5wmmq",
  "input": {
    "debug": false,
    "top_p": 1,
    "prompt": "Can you write a poem about open source machine learning? Let's make it in the style of E. E. Cummings.",
    "temperature": 0.5,
    "system_prompt": "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.",
    "max_new_tokens": 500,
    "min_new_tokens": -1
  },
  "logs": "MLC is currently not using any LoRAs.\r\nYour formatted prompt is:\r\n[INST] <<SYS>>\r\nYou are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\r\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\r\n<</SYS>>\r\nCan you write a poem about open source machine learning? Let's make it in the style of E. E. Cummings. [/INST]\r\nNot using LoRA\r\nhostname: model-d-c6559c5791b50af57b69f4a73f8e021c-7d68c66789-ntgcq",
  "metrics": {
    "input_token_count": 144,
    "tokens_per_second": 44.006748551816564,
    "output_token_count": 207,
    "predict_time": 4.703926,
    "total_time": 4.233429
  },
  "output": [
    "Sure",
    ",",
    " here",
    "'",
    "s",
    " a",
    " poem",
    " about",
    " open",
    " source",
    " machine",
    " learning",
    " in",
    " the",
    " style",
    " of",
    " E",
    ".",
    " E",
    ".",
    " C",
    "umm",
    "ings",
    ":",
    "\n\nopen",
    " source",
    " machine",
    " learning",
    "\na",
    " gift",
    " to",
    " the",
    " world",
    ",",
    " a",
    " tre",
    "asure",
    " tro",
    "ve",
    "\nof",
    " knowledge",
    ",",
    " free",
    " for",
    " all",
    " to",
    " use",
    "\nand",
    " learn",
    " from",
    ",",
    " like",
    " a",
    " digital",
    " muse",
    "\n\nwith",
    " code",
    " that",
    "'",
    "s",
    " open",
    " and",
    " transparent",
    "\nany",
    "one",
    " can",
    " see",
    ",",
    " anyone",
    " can",
    " t",
    "ink",
    "er",
    "\nand",
    " improve",
    " upon",
    ",",
    " a",
    " collabor",
    "ative",
    " dance",
    "\nof",
    " innov",
    "ation",
    ",",
    " without",
    " a",
    " single",
    " anchor",
    "\n\nthe",
    " community",
    " shares",
    " and",
    " shapes",
    " it",
    "\nt",
    "ogether",
    " they",
    " build",
    " and",
    " ref",
    "ine",
    "\na",
    " tool",
    " that",
    "'",
    "s",
    " greater",
    " than",
    " the",
    " sum",
    " of",
    " its",
    " parts",
    "\na",
    " true",
    " dem",
    "ocracy",
    " of",
    " mind",
    " and",
    " design",
    "\n\nfrom",
    " deep",
    " learning",
    " to",
    " natural",
    " language",
    "\nit",
    "'",
    "s",
    " all",
    " here",
    ",",
    " a",
    " bound",
    "less",
    " exp",
    "anse",
    "\nof",
    " possibility",
    ",",
    " a",
    " play",
    "ground",
    " for",
    " thought",
    "\nwhere",
    " anyone",
    " can",
    " explore",
    " and",
    " advance",
    "\n\nso",
    " come",
    " and",
    " join",
    " this",
    " movement",
    "\nemb",
    "race",
    " the",
    " power",
    ",",
    " the",
    " beauty",
    ",",
    " the",
    " grace",
    "\nof",
    " open",
    " source",
    " machine",
    " learning",
    "\nand",
    " watch",
    " the",
    " future",
    " unfold",
    " with",
    " ease",
    "."
  ],
  "started_at": "2023-12-12T10:05:02Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://streaming-api.svc.europe-west4.g.replicate.net/v1/predictions/ssetyuzew3nbe5ndybmon5wmmq",
    "get": "https://api.replicate.com/v1/predictions/ssetyuzew3nbe5ndybmon5wmmq",
    "cancel": "https://api.replicate.com/v1/predictions/ssetyuzew3nbe5ndybmon5wmmq/cancel"
  },
  "version": "hidden"
}

Generated in

4.7 seconds

Input tokens

140

Output tokens

200

Tokens per second

44.01 tokens / second

Tweak it Report

Pricing

This model is priced by how many input tokens are sent and how many output tokens are generated.

Type	Per unit	Per $1
Input	$0.65 / 1M tokens or 1.5M tokens / $1
Output	$2.75 / 1M tokens or 360K tokens / $1

For example, for $10 you can run around 5,195 predictions where the input is a sentence or two (15 tokens) and the output is a few paragraphs (700 tokens).

Check out our docs for more information about how per-token pricing works on Replicate.

Readme

Llama 2 is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. This is the repository for the 70 billion parameter chat model, which has been fine-tuned on instructions to make it better at being a chat bot.

Learn more about running Llama 2 with an API and the different models.

Please see ai.meta.com/llama for more information about the model, licensing, and acceptable use.

How to prompt Llama 2 chat

To use this model, you can simply pass a prompt or instruction to the prompt argument. We handle prompt formatting on the backend so that you don’t need to worry about it.

Formatting prompts for chat interfaces

However, if you’re managing dialogue state with multiple exchanges between a user and the model, you need to mark the dialogue turns with instruction tags that indicate the beginning ("[INST]") and end (`”/INST]”) of user input. For example, a properly formatted dialogue looks like:

prompt = """\
[INST] Hi! [/INST]
Hello! How are you?
[INST] I'm great, thanks for asking. Could you help me with a task? [/INST]"""

In this example, the hypothetical user has first prompted "Hi!" and received the response "Hello! How are you?". Then, the user responded "I'm great, thanks for asking. Could you help me with a task?".

Modifying the system prompt

In addition to supporting dialogue exchanges, this deployment also allows you to modify the system prompt that is used to guide model responses. By altering the input to the system_prompt argument, you can inject custom context or information that will be used to guide model output.

To learn more, see this guide to prompting Llama 2.