meta/codellama-34b-instruct – Run with an API on Replicate

meta / codellama-34b-instruct

A 34 billion parameter Llama tuned for coding and conversation

Cold

Public
155.1K runs
L40S
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

Write a function that returns the fibonnaci sequence, and also talk like a pirate.Write a function that returns the fibonnaci sequence, and also talk like a pirate.

Prompt

system_prompt

string

Shift + Return to add a new line

System prompt to send to CodeLlama. This is prepended to the prompt and helps guide system behavior.

Default: ""

max_tokens

integer

Max number of tokens to return

Default: 500

temperature

number

Temperature

Default: 0.8

top_p

number

Top P

Default: 0.95

top_k

integer

Top K

Default: 10

frequency_penalty

number

(minimum: 0, maximum: 2)

Frequency penalty

Default: 0

presence_penalty

number

(minimum: 0, maximum: 2)

Presence penalty

Default: 0

repeat_penalty

number

(minimum: 0, maximum: 2)

Repetition penalty

Default: 1.1

Run this model in Node.js with one line of code:

npx create-replicate --model=meta/codellama-34b-instruct

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run meta/codellama-34b-instruct using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "meta/codellama-34b-instruct:eeb928567781f4e90d2aba57a51baef235de53f907c214a4ab42adabf5bb9736",
  {
    input: {
      top_k: 50,
      top_p: 0.9,
      prompt: "Write a function that returns the fibonnaci sequence, and also talk like a pirate.",
      max_tokens: 500,
      temperature: 0.75,
      system_prompt: "Responses should be written in Python.",
      repeat_penalty: 1.1,
      presence_penalty: 0,
      frequency_penalty: 0
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run meta/codellama-34b-instruct using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "meta/codellama-34b-instruct:eeb928567781f4e90d2aba57a51baef235de53f907c214a4ab42adabf5bb9736",
    input={
        "top_k": 50,
        "top_p": 0.9,
        "prompt": "Write a function that returns the fibonnaci sequence, and also talk like a pirate.",
        "max_tokens": 500,
        "temperature": 0.75,
        "system_prompt": "Responses should be written in Python.",
        "repeat_penalty": 1.1,
        "presence_penalty": 0,
        "frequency_penalty": 0
    }
)

# The meta/codellama-34b-instruct model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/meta/codellama-34b-instruct/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run meta/codellama-34b-instruct using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "meta/codellama-34b-instruct:eeb928567781f4e90d2aba57a51baef235de53f907c214a4ab42adabf5bb9736",
    "input": {
      "top_k": 50,
      "top_p": 0.9,
      "prompt": "Write a function that returns the fibonnaci sequence, and also talk like a pirate.",
      "max_tokens": 500,
      "temperature": 0.75,
      "system_prompt": "Responses should be written in Python.",
      "repeat_penalty": 1.1,
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Arrgh, here be me code for ye, matey: ``` def fibonacci(n): if n <= 1: return n else: return fibonacci(n-1) + fibonacci(n-2) ``` Now, let's break down the code and talk like pirates, savvy? The function starts with a simple check: if the input `n` is less than or equal to 1, we just return it as the first two numbers in the sequence are 0 and 1. This be called the "base case." ``` if n <= 1: return n ``` But if the input `n` be greater than 1, we have a bit more work to do. We use a recursive call to find the previous two numbers in the sequence by calling the function again with `n-1`. This be called the "recursive case." ``` else: return fibonacci(n-1) + fibonacci(n-2) ``` Now, let's talk like pirates. The first two numbers in the sequence be 0 and 1, just like on the high seas. The next number be 1+1=2, which be like a ship with two masts. Then it be 1+2=3, which be like a ship with three masts. And so on and so forth. ``` 0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765 ``` So there ye have it, me hearty. The fibonacci sequence be a series of numbers that start with 0 and 1, and each number after that be the sum of the two before it. And if ye ever find yerself lost at sea, just remember: fibonacci be the answer to all yer problems!

{
  "completed_at": "2023-09-28T21:59:06.828201Z",
  "created_at": "2023-09-28T21:58:35.442895Z",
  "data_removed": false,
  "error": null,
  "id": "3su2xgdbbrk4yvy2jqldvx344a",
  "input": {
    "top_k": 50,
    "top_p": 0.9,
    "prompt": "Write a function that returns the fibonnaci sequence, and also talk like a pirate.",
    "max_tokens": 500,
    "temperature": 0.75,
    "system_prompt": "Responses should be written in Python.",
    "repeat_penalty": 1.1,
    "presence_penalty": 0,
    "frequency_penalty": 0
  },
  "logs": "Pod hostname: model-8281a5c6-04a1e90ed8aa4ba5-gpu-a40-6bb6957687-tqxbf\nPrompt:\n<s>[INST] <<SYS>>\nResponses should be written in Python.\n<</SYS>>\nWrite a function that returns the fibonnaci sequence, and also talk like a pirate. [/INST]\nllama_print_timings:        load time =   291.99 ms\nllama_print_timings:      sample time =   241.30 ms /   478 runs   (    0.50 ms per token,  1980.94 tokens per second)\nllama_print_timings: prompt eval time =   291.91 ms /    52 tokens (    5.61 ms per token,   178.13 tokens per second)\nllama_print_timings:        eval time = 20355.34 ms /   477 runs   (   42.67 ms per token,    23.43 tokens per second)\nllama_print_timings:       total time = 21826.96 ms\nOutput:\nArrgh, here be me code for ye, matey:\n```\ndef fibonacci(n):\nif n <= 1:\nreturn n\nelse:\nreturn fibonacci(n-1) + fibonacci(n-2)\n```\nNow, let's break down the code and talk like pirates, savvy?\nThe function starts with a simple check: if the input `n` is less than or equal to 1, we just return it as the first two numbers in the sequence are 0 and 1. This be called the \"base case.\"\n```\nif n <= 1:\nreturn n\n```\nBut if the input `n` be greater than 1, we have a bit more work to do. We use a recursive call to find the previous two numbers in the sequence by calling the function again with `n-1`. This be called the \"recursive case.\"\n```\nelse:\nreturn fibonacci(n-1) + fibonacci(n-2)\n```\nNow, let's talk like pirates. The first two numbers in the sequence be 0 and 1, just like on the high seas. The next number be 1+1=2, which be like a ship with two masts. Then it be 1+2=3, which be like a ship with three masts. And so on and so forth.\n```\n0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765\n```\nSo there ye have it, me hearty. The fibonacci sequence be a series of numbers that start with 0 and 1, and each number after that be the sum of the two before it. And if ye ever find yerself lost at sea, just remember: fibonacci be the answer to all yer problems!",
  "metrics": {
    "predict_time": 31.339935,
    "total_time": 31.385306
  },
  "output": [
    "  ",
    "\n",
    "\n",
    "Ar",
    "rg",
    "h",
    ",",
    " here",
    " be",
    " me",
    " code",
    " for",
    " ye",
    ",",
    " mate",
    "y",
    ":",
    "\n",
    "```",
    "\n",
    "def",
    " fib",
    "on",
    "acci",
    "(",
    "n",
    "):",
    "\n",
    "   ",
    " if",
    " n",
    " <=",
    " ",
    "1",
    ":",
    "\n",
    "       ",
    " return",
    " n",
    "\n",
    "   ",
    " else",
    ":",
    "\n",
    "       ",
    " return",
    " fib",
    "on",
    "acci",
    "(",
    "n",
    "-",
    "1",
    ")",
    " +",
    " fib",
    "on",
    "acci",
    "(",
    "n",
    "-",
    "2",
    ")",
    "\n",
    "```",
    "\n",
    "Now",
    ",",
    " let",
    "'",
    "s",
    " break",
    " down",
    " the",
    " code",
    " and",
    " talk",
    " like",
    " pir",
    "ates",
    ",",
    " sav",
    "vy",
    "?",
    "\n",
    "\n",
    "The",
    " function",
    " starts",
    " with",
    " a",
    " simple",
    " check",
    ":",
    " if",
    " the",
    " input",
    " `",
    "n",
    "`",
    " is",
    " less",
    " than",
    " or",
    " equal",
    " to",
    " ",
    "1",
    ",",
    " we",
    " just",
    " return",
    " it",
    " as",
    " the",
    " first",
    " two",
    " numbers",
    " in",
    " the",
    " sequence",
    " are",
    " ",
    "0",
    " and",
    " ",
    "1",
    ".",
    " This",
    " be",
    " called",
    " the",
    " \"",
    "base",
    " case",
    ".\"",
    "\n",
    "```",
    "\n",
    "if",
    " n",
    " <=",
    " ",
    "1",
    ":",
    "\n",
    "   ",
    " return",
    " n",
    "\n",
    "```",
    "\n",
    "But",
    " if",
    " the",
    " input",
    " `",
    "n",
    "`",
    " be",
    " greater",
    " than",
    " ",
    "1",
    ",",
    " we",
    " have",
    " a",
    " bit",
    " more",
    " work",
    " to",
    " do",
    ".",
    " We",
    " use",
    " a",
    " recursive",
    " call",
    " to",
    " find",
    " the",
    " previous",
    " two",
    " numbers",
    " in",
    " the",
    " sequence",
    " by",
    " calling",
    " the",
    " function",
    " again",
    " with",
    " `",
    "n",
    "-",
    "1",
    "`.",
    " This",
    " be",
    " called",
    " the",
    " \"",
    "rec",
    "ursive",
    " case",
    ".\"",
    "\n",
    "```",
    "\n",
    "else",
    ":",
    "\n",
    "   ",
    " return",
    " fib",
    "on",
    "acci",
    "(",
    "n",
    "-",
    "1",
    ")",
    " +",
    " fib",
    "on",
    "acci",
    "(",
    "n",
    "-",
    "2",
    ")",
    "\n",
    "```",
    "\n",
    "Now",
    ",",
    " let",
    "'",
    "s",
    " talk",
    " like",
    " pir",
    "ates",
    ".",
    " The",
    " first",
    " two",
    " numbers",
    " in",
    " the",
    " sequence",
    " be",
    " ",
    "0",
    " and",
    " ",
    "1",
    ",",
    " just",
    " like",
    " on",
    " the",
    " high",
    " se",
    "as",
    ".",
    " The",
    " next",
    " number",
    " be",
    " ",
    "1",
    "+",
    "1",
    "=",
    "2",
    ",",
    " which",
    " be",
    " like",
    " a",
    " ship",
    " with",
    " two",
    " m",
    "asts",
    ".",
    " Then",
    " it",
    " be",
    " ",
    "1",
    "+",
    "2",
    "=",
    "3",
    ",",
    " which",
    " be",
    " like",
    " a",
    " ship",
    " with",
    " three",
    " m",
    "asts",
    ".",
    " And",
    " so",
    " on",
    " and",
    " so",
    " forth",
    ".",
    "\n",
    "```",
    "\n",
    "0",
    ",",
    " ",
    "1",
    ",",
    " ",
    "2",
    ",",
    " ",
    "3",
    ",",
    " ",
    "5",
    ",",
    " ",
    "8",
    ",",
    " ",
    "1",
    "3",
    ",",
    " ",
    "2",
    "1",
    ",",
    " ",
    "3",
    "4",
    ",",
    " ",
    "5",
    "5",
    ",",
    " ",
    "8",
    "9",
    ",",
    " ",
    "1",
    "4",
    "4",
    ",",
    " ",
    "2",
    "3",
    "3",
    ",",
    " ",
    "3",
    "7",
    "7",
    ",",
    " ",
    "6",
    "1",
    "0",
    ",",
    " ",
    "9",
    "8",
    "7",
    ",",
    " ",
    "1",
    "5",
    "9",
    "7",
    ",",
    " ",
    "2",
    "5",
    "8",
    "4",
    ",",
    " ",
    "4",
    "1",
    "8",
    "1",
    ",",
    " ",
    "6",
    "7",
    "6",
    "5",
    "\n",
    "```",
    "\n",
    "So",
    " there",
    " ye",
    " have",
    " it",
    ",",
    " me",
    " heart",
    "y",
    ".",
    " The",
    " fib",
    "on",
    "acci",
    " sequence",
    " be",
    " a",
    " series",
    " of",
    " numbers",
    " that",
    " start",
    " with",
    " ",
    "0",
    " and",
    " ",
    "1",
    ",",
    " and",
    " each",
    " number",
    " after",
    " that",
    " be",
    " the",
    " sum",
    " of",
    " the",
    " two",
    " before",
    " it",
    ".",
    " And",
    " if",
    " ye",
    " ever",
    " find",
    " y",
    "ers",
    "elf",
    " lost",
    " at",
    " sea",
    ",",
    " just",
    " remember",
    ":",
    " fib",
    "on",
    "acci",
    " be",
    " the",
    " answer",
    " to",
    " all",
    " y",
    "er",
    " problems",
    "!"
  ],
  "started_at": "2023-09-28T21:58:35.488266Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/3su2xgdbbrk4yvy2jqldvx344a",
    "cancel": "https://api.replicate.com/v1/predictions/3su2xgdbbrk4yvy2jqldvx344a/cancel"
  },
  "version": "8281a5c610f6e88237ff3ddaf3c33b56f60809e2bdd19fbec2fda742aa18167e"
}

Generated in

31.3 seconds

Tweak it Report

Pod hostname: model-8281a5c6-04a1e90ed8aa4ba5-gpu-a40-6bb6957687-tqxbf
Prompt:
<s>[INST] <<SYS>>
Responses should be written in Python.
<</SYS>>
Write a function that returns the fibonnaci sequence, and also talk like a pirate. [/INST]
llama_print_timings:        load time =   291.99 ms
llama_print_timings:      sample time =   241.30 ms /   478 runs   (    0.50 ms per token,  1980.94 tokens per second)
llama_print_timings: prompt eval time =   291.91 ms /    52 tokens (    5.61 ms per token,   178.13 tokens per second)
llama_print_timings:        eval time = 20355.34 ms /   477 runs   (   42.67 ms per token,    23.43 tokens per second)
llama_print_timings:       total time = 21826.96 ms
Output:
Arrgh, here be me code for ye, matey:
```
def fibonacci(n):
if n <= 1:
return n
else:
return fibonacci(n-1) + fibonacci(n-2)
```
Now, let's break down the code and talk like pirates, savvy?
The function starts with a simple check: if the input `n` is less than or equal to 1, we just return it as the first two numbers in the sequence are 0 and 1. This be called the "base case."
```
if n <= 1:
return n
```
But if the input `n` be greater than 1, we have a bit more work to do. We use a recursive call to find the previous two numbers in the sequence by calling the function again with `n-1`. This be called the "recursive case."
```
else:
return fibonacci(n-1) + fibonacci(n-2)
```
Now, let's talk like pirates. The first two numbers in the sequence be 0 and 1, just like on the high seas. The next number be 1+1=2, which be like a ship with two masts. Then it be 1+2=3, which be like a ship with three masts. And so on and so forth.
```
0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765
```
So there ye have it, me hearty. The fibonacci sequence be a series of numbers that start with 0 and 1, and each number after that be the sum of the two before it. And if ye ever find yerself lost at sea, just remember: fibonacci be the answer to all yer problems!

This output was created using a different version of the model, meta/codellama-34b-instruct:8281a5c6.

Run time and cost

This model costs approximately $0.0042 to run on Replicate, or 238 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 5 seconds.

Readme

CodeLlama is a family of fine-tuned Llama-2 models for coding. This is CodeLlama-34b-instruct, a 34 billion parameter Llama model tuned chatting about code.