Run Code Llama 70B with an API

Posted by @cbh123

Code Llama is a code generation model built on top of Llama 2. It can generate code and natural language about code in many programming languages, including Python, JavaScript, TypeScript, C++, Java, PHP, C#, Bash and more.

Today, Meta announced a more powerful new version of Code Llama with 70 billion parameters. It’s one of the highest performing open models. Meta reports a 67.8 on HumanEval, which beats zero-shot GPT-4.

With Replicate, you can run Code Llama 70B in the cloud with one line of code.

Contents

Code Llama 70B variants

There are three variants of Code Llama 70B. The code snippets in this guide use codellama-70b-instruct, but all three variants are available on Replicate:

Run Code Llama 70B with JavaScript

You can run Code Llama 70B with our official JavaScript client:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<your-api-token>

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run meta/codellama-70b-instruct using Replicate’s API:

const output = await replicate.run(
  "meta/codellama-70b-instruct:a279116fe47a0f65701a8817188601e2fe8f4b9e04a518789655ea7b995851bf",
  input: {
    prompt:"In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?",
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Run Code Llama 70B with Python

You can run Code Llama 70B with our official Python client:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<your-api-token>

Run meta/codellama-70b-instruct using Replicate’s API:

import replicate

output = replicate.run(
    "meta/codellama-70b-instruct:a279116fe47a0f65701a8817188601e2fe8f4b9e04a518789655ea7b995851bf",
    input={
        "prompt": "In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?",
      }
)
print("".join(output))

To learn more, take a look at the guide on getting started with Python.

Run Code Llama 70B with cURL

You can call the HTTP API directly with tools like cURL:

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<your-api-token>

Run meta/codellama-70b-instruct using Replicate’s API:

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d $'{
    "version": "a279116fe47a0f65701a8817188601e2fe8f4b9e04a518789655ea7b995851bf",
    "input": {
      "top_k": 10,
      "top_p": 0.95,
      "prompt": "In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?",
      "max_tokens": 500,
      "temperature": 0.8,
      "system_prompt": "",
      "repeat_penalty": 1.1,
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can also run Code Llama 70B using other Replicate client libraries for Go, Swift, and others

Keep up to speed