nateraw/goliath-120b – Run with an API on Replicate

nateraw / goliath-120b

An auto-regressive causal LM created by combining 2x finetuned Llama-2 70B into one.

Cold

Public
237.5K runs
A100 (80GB)

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. 

max_new_tokens

integer

The maximum number of tokens the model should generate as output.

Default: 512

temperature

number

The value used to modulate the next token probabilities.

Default: 0.8

top_p

number

A probability threshold for generating the output. If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751).

Default: 0.95

top_k

integer

The number of highest probability tokens to consider for generating the output. If > 0, only keep the top k tokens with highest probability (top-k filtering).

Default: 50

presence_penalty

number

Presence penalty

Default: 0

frequency_penalty

number

Frequency penalty

Default: 0

prompt_template

string

Shift + Return to add a new line

You are a helpful AI assistant.

USER: {prompt}
ASSISTANT:You are a helpful AI assistant.

USER: {prompt}
ASSISTANT:

The template used to format the prompt. The input prompt is inserted into the template using the `{prompt}` placeholder.

Default: "You are a helpful AI assistant.\n\nUSER: {prompt}\nASSISTANT:"

Run this model in Node.js with one line of code:

npx create-replicate --model=nateraw/goliath-120b

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run nateraw/goliath-120b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "nateraw/goliath-120b:39c0be5d2e96df975da94e8923535733d81179abed5f37c999afc5f7c33bf04e",
  {
    input: {
      top_k: 50,
      top_p: 0.95,
      prompt: "write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. ",
      temperature: 0.8,
      max_new_tokens: 2048,
      prompt_template: "You are a helpful AI assistant.\n\nUSER: {prompt}\nASSISTANT:",
      presence_penalty: 1,
      frequency_penalty: 0
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run nateraw/goliath-120b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "nateraw/goliath-120b:39c0be5d2e96df975da94e8923535733d81179abed5f37c999afc5f7c33bf04e",
    input={
        "top_k": 50,
        "top_p": 0.95,
        "prompt": "write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. ",
        "temperature": 0.8,
        "max_new_tokens": 2048,
        "prompt_template": "You are a helpful AI assistant.\n\nUSER: {prompt}\nASSISTANT:",
        "presence_penalty": 1,
        "frequency_penalty": 0
    }
)

# The nateraw/goliath-120b model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/nateraw/goliath-120b/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run nateraw/goliath-120b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "nateraw/goliath-120b:39c0be5d2e96df975da94e8923535733d81179abed5f37c999afc5f7c33bf04e",
    "input": {
      "top_k": 50,
      "top_p": 0.95,
      "prompt": "write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. ",
      "temperature": 0.8,
      "max_new_tokens": 2048,
      "prompt_template": "You are a helpful AI assistant.\\n\\nUSER: {prompt}\\nASSISTANT:",
      "presence_penalty": 1,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Title: The King of Cheeses: All Hail Parmigiano Reggiano! Introduction: Welcome, cheese aficionados! Today, we're going to dive into the world of the one and only Parmigiano Reggiano. That's right, I'm talking about the real deal, not that pre-grated, mass-produced "Parmesan" you find in American grocery stores. We're here to pay homage to the real thing - the King of Cheeses! What is Parmigiano Reggiano? Parmigiano Reggiano (pronounced par-mee-JYAH-no reh-JYAH-no) is a hard, aged Italian cheese that hails from the Emilia-Romagna region of Italy. It has been made using traditional methods for over 800 years! This PDO (Protected Designation of Origin) product means that it can only be called Parmigiano Reggiano if it's made within the specified area and adheres to strict production guidelines. Why is it better than American "Parmesan"? American "Parmesan" cheese is often a pale imitation of the real thing. It's typically made from a blend of cheeses, including Cheddar and other cheaper varieties, and is aged for a much shorter period - usually just a few months compared to the minimum 12-24 months for Parmigiano Reggiano. The result? A cheese that lacks depth of flavor, complexity, and that unmistakable umami kick. Tips for Buying Parmigiano Reggiano: 1. Look for the mark: Authentic Parmigiano Reggiano will have the words "Parmigiano Reggiano Stagionato" (or similar, depending on the aging) stamped on the rind. The rind should also have the brand of the dairy and the month and year of production. 2. Check the texture: The cheese should be firm but slightly elastic. Avoid any pieces with cracks or mold (other than the natural mold that forms on the rind). 3. Smell the cheese: It should have a nutty, grassy aroma with a hint of pungency. If it smells sour or ammonia-like, pass it by. Grating your own vs. pre-grated: While pre-grated Parmigiano Reggiano may seem convenient, nothing beats grating your own cheese. Pre-grated cheese can contain anti-caking agents and preservatives to maintain its consistency in the bag. Grating your own ensures you get the freshest, most flavorful cheese possible. Plus, it's therapeutic! Using Parmigiano Reggiano Rinds: Don't throw away those precious rinds! Here are some creative ways to use them: 1. Soup and stew booster: Add the rind to your favorite soup or stew during cooking to infuse it with rich, cheesy flavor. Remove before serving. 2. Risotto enhancer: Stir a piece of rind into your risotto during cooking for an extra burst of umami goodness. 3. Pesto elevator: Throw a rind into your food processor when making pesto for a subtle cheesy undertone. Discard before serving. 4. Mac and cheese upgrade: Add a rind to your mac and cheese sauce while it's simmering for a cheesier taste without adding more cheese. Conclusion: In conclusion, friends, there's no denying that Parmigiano Reggiano is in a league of its own. Next time you're at the cheese counter, skip the imposters and treat yourself to a wedge of the real deal. Your taste buds (and your pasta dishes) will thank you! And remember, always #SayNoToAmericanParmesan and #EatRealCheese.

{
  "completed_at": "2023-11-16T06:30:00.115062Z",
  "created_at": "2023-11-16T06:23:35.783633Z",
  "data_removed": false,
  "error": null,
  "id": "6m67ozlbrgeocuamfsin7dxnl4",
  "input": {
    "top_k": 50,
    "top_p": 0.95,
    "prompt": "write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. ",
    "temperature": 0.8,
    "max_new_tokens": 2048,
    "prompt_template": "You are a helpful AI assistant.\n\nUSER: {prompt}\nASSISTANT:",
    "presence_penalty": 1,
    "frequency_penalty": 0
  },
  "logs": "INFO 11-16 06:28:37 async_llm_engine.py:370] Received request 0: prompt: 'You are a helpful AI assistant.\\n\\nUSER: write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. \\nASSISTANT:', sampling params: SamplingParams(n=1, best_of=1, presence_penalty=1.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.8, top_p=0.95, top_k=50, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['</s>'], ignore_eos=False, max_tokens=2048, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True), prompt token ids: None.\nINFO 11-16 06:28:38 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%\nINFO 11-16 06:28:43 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%\nINFO 11-16 06:28:48 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%\nINFO 11-16 06:28:53 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%\nINFO 11-16 06:28:58 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:03 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:08 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:13 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:18 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.4%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:23 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:28 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.1%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:33 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:38 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:43 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:48 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.5%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:53 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.9%, CPU KV cache usage: 0.0%\nINFO 11-16 06:29:58 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.2%, CPU KV cache usage: 0.0%\nINFO 11-16 06:30:00 async_llm_engine.py:111] Finished request 0.\ngeneration took 82.316s",
  "metrics": {
    "predict_time": 82.365771,
    "total_time": 384.331429
  },
  "output": [
    " Title",
    ":",
    " The",
    " King",
    " of",
    " Che",
    "eses",
    ":",
    " All",
    " H",
    "ail",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    "!",
    "\n",
    "\n",
    "Introduction",
    ":",
    "\n",
    "\n",
    "Wel",
    "come",
    ",",
    " che",
    "ese",
    " af",
    "icion",
    "ados",
    "!",
    " Today",
    ",",
    " we",
    "'",
    "re",
    " going",
    " to",
    " d",
    "ive",
    " into",
    " the",
    " world",
    " of",
    " the",
    " one",
    " and",
    " only",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    ".",
    " That",
    "'",
    "s",
    " right",
    ",",
    " I",
    "'",
    "m",
    " talking",
    " about",
    " the",
    " real",
    " deal",
    ",",
    " not",
    " that",
    " pre",
    "-",
    "gr",
    "ated",
    ",",
    " mass",
    "-",
    "produ",
    "ced",
    " \"",
    "Par",
    "mes",
    "an",
    "\"",
    " you",
    " find",
    " in",
    " American",
    " gro",
    "c",
    "ery",
    " stores",
    ".",
    " We",
    "'",
    "re",
    " here",
    " to",
    " pay",
    " hom",
    "age",
    " to",
    " the",
    " real",
    " thing",
    " -",
    " the",
    " King",
    " of",
    " Che",
    "eses",
    "!",
    "\n",
    "\n",
    "What",
    " is",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    "?",
    "\n",
    "\n",
    "Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " (",
    "pr",
    "on",
    "ounced",
    " par",
    "-",
    "mee",
    "-",
    "J",
    "Y",
    "A",
    "H",
    "-",
    "no",
    " re",
    "h",
    "-",
    "J",
    "Y",
    "A",
    "H",
    "-",
    "no",
    ")",
    " is",
    " a",
    " hard",
    ",",
    " aged",
    " Italian",
    " che",
    "ese",
    " that",
    " ha",
    "ils",
    " from",
    " the",
    " Em",
    "ilia",
    "-",
    "R",
    "om",
    "agna",
    " region",
    " of",
    " Italy",
    ".",
    " It",
    " has",
    " been",
    " made",
    " using",
    " traditional",
    " methods",
    " for",
    " over",
    " ",
    "8",
    "0",
    "0",
    " years",
    "!",
    " This",
    " PDO",
    " (",
    "Pro",
    "te",
    "cted",
    " Design",
    "ation",
    " of",
    " Origin",
    ")",
    " product",
    " means",
    " that",
    " it",
    " can",
    " only",
    " be",
    " called",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " if",
    " it",
    "'",
    "s",
    " made",
    " within",
    " the",
    " specified",
    " area",
    " and",
    " ad",
    "her",
    "es",
    " to",
    " strict",
    " production",
    " gu",
    "idel",
    "ines",
    ".",
    "\n",
    "\n",
    "Why",
    " is",
    " it",
    " better",
    " than",
    " American",
    " \"",
    "Par",
    "mes",
    "an",
    "\"?",
    "\n",
    "\n",
    "American",
    " \"",
    "Par",
    "mes",
    "an",
    "\"",
    " che",
    "ese",
    " is",
    " often",
    " a",
    " pale",
    " im",
    "itation",
    " of",
    " the",
    " real",
    " thing",
    ".",
    " It",
    "'",
    "s",
    " typically",
    " made",
    " from",
    " a",
    " bl",
    "end",
    " of",
    " che",
    "eses",
    ",",
    " including",
    " Ch",
    "ed",
    "dar",
    " and",
    " other",
    " che",
    "aper",
    " vari",
    "eties",
    ",",
    " and",
    " is",
    " aged",
    " for",
    " a",
    " much",
    " shorter",
    " period",
    " -",
    " usually",
    " just",
    " a",
    " few",
    " months",
    " compared",
    " to",
    " the",
    " minimum",
    " ",
    "1",
    "2",
    "-",
    "2",
    "4",
    " months",
    " for",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    ".",
    " The",
    " result",
    "?",
    " A",
    " che",
    "ese",
    " that",
    " la",
    "cks",
    " depth",
    " of",
    " flav",
    "or",
    ",",
    " complexity",
    ",",
    " and",
    " that",
    " un",
    "m",
    "ist",
    "ak",
    "able",
    " um",
    "ami",
    " kick",
    ".",
    "\n",
    "\n",
    "T",
    "ips",
    " for",
    " Bu",
    "ying",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    ":",
    "\n",
    "\n",
    "1",
    ".",
    " Look",
    " for",
    " the",
    " mark",
    ":",
    " Auth",
    "entic",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " will",
    " have",
    " the",
    " words",
    " \"",
    "Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " St",
    "ag",
    "ion",
    "ato",
    "\"",
    " (",
    "or",
    " similar",
    ",",
    " depending",
    " on",
    " the",
    " ag",
    "ing",
    ")",
    " stamp",
    "ed",
    " on",
    " the",
    " r",
    "ind",
    ".",
    " The",
    " r",
    "ind",
    " should",
    " also",
    " have",
    " the",
    " brand",
    " of",
    " the",
    " da",
    "iry",
    " and",
    " the",
    " month",
    " and",
    " year",
    " of",
    " production",
    ".",
    "\n",
    "2",
    ".",
    " Check",
    " the",
    " texture",
    ":",
    " The",
    " che",
    "ese",
    " should",
    " be",
    " firm",
    " but",
    " slightly",
    " el",
    "astic",
    ".",
    " A",
    "void",
    " any",
    " pieces",
    " with",
    " cra",
    "cks",
    " or",
    " m",
    "old",
    " (",
    "other",
    " than",
    " the",
    " natural",
    " m",
    "old",
    " that",
    " forms",
    " on",
    " the",
    " r",
    "ind",
    ").",
    "\n",
    "3",
    ".",
    " Sm",
    "ell",
    " the",
    " che",
    "ese",
    ":",
    " It",
    " should",
    " have",
    " a",
    " nut",
    "ty",
    ",",
    " grass",
    "y",
    " a",
    "rom",
    "a",
    " with",
    " a",
    " hint",
    " of",
    " p",
    "ungen",
    "cy",
    ".",
    " If",
    " it",
    " sm",
    "ells",
    " s",
    "our",
    " or",
    " am",
    "mon",
    "ia",
    "-",
    "like",
    ",",
    " pass",
    " it",
    " by",
    ".",
    "\n",
    "\n",
    "Gr",
    "ating",
    " your",
    " own",
    " vs",
    ".",
    " pre",
    "-",
    "gr",
    "ated",
    ":",
    "\n",
    "\n",
    "Wh",
    "ile",
    " pre",
    "-",
    "gr",
    "ated",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " may",
    " seem",
    " convenient",
    ",",
    " nothing",
    " be",
    "ats",
    " gr",
    "ating",
    " your",
    " own",
    " che",
    "ese",
    ".",
    " Pre",
    "-",
    "gr",
    "ated",
    " che",
    "ese",
    " can",
    " contain",
    " anti",
    "-",
    "c",
    "aking",
    " agents",
    " and",
    " pres",
    "erv",
    "atives",
    " to",
    " maintain",
    " its",
    " consist",
    "ency",
    " in",
    " the",
    " bag",
    ".",
    " Gr",
    "ating",
    " your",
    " own",
    " ens",
    "ures",
    " you",
    " get",
    " the",
    " fresh",
    "est",
    ",",
    " most",
    " flav",
    "or",
    "ful",
    " che",
    "ese",
    " possible",
    ".",
    " Plus",
    ",",
    " it",
    "'",
    "s",
    " th",
    "era",
    "pe",
    "ut",
    "ic",
    "!",
    "\n",
    "\n",
    "Using",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " R",
    "inds",
    ":",
    "\n",
    "\n",
    "Don",
    "'",
    "t",
    " throw",
    " away",
    " those",
    " pre",
    "cious",
    " r",
    "inds",
    "!",
    " Here",
    " are",
    " some",
    " cre",
    "ative",
    " ways",
    " to",
    " use",
    " them",
    ":",
    "\n",
    "\n",
    "1",
    ".",
    " S",
    "oup",
    " and",
    " ste",
    "w",
    " boost",
    "er",
    ":",
    " Add",
    " the",
    " r",
    "ind",
    " to",
    " your",
    " favorite",
    " soup",
    " or",
    " ste",
    "w",
    " during",
    " cook",
    "ing",
    " to",
    " inf",
    "use",
    " it",
    " with",
    " rich",
    ",",
    " che",
    "es",
    "y",
    " flav",
    "or",
    ".",
    " Remove",
    " before",
    " serving",
    ".",
    "\n",
    "2",
    ".",
    " Ris",
    "otto",
    " en",
    "han",
    "cer",
    ":",
    " St",
    "ir",
    " a",
    " piece",
    " of",
    " r",
    "ind",
    " into",
    " your",
    " ris",
    "otto",
    " during",
    " cook",
    "ing",
    " for",
    " an",
    " extra",
    " burst",
    " of",
    " um",
    "ami",
    " good",
    "ness",
    ".",
    "\n",
    "3",
    ".",
    " P",
    "esto",
    " elev",
    "ator",
    ":",
    " Th",
    "row",
    " a",
    " r",
    "ind",
    " into",
    " your",
    " food",
    " processor",
    " when",
    " making",
    " p",
    "esto",
    " for",
    " a",
    " subt",
    "le",
    " che",
    "es",
    "y",
    " undert",
    "one",
    ".",
    " Dis",
    "card",
    " before",
    " serving",
    ".",
    "\n",
    "4",
    ".",
    " Mac",
    " and",
    " che",
    "ese",
    " upgrade",
    ":",
    " Add",
    " a",
    " r",
    "ind",
    " to",
    " your",
    " mac",
    " and",
    " che",
    "ese",
    " sau",
    "ce",
    " while",
    " it",
    "'",
    "s",
    " sim",
    "mer",
    "ing",
    " for",
    " a",
    " che",
    "es",
    "ier",
    " taste",
    " without",
    " adding",
    " more",
    " che",
    "ese",
    ".",
    "\n",
    "\n",
    "Con",
    "clusion",
    ":",
    "\n",
    "\n",
    "In",
    " conclusion",
    ",",
    " friends",
    ",",
    " there",
    "'",
    "s",
    " no",
    " den",
    "ying",
    " that",
    " Par",
    "m",
    "ig",
    "iano",
    " Reg",
    "g",
    "iano",
    " is",
    " in",
    " a",
    " league",
    " of",
    " its",
    " own",
    ".",
    " Next",
    " time",
    " you",
    "'",
    "re",
    " at",
    " the",
    " che",
    "ese",
    " counter",
    ",",
    " skip",
    " the",
    " imp",
    "ost",
    "ers",
    " and",
    " treat",
    " yourself",
    " to",
    " a",
    " w",
    "edge",
    " of",
    " the",
    " real",
    " deal",
    ".",
    " Your",
    " taste",
    " bud",
    "s",
    " (",
    "and",
    " your",
    " past",
    "a",
    " d",
    "ishes",
    ")",
    " will",
    " thank",
    " you",
    "!",
    " And",
    " remember",
    ",",
    " always",
    " #",
    "S",
    "ay",
    "No",
    "To",
    "American",
    "Par",
    "mes",
    "an",
    " and",
    " #",
    "E",
    "at",
    "Real",
    "Che",
    "ese",
    ".",
    ""
  ],
  "started_at": "2023-11-16T06:28:37.749291Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/6m67ozlbrgeocuamfsin7dxnl4",
    "cancel": "https://api.replicate.com/v1/predictions/6m67ozlbrgeocuamfsin7dxnl4/cancel"
  },
  "version": "39c0be5d2e96df975da94e8923535733d81179abed5f37c999afc5f7c33bf04e"
}

Generated in

82.4 seconds

Tweak it Report

INFO 11-16 06:28:37 async_llm_engine.py:370] Received request 0: prompt: 'You are a helpful AI assistant.\n\nUSER: write me a blogpost all about parmigiano reggiano. Include helpful tips, like what to do with the rinds, whether you should buy pre-grated vs non-grated, etc, etc. Grumble about poor quality american Parmesan. \nASSISTANT:', sampling params: SamplingParams(n=1, best_of=1, presence_penalty=1.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.8, top_p=0.95, top_k=50, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['</s>'], ignore_eos=False, max_tokens=2048, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True), prompt token ids: None.
INFO 11-16 06:28:38 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%
INFO 11-16 06:28:43 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%
INFO 11-16 06:28:48 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%
INFO 11-16 06:28:53 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%
INFO 11-16 06:28:58 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.0%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:03 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.3%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:08 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.7%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:13 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.0%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:18 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.4%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:23 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.8%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:28 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.1%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:33 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.5%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:38 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.8%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:43 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.2%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:48 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.5%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:53 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.9%, CPU KV cache usage: 0.0%
INFO 11-16 06:29:58 llm_engine.py:624] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.2%, CPU KV cache usage: 0.0%
INFO 11-16 06:30:00 async_llm_engine.py:111] Finished request 0.
generation took 82.316s

Run time and cost

This model costs approximately $0.043 to run on Replicate, or 23 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 31 seconds. The predict time for this model varies significantly based on the inputs.

Readme

See the full model card here

The model we are using here is the AWQ quantized version, which can be found here