nateraw/zephyr-7b-beta – Run with an API on Replicate

nateraw / zephyr-7b-beta

Zephyr-7B-beta, an LLM trained to act as a helpful assistant.

Cold

Public
5.7K runs
L40S
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

Write me an itinerary for my dog's birthday partyWrite me an itinerary for my dog's birthday party

max_new_tokens

integer

The maximum number of tokens the model should generate as output.

Default: 512

temperature

number

The value used to modulate the next token probabilities.

Default: 0.8

top_p

number

A probability threshold for generating the output. If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751).

Default: 0.95

top_k

integer

The number of highest probability tokens to consider for generating the output. If > 0, only keep the top k tokens with highest probability (top-k filtering).

Default: 50

presence_penalty

number

Presence penalty

Default: 0

frequency_penalty

number

Frequency penalty

Default: 0

prompt_template

string

Shift + Return to add a new line

<|system|>
</s>
<|user|>
{prompt}</s>
<|assistant|>
<|system|>
</s>
<|user|>
{prompt}</s>
<|assistant|>

The template used to format the prompt. The input prompt is inserted into the template using the `{prompt}` placeholder.

Default: "<|system|>\n</s>\n<|user|>\n{prompt}</s>\n<|assistant|>\n"

Run this model in Node.js with one line of code:

npx create-replicate --model=nateraw/zephyr-7b-beta

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run nateraw/zephyr-7b-beta using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "nateraw/zephyr-7b-beta:b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931",
  {
    input: {
      top_k: 50,
      top_p: 0.95,
      prompt: "Write me an itinerary for my dog's birthday party",
      temperature: 0.8,
      max_new_tokens: 512,
      prompt_template: "<|system|>\n</s>\n<|user|>\n{prompt}</s>\n<|assistant|>\n",
      presence_penalty: 0,
      frequency_penalty: 0
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run nateraw/zephyr-7b-beta using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "nateraw/zephyr-7b-beta:b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931",
    input={
        "top_k": 50,
        "top_p": 0.95,
        "prompt": "Write me an itinerary for my dog's birthday party",
        "temperature": 0.8,
        "max_new_tokens": 512,
        "prompt_template": "<|system|>\n</s>\n<|user|>\n{prompt}</s>\n<|assistant|>\n",
        "presence_penalty": 0,
        "frequency_penalty": 0
    }
)

# The nateraw/zephyr-7b-beta model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/nateraw/zephyr-7b-beta/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run nateraw/zephyr-7b-beta using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931",
    "input": {
      "top_k": 50,
      "top_p": 0.95,
      "prompt": "Write me an itinerary for my dog\'s birthday party",
      "temperature": 0.8,
      "max_new_tokens": 512,
      "prompt_template": "<|system|>\\n</s>\\n<|user|>\\n{prompt}</s>\\n<|assistant|>\\n",
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/nateraw/zephyr-7b-beta@sha256:b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931 \
  -i 'top_k=50' \
  -i 'top_p=0.95' \
  -i $'prompt="Write me an itinerary for my dog\'s birthday party"' \
  -i 'temperature=0.8' \
  -i 'max_new_tokens=512' \
  -i $'prompt_template="<|system|>\\n</s>\\n<|user|>\\n{prompt}</s>\\n<|assistant|>\\n"' \
  -i 'presence_penalty=0' \
  -i 'frequency_penalty=0'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/nateraw/zephyr-7b-beta@sha256:b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "top_k": 50,
      "top_p": 0.95,
      "prompt": "Write me an itinerary for my dog\'s birthday party",
      "temperature": 0.8,
      "max_new_tokens": 512,
      "prompt_template": "<|system|>\\n</s>\\n<|user|>\\n{prompt}</s>\\n<|assistant|>\\n",
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

Dog's Birthday Party Itinerary: 1. Introduction and Greetings (5-10 minutes) - Welcome guests and ask them to introduce themselves and their furry friends - Explain the rules and expectations for the party (no aggressive dogs, no jumping on people or other dogs, no stealing of toys or treats) - Introduce the birthday dog and let them explore their surroundings 2. Games and Activities (30 minutes) - Fetch: Set up a designated area for fetch, and have some volunteers toss a ball or Frisbee to the birthday dog and let the other dogs have a turn - Tug-of-war: Using a strong rope, tie a knot at each end and let the dogs have a pull. You could also use a toy like a chew rope or a tennis ball tied to the rope - Puzzle Feeders: Set up some puzzle feeders with treats inside, and let the dogs work to figure out how to get the treats out 3. Doggie Cupcakes and Treats (15 minutes) - Bake or buy some dog-friendly cupcakes and treats, and let the dogs enjoy a sweet treat - You could also set up a DIY treat station, where guests can make their own doggie cupcakes or treats using ingredients like peanut butter, oats, and bananas 4. Birthday Wishes and Gifts (15 minutes) - Let the birthday dog open their presents, and have the guests cheer and applaud - Ask the guests to share their birthday wishes for the birthday dog 5. Farewell and Thank Yous (10 minutes) - Thank the guests for coming and for making the birthday party a success - Ask everyone to clean up after themselves and their dogs, and to say goodbye to the birthday dog Tips: - Make sure all the dogs are comfortable around each other and that there are no aggressive dogs present - Provide plenty of water and shade for the dogs - Have some toys and balls on hand for the dogs to play with - Don't forget to have some cleanup supplies like poop bags and wet wipes - Make sure the birthday dog gets plenty of attention and love throughout the party - If you're having a large group, consider hiring a professional dog trainer to help manage the dogs and ensure

{
  "completed_at": "2023-10-31T19:10:18.732745Z",
  "created_at": "2023-10-31T19:10:10.100943Z",
  "data_removed": false,
  "error": null,
  "id": "vczhzxdbr2fx4l53uhqh7vipia",
  "input": {
    "top_k": 50,
    "top_p": 0.95,
    "prompt": "Write me an itinerary for my dog's birthday party",
    "temperature": 0.8,
    "max_new_tokens": 512,
    "prompt_template": "<|system|>\n</s>\n<|user|>\n{prompt}</s>\n<|assistant|>\n",
    "presence_penalty": 0,
    "frequency_penalty": 0
  },
  "logs": "Generated 512 tokens in 6.390033006668091 seconds.",
  "metrics": {
    "predict_time": 6.451349,
    "total_time": 8.631802
  },
  "output": [
    "D",
    "og",
    "'",
    "s",
    " Birth",
    "day",
    " Party",
    " It",
    "iner",
    "ary",
    ":",
    "\n",
    "\n",
    "1",
    ".",
    " Introduction",
    " and",
    " Gre",
    "et",
    "ings",
    " (",
    "5",
    "-",
    "1",
    "0",
    " minutes",
    ")",
    "\n",
    "-",
    " Welcome",
    " guests",
    " and",
    " ask",
    " them",
    " to",
    " introduce",
    " themselves",
    " and",
    " their",
    " fur",
    "ry",
    " friends",
    "\n",
    "-",
    " Expl",
    "ain",
    " the",
    " rules",
    " and",
    " expectations",
    " for",
    " the",
    " party",
    " (",
    "no",
    " aggressive",
    " dogs",
    ",",
    " no",
    " jumping",
    " on",
    " people",
    " or",
    " other",
    " dogs",
    ",",
    " no",
    " ste",
    "aling",
    " of",
    " toys",
    " or",
    " treats",
    ")",
    "\n",
    "-",
    " Int",
    "rodu",
    "ce",
    " the",
    " birthday",
    " dog",
    " and",
    " let",
    " them",
    " explore",
    " their",
    " surroundings",
    "\n",
    "\n",
    "2",
    ".",
    " Games",
    " and",
    " Activ",
    "ities",
    " (",
    "3",
    "0",
    " minutes",
    ")",
    "\n",
    "-",
    " F",
    "etch",
    ":",
    " Set",
    " up",
    " a",
    " designated",
    " area",
    " for",
    " fetch",
    ",",
    " and",
    " have",
    " some",
    " volunteers",
    " toss",
    " a",
    " ball",
    " or",
    " Fr",
    "is",
    "bee",
    " to",
    " the",
    " birthday",
    " dog",
    " and",
    " let",
    " the",
    " other",
    " dogs",
    " have",
    " a",
    " turn",
    "\n",
    "-",
    " T",
    "ug",
    "-",
    "of",
    "-",
    "war",
    ":",
    " Using",
    " a",
    " strong",
    " rope",
    ",",
    " tie",
    " a",
    " kn",
    "ot",
    " at",
    " each",
    " end",
    " and",
    " let",
    " the",
    " dogs",
    " have",
    " a",
    " pull",
    ".",
    " You",
    " could",
    " also",
    " use",
    " a",
    " toy",
    " like",
    " a",
    " che",
    "w",
    " rope",
    " or",
    " a",
    " tennis",
    " ball",
    " tied",
    " to",
    " the",
    " rope",
    "\n",
    "-",
    " P",
    "uzz",
    "le",
    " Fe",
    "ed",
    "ers",
    ":",
    " Set",
    " up",
    " some",
    " puzzle",
    " feed",
    "ers",
    " with",
    " treats",
    " inside",
    ",",
    " and",
    " let",
    " the",
    " dogs",
    " work",
    " to",
    " figure",
    " out",
    " how",
    " to",
    " get",
    " the",
    " treats",
    " out",
    "\n",
    "\n",
    "3",
    ".",
    " Dog",
    "gie",
    " Cup",
    "c",
    "akes",
    " and",
    " Treat",
    "s",
    " (",
    "1",
    "5",
    " minutes",
    ")",
    "\n",
    "-",
    " B",
    "ake",
    " or",
    " buy",
    " some",
    " dog",
    "-",
    "friendly",
    " cup",
    "c",
    "akes",
    " and",
    " treats",
    ",",
    " and",
    " let",
    " the",
    " dogs",
    " enjoy",
    " a",
    " sweet",
    " treat",
    "\n",
    "-",
    " You",
    " could",
    " also",
    " set",
    " up",
    " a",
    " DI",
    "Y",
    " treat",
    " station",
    ",",
    " where",
    " guests",
    " can",
    " make",
    " their",
    " own",
    " dog",
    "gie",
    " cup",
    "c",
    "akes",
    " or",
    " treats",
    " using",
    " ingredients",
    " like",
    " pe",
    "an",
    "ut",
    " butter",
    ",",
    " o",
    "ats",
    ",",
    " and",
    " ban",
    "anas",
    "\n",
    "\n",
    "4",
    ".",
    " Birth",
    "day",
    " W",
    "ishes",
    " and",
    " G",
    "ifts",
    " (",
    "1",
    "5",
    " minutes",
    ")",
    "\n",
    "-",
    " Let",
    " the",
    " birthday",
    " dog",
    " open",
    " their",
    " presents",
    ",",
    " and",
    " have",
    " the",
    " guests",
    " cheer",
    " and",
    " appl",
    "aud",
    "\n",
    "-",
    " Ask",
    " the",
    " guests",
    " to",
    " share",
    " their",
    " birthday",
    " wishes",
    " for",
    " the",
    " birthday",
    " dog",
    "\n",
    "\n",
    "5",
    ".",
    " F",
    "are",
    "well",
    " and",
    " Thank",
    " Y",
    "ous",
    " (",
    "1",
    "0",
    " minutes",
    ")",
    "\n",
    "-",
    " Thank",
    " the",
    " guests",
    " for",
    " coming",
    " and",
    " for",
    " making",
    " the",
    " birthday",
    " party",
    " a",
    " success",
    "\n",
    "-",
    " Ask",
    " everyone",
    " to",
    " clean",
    " up",
    " after",
    " themselves",
    " and",
    " their",
    " dogs",
    ",",
    " and",
    " to",
    " say",
    " good",
    "bye",
    " to",
    " the",
    " birthday",
    " dog",
    "\n",
    "\n",
    "T",
    "ips",
    ":",
    "\n",
    "\n",
    "-",
    " Make",
    " sure",
    " all",
    " the",
    " dogs",
    " are",
    " comfortable",
    " around",
    " each",
    " other",
    " and",
    " that",
    " there",
    " are",
    " no",
    " aggressive",
    " dogs",
    " present",
    "\n",
    "-",
    " Prov",
    "ide",
    " plenty",
    " of",
    " water",
    " and",
    " shade",
    " for",
    " the",
    " dogs",
    "\n",
    "-",
    " Have",
    " some",
    " toys",
    " and",
    " balls",
    " on",
    " hand",
    " for",
    " the",
    " dogs",
    " to",
    " play",
    " with",
    "\n",
    "-",
    " Don",
    "'",
    "t",
    " forget",
    " to",
    " have",
    " some",
    " cleanup",
    " supplies",
    " like",
    " po",
    "op",
    " bags",
    " and",
    " wet",
    " w",
    "ipes",
    "\n",
    "-",
    " Make",
    " sure",
    " the",
    " birthday",
    " dog",
    " gets",
    " plenty",
    " of",
    " attention",
    " and",
    " love",
    " throughout",
    " the",
    " party",
    "\n",
    "-",
    " If",
    " you",
    "'",
    "re",
    " having",
    " a",
    " large",
    " group",
    ",",
    " consider",
    " hiring",
    " a",
    " professional",
    " dog",
    " trainer",
    " to",
    " help",
    " manage",
    " the",
    " dogs",
    " and",
    " ensure"
  ],
  "started_at": "2023-10-31T19:10:12.281396Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/vczhzxdbr2fx4l53uhqh7vipia",
    "cancel": "https://api.replicate.com/v1/predictions/vczhzxdbr2fx4l53uhqh7vipia/cancel"
  },
  "version": "b79f33de5c6c4e34087d44eaea4a9d98ce5d3f3a09522f7328eea0685003a931"
}

Generated in

6.5 seconds

Tweak it Report

Run time and cost

This model costs approximately $0.13 to run on Replicate, or 7 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 135 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr-7B-β is the second model in the series, and is a fine-tuned version of mistralai/Mistral-7B-v0.1 that was trained on on a mix of publicly available, synthetic datasets using Direct Preference Optimization (DPO). We found that removing the in-built alignment of these datasets boosted performance on MT Bench and made the model more helpful. However, this means that model is likely to generate problematic text when prompted to do so and should only be used for educational and research purposes. You can find more details in the technical report.

See the full model card for more details: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

Thank you to TheBloke for releasing the AWQ version of the model, which is served here. Link: https://huggingface.co/TheBloke/zephyr-7B-beta-AWQ