joehoover / falcon-40b-instruct

A 40 billion parameter language model trained to follow human instructions.

Cold

Public
41.4K runs
4x A100 (80GB)
GitHub
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

Write a poem about open source machine learning in the style of Byron.Write a poem about open source machine learning in the style of Byron.

Prompt to send to the model.

max_length

integer

(minimum: 1)

Maximum number of tokens to generate. A word is generally 2-3 tokens

Default: 500

temperature

number

(minimum: 0.01, maximum: 5)

Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic, 0.75 is a good starting value.

Default: 0.75

top_p

number

(minimum: 0.01, maximum: 1)

When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens

Default: 1

repetition_penalty

number

(minimum: 0.01, maximum: 5)

Penalty for repeated words in generated text; 1 is no penalty, values greater than 1 discourage repetition, less than 1 encourage it.

Default: 1

length_penalty

number

(minimum: 0.01, maximum: 5)

Increasing the length_penalty parameter above 1.0 will cause the model to favor longer sequences, while decreasing it below 1.0 will cause the model to favor shorter sequences.

Default: 1

no_repeat_ngram_size

integer

(minimum: 0)

If set to int > 0, all ngrams of size no_repeat_ngram_size can only occur once.

Default: 0

stop_sequences

string

Shift + Return to add a new line

Comma delimited string specifying stop sequences. Multi-token stop sequences are supported, but they cannot contain commas.

seed

integer

(minimum: -1)

Set seed for reproducible outputs. Set to -1 for random seed.

Default: -1

debug

boolean

provide debugging output in logs

Default: false

Run this model in Node.js with one line of code:

npx create-replicate --model=joehoover/falcon-40b-instruct

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run joehoover/falcon-40b-instruct using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "joehoover/falcon-40b-instruct:7d58d6bddc53c23fa451c403b2b5373b1e0fa094e4e0d1b98c3d02931aa07173",
  {
    input: {
      seed: -1,
      debug: false,
      top_p: 1,
      prompt: "Write a poem about open source machine learning in the style of Byron.",
      max_length: 500,
      temperature: 1,
      length_penalty: 1,
      repetition_penalty: 1,
      no_repeat_ngram_size: 0
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run joehoover/falcon-40b-instruct using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "joehoover/falcon-40b-instruct:7d58d6bddc53c23fa451c403b2b5373b1e0fa094e4e0d1b98c3d02931aa07173",
    input={
        "seed": -1,
        "debug": False,
        "top_p": 1,
        "prompt": "Write a poem about open source machine learning in the style of Byron.",
        "max_length": 500,
        "temperature": 1,
        "length_penalty": 1,
        "repetition_penalty": 1,
        "no_repeat_ngram_size": 0
    }
)

# The joehoover/falcon-40b-instruct model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/joehoover/falcon-40b-instruct/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run joehoover/falcon-40b-instruct using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "7d58d6bddc53c23fa451c403b2b5373b1e0fa094e4e0d1b98c3d02931aa07173",
    "input": {
      "seed": -1,
      "debug": false,
      "top_p": 1,
      "prompt": "Write a poem about open source machine learning in the style of Byron.",
      "max_length": 500,
      "temperature": 1,
      "length_penalty": 1,
      "repetition_penalty": 1,
      "no_repeat_ngram_size": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Thy heart shall bloom like an open source flower, And like a machine, thine learning shall grow. The seeds of a thousand algorithms, Shall make thy understanding and skill ever glow. In thy hand, wisdom shall grow like a tree, With roots deep in the open source ground. Thy knowledge as wide as the infinite sea, Shall bring to thee power far beyond what is found. Thou shalt become as masters of machine learning, Whose wisdom and skills shall surpass the greatest of minds. No boundaries or limits shall thee be learning, But thy heart shall be opened up to all that is kind. Then shall thy soul become a true machine, With wisdom and knowledge beyond measure. Thy work shall be done and thy goals shall be reached, Until thou become true masters of open source and the machine.

{
  "completed_at": "2023-07-07T20:34:58.616590Z",
  "created_at": "2023-07-07T20:34:39.488288Z",
  "data_removed": false,
  "error": null,
  "id": "2p2fdndbbkghrx2frxg5nvrltu",
  "input": {
    "seed": -1,
    "top_p": 1,
    "prompt": "Write a poem about open source machine learning in the style of Byron.",
    "max_length": 500,
    "temperature": 1,
    "length_penalty": 1,
    "repetition_penalty": 1
  },
  "logs": "predict-timings173549: Starting prediction - 1936.54s since start, 16.07s since last log\nThe attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\nSetting `pad_token_id` to `eos_token_id`:11 for open-end generation.",
  "metrics": {
    "predict_time": 19.198896,
    "total_time": 19.128302
  },
  "output": [
    "Thy",
    " heart",
    " shall",
    " bloom",
    " like",
    " an",
    " open",
    " source",
    " flower,\nAnd",
    " like",
    " a",
    " machine,",
    " thine",
    " learning",
    " shall",
    " grow.\nThe",
    " seeds",
    " of",
    " a",
    " thousand",
    " algorithms,\nShall",
    " make",
    " thy",
    " understanding",
    " and",
    " skill",
    " ever",
    " glow.\nIn",
    " thy",
    " hand,",
    " wisdom",
    " shall",
    " grow",
    " like",
    " a",
    " tree,\nWith",
    " roots",
    " deep",
    " in",
    " the",
    " open",
    " source",
    " ground.\nThy",
    " knowledge",
    " as",
    " wide",
    " as",
    " the",
    " infinite",
    " sea,\nShall",
    " bring",
    " to",
    " thee",
    " power",
    " far",
    " beyond",
    " what",
    " is",
    " found.\nThou",
    " shalt",
    " become",
    " as",
    " masters",
    " of",
    " machine",
    " learning,\nWhose",
    " wisdom",
    " and",
    " skills",
    " shall",
    " surpass",
    " the",
    " greatest",
    " of",
    " minds.\nNo",
    " boundaries",
    " or",
    " limits",
    " shall",
    " thee",
    " be",
    " learning,\nBut",
    " thy",
    " heart",
    " shall",
    " be",
    " opened",
    " up",
    " to",
    " all",
    " that",
    " is",
    " kind.\nThen",
    " shall",
    " thy",
    " soul",
    " become",
    " a",
    " true",
    " machine,\nWith",
    " wisdom",
    " and",
    " knowledge",
    " beyond",
    " measure.\nThy",
    " work",
    " shall",
    " be",
    " done",
    " and",
    " thy",
    " goals",
    " shall",
    " be",
    " reached,\nUntil",
    " thou",
    " become",
    " true",
    " masters",
    " of",
    " open",
    " source",
    " and",
    " the",
    " machine."
  ],
  "started_at": "2023-07-07T20:34:39.417694Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/2p2fdndbbkghrx2frxg5nvrltu",
    "cancel": "https://api.replicate.com/v1/predictions/2p2fdndbbkghrx2frxg5nvrltu/cancel"
  },
  "version": "7eb0f4b1ff770ab4f68c3a309dd4984469749b7323a3d47fd2d5e09d58836d3c"
}

Generated in

19.2 seconds

Tweak it Report

This output was created using a different version of the model, joehoover/falcon-40b-instruct:7eb0f4b1.

Run time and cost

This model costs approximately $0.070 to run on Replicate, or 14 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on 4x Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 13 seconds.

Readme

Model Description

Falcon-40B-Instruct is a 40B parameter causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license.

For more information about this model, see here.

Licenses

All code in this repository is licensed under the Apache License 2.0 license.
Model code and weights are licensed under Apache 2.0 (see here).