tomasmcm/docsgpt-7b-mistral – Run with an API on Replicate

tomasmcm / docsgpt-7b-mistral

Source: Arc53/docsgpt-7b-mistral ✦ Quant: TheBloke/docsgpt-7B-mistral-AWQ ✦ DocsGPT is optimized for Documentation (RAG), fine-tuned for providing answers that are based on context

Cold

Public
74 runs
L40S
Paper
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

### Instruction
When was Aquaman and the Lost Kingdom released?
### Context
Aquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.
### Answer### Instruction
When was Aquaman and the Lost Kingdom released?
### Context
Aquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.
### Answer

Text prompt to send to the model.

max_tokens

integer

Maximum number of tokens to generate per output sequence.

Default: 128

presence_penalty

number

(minimum: -5, maximum: 5)

Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

Default: 0

frequency_penalty

number

(minimum: -5, maximum: 5)

Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

Default: 0

temperature

number

(minimum: 0.01, maximum: 5)

Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.

Default: 0.8

top_p

number

(minimum: 0.01, maximum: 1)

Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.

Default: 0.95

top_k

integer

Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.

Default: -1

stop

string

Shift + Return to add a new line

List of strings that stop the generation when they are generated. The returned output will not contain the stop strings.

Run this model in Node.js with one line of code:

npx create-replicate --model=tomasmcm/docsgpt-7b-mistral

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run tomasmcm/docsgpt-7b-mistral using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "tomasmcm/docsgpt-7b-mistral:2f13e3048eb30d468bfd30bdbdc403c9d7fc4d90ccd76b6f0674ba5cb1c70f0e",
  {
    input: {
      top_k: -1,
      top_p: 0.95,
      prompt: "### Instruction\nWhen was Aquaman and the Lost Kingdom released?\n### Context\nAquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.\n### Answer",
      max_tokens: 128,
      temperature: 0.8,
      presence_penalty: 0,
      frequency_penalty: 0
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run tomasmcm/docsgpt-7b-mistral using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "tomasmcm/docsgpt-7b-mistral:2f13e3048eb30d468bfd30bdbdc403c9d7fc4d90ccd76b6f0674ba5cb1c70f0e",
    input={
        "top_k": -1,
        "top_p": 0.95,
        "prompt": "### Instruction\nWhen was Aquaman and the Lost Kingdom released?\n### Context\nAquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.\n### Answer",
        "max_tokens": 128,
        "temperature": 0.8,
        "presence_penalty": 0,
        "frequency_penalty": 0
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run tomasmcm/docsgpt-7b-mistral using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "2f13e3048eb30d468bfd30bdbdc403c9d7fc4d90ccd76b6f0674ba5cb1c70f0e",
    "input": {
      "top_k": -1,
      "top_p": 0.95,
      "prompt": "### Instruction\\nWhen was Aquaman and the Lost Kingdom released?\\n### Context\\nAquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.\\n### Answer",
      "max_tokens": 128,
      "temperature": 0.8,
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/tomasmcm/docsgpt-7b-mistral@sha256:2f13e3048eb30d468bfd30bdbdc403c9d7fc4d90ccd76b6f0674ba5cb1c70f0e \
  -i 'top_k=-1' \
  -i 'top_p=0.95' \
  -i $'prompt="### Instruction\\nWhen was Aquaman and the Lost Kingdom released?\\n### Context\\nAquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.\\n### Answer"' \
  -i 'max_tokens=128' \
  -i 'temperature=0.8' \
  -i 'presence_penalty=0' \
  -i 'frequency_penalty=0'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/tomasmcm/docsgpt-7b-mistral@sha256:2f13e3048eb30d468bfd30bdbdc403c9d7fc4d90ccd76b6f0674ba5cb1c70f0e
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "top_k": -1,
      "top_p": 0.95,
      "prompt": "### Instruction\\nWhen was Aquaman and the Lost Kingdom released?\\n### Context\\nAquaman and the Lost Kingdom premiered at a fan event at the Grove, Los Angeles on December 19, 2023, and was released in the United States on December 22, by Warner Bros. Pictures. The film received mixed reviews from critics, who deemed it inferior to its predecessor. The film has grossed $138 million worldwide.\\n### Answer",
      "max_tokens": 128,
      "temperature": 0.8,
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

Aquaman and the Lost Kingdom was released on December 22, 2023, in the United States by Warner Bros. Pictures.

Generated in

0.7 seconds

Tweak it Report

Run time and cost

This model costs approximately $0.0037 to run on Replicate, or 270 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 4 seconds. The predict time for this model varies significantly based on the inputs.

Readme

DocsGPT is optimized for Documentation (RAG optimised): Specifically fine-tuned for providing answers that are based on context, making it particularly useful for developers and technical support teams. We used the Lora fine tuning process. This model is fine tuned on top of zephyr-7b-beta

It’s an apache-2.0 license so you can use it for commercial purposes too.

Benchmarks:

Bacon: The BACON test is an internal assessment designed to evaluate the capabilities of neural networks in handling questions with substantial content. It focuses on testing the model’s understanding of context-driven queries, as well as its tendency for hallucination and attention span. The questions in both parts are carefully crafted, drawing from diverse sources such as scientific papers, complex code problems, and instructional prompts, providing a comprehensive test of the model’s ability to process and generate information in various domains. | Model | Score | |------------------------------|-------| | gpt-4 | 8.74 | | DocsGPT-7b-Mistral | 8.64 | | gpt-3.5-turbo | 8.42 | | zephyr-7b-beta | 8.37 | | neural-chat-7b-v3-1 | 7.88 | | Mistral-7B-Instruct-v0.1 | 7.44 | | openinstruct-mistral-7b | 5.86 | | llama-2-13b | 2.29 |

image/png

MTbench with llm judge:

image/png

#### First turn

Model	Turn	Score
gpt-4	1	8.956250
gpt-3.5-turbo	1	8.075000
DocsGPT-7b-Mistral	1	7.593750
zephyr-7b-beta	1	7.412500
vicuna-13b-v1.3	1	6.812500
alpaca-13b	1	4.975000
deepseek-coder-6.7b	1	4.506329

#### Second turn

Model	Turn	Score
gpt-4	2	9.025000
gpt-3.5-turbo	2	7.812500
DocsGPT-7b-Mistral	2	6.740000
zephyr-7b-beta	2	6.650000
vicuna-13b-v1.3	2	5.962500
deepseek-coder-6.7b	2	5.025641
alpaca-13b	2	4.087500

#### Average

Model	Score
gpt-4	8.990625
gpt-3.5-turbo	7.943750
DocsGPT-7b-Mistral	7.166875
zephyr-7b-beta	7.031250
vicuna-13b-v1.3	6.387500
deepseek-coder-6.7b	4.764331
alpaca-13b	4.531250

To prepare your prompts make sure you keep this format:

### Instruction
(where the question goes)
### Context
(your document retrieval + system instructions)
### Answer