tomasmcm/neural-chat-7b-v3-1 – Run with an API on Replicate

tomasmcm / neural-chat-7b-v3-1

Source: Intel/neural-chat-7b-v3-1 ✦ Quant: TheBloke/neural-chat-7B-v3-1-AWQ ✦ Fine-tuned model based on mistralai/Mistral-7B-v0.1

Cold

Public
775 runs
L40S
Paper
License

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

### System:
- You are a helpful assistant chatbot trained by Intel.
- You answer questions.
- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>
### User: What are the top five must-visit attractions in Shanghai?
### Assistant: ### System:
- You are a helpful assistant chatbot trained by Intel.
- You answer questions.
- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>
### User: What are the top five must-visit attractions in Shanghai?
### Assistant: 

Text prompt to send to the model.

max_tokens

integer

Maximum number of tokens to generate per output sequence.

Default: 128

presence_penalty

number

(minimum: -5, maximum: 5)

Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

Default: 0

frequency_penalty

number

(minimum: -5, maximum: 5)

Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

Default: 0

temperature

number

(minimum: 0.01, maximum: 5)

Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.

Default: 0.8

top_p

number

(minimum: 0.01, maximum: 1)

Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.

Default: 0.95

top_k

integer

Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.

Default: -1

stop

string

Shift + Return to add a new line

List of strings that stop the generation when they are generated. The returned output will not contain the stop strings.

Run this model in Node.js with one line of code:

npx create-replicate --model=tomasmcm/neural-chat-7b-v3-1

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run tomasmcm/neural-chat-7b-v3-1 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "tomasmcm/neural-chat-7b-v3-1:acb450496b49e19a1e410b50c574a34acacd54820bc36c19cbfe05148de2ba57",
  {
    input: {
      top_k: -1,
      top_p: 0.95,
      prompt: "### System:\n- You are a helpful assistant chatbot trained by Intel.\n- You answer questions.\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>\n### User: What are the top five must-visit attractions in Shanghai?\n### Assistant: ",
      max_tokens: 128,
      temperature: 0.8,
      presence_penalty: 0,
      frequency_penalty: 0
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run tomasmcm/neural-chat-7b-v3-1 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "tomasmcm/neural-chat-7b-v3-1:acb450496b49e19a1e410b50c574a34acacd54820bc36c19cbfe05148de2ba57",
    input={
        "top_k": -1,
        "top_p": 0.95,
        "prompt": "### System:\n- You are a helpful assistant chatbot trained by Intel.\n- You answer questions.\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>\n### User: What are the top five must-visit attractions in Shanghai?\n### Assistant: ",
        "max_tokens": 128,
        "temperature": 0.8,
        "presence_penalty": 0,
        "frequency_penalty": 0
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run tomasmcm/neural-chat-7b-v3-1 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "tomasmcm/neural-chat-7b-v3-1:acb450496b49e19a1e410b50c574a34acacd54820bc36c19cbfe05148de2ba57",
    "input": {
      "top_k": -1,
      "top_p": 0.95,
      "prompt": "### System:\\n- You are a helpful assistant chatbot trained by Intel.\\n- You answer questions.\\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>\\n### User: What are the top five must-visit attractions in Shanghai?\\n### Assistant: ",
      "max_tokens": 128,
      "temperature": 0.8,
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/tomasmcm/neural-chat-7b-v3-1@sha256:acb450496b49e19a1e410b50c574a34acacd54820bc36c19cbfe05148de2ba57 \
  -i 'top_k=-1' \
  -i 'top_p=0.95' \
  -i $'prompt="### System:\\n- You are a helpful assistant chatbot trained by Intel.\\n- You answer questions.\\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>\\n### User: What are the top five must-visit attractions in Shanghai?\\n### Assistant: "' \
  -i 'max_tokens=128' \
  -i 'temperature=0.8' \
  -i 'presence_penalty=0' \
  -i 'frequency_penalty=0'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/tomasmcm/neural-chat-7b-v3-1@sha256:acb450496b49e19a1e410b50c574a34acacd54820bc36c19cbfe05148de2ba57
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "top_k": -1,
      "top_p": 0.95,
      "prompt": "### System:\\n- You are a helpful assistant chatbot trained by Intel.\\n- You answer questions.\\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>\\n### User: What are the top five must-visit attractions in Shanghai?\\n### Assistant: ",
      "max_tokens": 128,
      "temperature": 0.8,
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

In Shanghai, there are numerous must-visit attractions that cater to various interests. Here's a list of top five attractions you simply cannot miss: 1. The Bund: A famous waterfront promenade lined with beautiful colonial architecture. 2. Yuyuan Garden: A historic garden filled with traditional Chinese elements, rich in history and culture. 3. Shanghai Disney Resort: A stunning amusement park with themed areas, shows, and entertainment. 4. Jade Buddha Temple: A renowned Buddhist temple featuring exquisite jade Buddha statues. 5.

{
  "completed_at": "2023-11-18T12:48:51.969625Z",
  "created_at": "2023-11-18T12:47:39.330133Z",
  "data_removed": false,
  "error": null,
  "id": "dgbf7z3btypfzalhyl642tkq4e",
  "input": {
    "top_k": -1,
    "top_p": 0.95,
    "prompt": "### System:\n- You are a helpful assistant chatbot trained by Intel.\n- You answer questions.\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</s>\n### User: What are the top five must-visit attractions in Shanghai?\n### Assistant: ",
    "max_tokens": 128,
    "temperature": 0.8,
    "presence_penalty": 0,
    "frequency_penalty": 0
  },
  "logs": "Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s]\nProcessed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.63s/it]\nProcessed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.63s/it]\nGenerated 128 tokens in 1.629446268081665 seconds.",
  "metrics": {
    "predict_time": 1.692145,
    "total_time": 72.639492
  },
  "output": "\n In Shanghai, there are numerous must-visit attractions that cater to various interests. Here's a list of top five attractions you simply cannot miss:\n\n1. The Bund: A famous waterfront promenade lined with beautiful colonial architecture.\n2. Yuyuan Garden: A historic garden filled with traditional Chinese elements, rich in history and culture.\n3. Shanghai Disney Resort: A stunning amusement park with themed areas, shows, and entertainment.\n4. Jade Buddha Temple: A renowned Buddhist temple featuring exquisite jade Buddha statues.\n5.",
  "started_at": "2023-11-18T12:48:50.277480Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/dgbf7z3btypfzalhyl642tkq4e",
    "cancel": "https://api.replicate.com/v1/predictions/dgbf7z3btypfzalhyl642tkq4e/cancel"
  },
  "version": "acb450496b49e19a1e410b50c574a34acacd54820bc36c19cbfe05148de2ba57"
}

Generated in

1.7 seconds

Tweak it Report

Run time and cost

This model costs approximately $0.0020 to run on Replicate, or 500 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 3 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Fine-tuning on Habana Gaudi2

This model is a fine-tuned model based on mistralai/Mistral-7B-v0.1 on the open source dataset Open-Orca/SlimOrca. Then we align it with DPO algorithm. For more details, you can refer our blog: The Practice of Supervised Fine-tuning and Direct Preference Optimization on Habana Gaudi2.

Model date

Neural-chat-7b-v3-1 was trained between September and October, 2023.

Evaluation

We submit our model to open_llm_leaderboard, and the model performance has been improved significantly as we see from the average metric of 7 tasks from the leaderboard.

Model	Average ⬆️	ARC (25-s) ⬆️	HellaSwag (10-s) ⬆️	MMLU (5-s) ⬆️	TruthfulQA (MC) (0-s) ⬆️	Winogrande (5-s)	GSM8K (5-s)	DROP (3-s)
mistralai/Mistral-7B-v0.1	50.32	59.58	83.31	64.16	42.15	78.37	18.12	6.14
Intel/neural-chat-7b-v3	57.31	67.15	83.29	62.26	58.77	78.06	1.21	50.43
Intel/neural-chat-7b-v3-1	59.06	66.21	83.64	62.37	59.65	78.14	19.56	43.84

Training procedure

Training hyperparameters

The following hyperparameters were used during training: - learning_rate: 1e-04 - train_batch_size: 1 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-HPU - num_devices: 8 - gradient_accumulation_steps: 8 - total_train_batch_size: 64 - total_eval_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_ratio: 0.03 - num_epochs: 2.0

Prompt Template

### System:
{system}
### User:
{usr}
### Assistant:

Inference with transformers

import transformers
model = transformers.AutoModelForCausalLM.from_pretrained(
  'Intel/neural-chat-7b-v3-1'
)

Ethical Considerations and Limitations

neural-chat-7b-v3-1 can produce factually incorrect output, and should not be relied on to produce factually accurate information. neural-chat-7b-v3-1 was trained on Open-Orca/SlimOrca based on mistralai/Mistral-7B-v0.1. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of neural-chat-7b-v3-1, developers should perform safety testing.

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.

Organizations developing the model

The NeuralChat team with members from Intel/SATG/AIA/AIPT. Core team members: Kaokao Lv, Liang Lv, Chang Wang, Wenxin Zhang, Xuhui Ren, and Haihao Shen.

Useful links

Intel Neural Compressor link
Intel Extension for Transformers link