andreasjansson/wizardcoder-python-34b-v1-gguf – Run with an API on Replicate

andreasjansson / wizardcoder-python-34b-v1-gguf

WizardCoder-python-34B-v1.0 with support for grammars and jsonschema

Cold

Public
766 runs
L40S
GitHub
License

Run with an API

Playground API Examples README Versions

Input

prompt

*string

Shift + Return to add a new line

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write a short program that counts the number of occurrences of a character in a string.

### Response:Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write a short program that counts the number of occurrences of a character in a string.

### Response:

Prompt

grammar

string

Shift + Return to add a new line

root        ::= "```python\ndef num_occurrences(c: str, s: str) -> int:\n    " code "```"
code        ::= [^`()]+root        ::= "```python\ndef num_occurrences(c: str, s: str) -> int:\n    " code "```"
code        ::= [^`()]+

Grammar in GBNF format. Use either grammar or jsonschema.

jsonschema

string

Shift + Return to add a new line

JSON schema for the generated output. Use either grammar or jsonschema.

max_tokens

integer

Max number of tokens to return

Default: 500

temperature

number

Temperature

Default: 0.8

top_p

number

Top P

Default: 0.95

top_k

integer

Top K

Default: 10

frequency_penalty

number

(minimum: 0, maximum: 2)

Frequency penalty

Default: 0

presence_penalty

number

(minimum: 0, maximum: 2)

Presence

Default: 0

repeat_penalty

number

(minimum: 0, maximum: 2)

Repetition penalty

Default: 1.1

mirostat_mode

string

Mirostat sampling mode

Default: "Disabled"

mirostat_learning_rate

number

(minimum: 0, maximum: 1)

Mirostat learning rate, if mirostat_mode is not Disabled

Default: 0.1

mirostat_entropy

number

(minimum: 0, maximum: 10)

Mirostat target entropy

Default: 5

Run this model in Node.js with one line of code:

npx create-replicate --model=andreasjansson/wizardcoder-python-34b-v1-gguf

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run andreasjansson/wizardcoder-python-34b-v1-gguf using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "andreasjansson/wizardcoder-python-34b-v1-gguf:efd89f8301f7f5cdad054cbb63139c894ef64ea96dbc12b4c82a2c0c720430f8",
  {
    input: {
      top_k: 10,
      top_p: 0.95,
      prompt: "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a short program that counts the number of occurrences of a character in a string.\n\n### Response:",
      grammar: "root        ::= \"```python\\ndef num_occurrences(c: str, s: str) -> int:\\n    \" code \"```\"\ncode        ::= [^`()]+",
      max_tokens: 500,
      temperature: 0.8,
      mirostat_mode: "Disabled",
      repeat_penalty: 1.1,
      mirostat_entropy: 5,
      presence_penalty: 0,
      frequency_penalty: 0,
      mirostat_learning_rate: 0.1
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run andreasjansson/wizardcoder-python-34b-v1-gguf using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "andreasjansson/wizardcoder-python-34b-v1-gguf:efd89f8301f7f5cdad054cbb63139c894ef64ea96dbc12b4c82a2c0c720430f8",
    input={
        "top_k": 10,
        "top_p": 0.95,
        "prompt": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a short program that counts the number of occurrences of a character in a string.\n\n### Response:",
        "grammar": "root        ::= \"```python\\ndef num_occurrences(c: str, s: str) -> int:\\n    \" code \"```\"\ncode        ::= [^`()]+",
        "max_tokens": 500,
        "temperature": 0.8,
        "mirostat_mode": "Disabled",
        "repeat_penalty": 1.1,
        "mirostat_entropy": 5,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "mirostat_learning_rate": 0.1
    }
)

# The andreasjansson/wizardcoder-python-34b-v1-gguf model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/andreasjansson/wizardcoder-python-34b-v1-gguf/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run andreasjansson/wizardcoder-python-34b-v1-gguf using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "andreasjansson/wizardcoder-python-34b-v1-gguf:efd89f8301f7f5cdad054cbb63139c894ef64ea96dbc12b4c82a2c0c720430f8",
    "input": {
      "top_k": 10,
      "top_p": 0.95,
      "prompt": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\\n\\n### Instruction:\\nWrite a short program that counts the number of occurrences of a character in a string.\\n\\n### Response:",
      "grammar": "root        ::= \\"```python\\\\ndef num_occurrences(c: str, s: str) -> int:\\\\n    \\" code \\"```\\"\\ncode        ::= [^`()]+",
      "max_tokens": 500,
      "temperature": 0.8,
      "mirostat_mode": "Disabled",
      "repeat_penalty": 1.1,
      "mirostat_entropy": 5,
      "presence_penalty": 0,
      "frequency_penalty": 0,
      "mirostat_learning_rate": 0.1
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

```python def num_occurrences(c: str, s: str) -> int: """Counts the number of occurrences of a character c in a string s.""" count = 0 for char in s: if char == c: count += 1 return count ```

{
  "completed_at": "2023-09-22T22:10:46.157018Z",
  "created_at": "2023-09-22T22:10:42.257147Z",
  "data_removed": false,
  "error": null,
  "id": "s5oeuzlb6ij36miihdoy4mnj34",
  "input": {
    "top_k": 10,
    "top_p": 0.95,
    "prompt": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a short program that counts the number of occurrences of a character in a string.\n\n### Response:",
    "grammar": "root        ::= \"```python\\ndef num_occurrences(c: str, s: str) -> int:\\n    \" code \"```\"\ncode        ::= [^`()]+",
    "max_tokens": 500,
    "temperature": 0.8,
    "mirostat_mode": "Disabled",
    "repeat_penalty": 1.1,
    "mirostat_entropy": 5,
    "presence_penalty": 0,
    "frequency_penalty": 0,
    "mirostat_learning_rate": 0.1
  },
  "logs": "from_string grammar:\nroot ::= [`] [`] [`] [p] [y] [t] [h] [o] [n] [<U+000A>] [d] [e] [f] [ ] [n] [u] [m] [_] [o] [c] [c] [u] [r] [r] [e] [n] [c] [e] [s] [(] [c] [:] [ ] [s] [t] [r] [,] [ ] [s] [:] [ ] [s] [t] [r] [)] [ ] [-] [>] [ ] [i] [n] [t] [:] [<U+000A>] [ ] [ ] [ ] [ ] code [`] [`] [`] \ncode ::= code_2\ncode_2 ::= [^`()] code_2 | [^`()]\nPrompt:\nBelow is an instruction that describes a task. Write a response that appropriately completes the request.\n### Instruction:\nWrite a short program that counts the number of occurrences of a character in a string.\n### Response:\nLlama.generate: prefix-match hit\nllama_print_timings:        load time =   323.93 ms\nllama_print_timings:      sample time =   644.67 ms /    74 runs   (    8.71 ms per token,   114.79 tokens per second)\nllama_print_timings: prompt eval time =     0.00 ms /     1 tokens (    0.00 ms per token,      inf tokens per second)\nllama_print_timings:        eval time =  3161.00 ms /    74 runs   (   42.72 ms per token,    23.41 tokens per second)\nllama_print_timings:       total time =  3983.35 ms",
  "metrics": {
    "predict_time": 3.994199,
    "total_time": 3.899871
  },
  "output": [
    "```",
    "python",
    "\n",
    "def",
    " num",
    "_",
    "occ",
    "urr",
    "ences",
    "(",
    "c",
    ":",
    " str",
    ",",
    " s",
    ":",
    " str",
    ")",
    " ->",
    " int",
    ":",
    "\n",
    "   ",
    " \"\"\"",
    "Count",
    "s",
    " the",
    " number",
    " of",
    " occurr",
    "ences",
    " of",
    " a",
    " character",
    " c",
    " in",
    " a",
    " string",
    " s",
    ".\"",
    "\"\"",
    "\n",
    "   ",
    " count",
    " =",
    " ",
    "0",
    "\n",
    "   ",
    " for",
    " char",
    " in",
    " s",
    ":",
    "\n",
    "       ",
    " if",
    " char",
    " ==",
    " c",
    ":",
    "\n",
    "           ",
    " count",
    " +=",
    " ",
    "1",
    "\n",
    "   ",
    " return",
    " count",
    "\n",
    "```"
  ],
  "started_at": "2023-09-22T22:10:42.162819Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/s5oeuzlb6ij36miihdoy4mnj34",
    "cancel": "https://api.replicate.com/v1/predictions/s5oeuzlb6ij36miihdoy4mnj34/cancel"
  },
  "version": "b357a162acd41ce53c04b14a25df0f1d476c2e39d465895d28cdba87944506c6"
}

Generated in

4.0 seconds

Tweak it Report

This output was created using a different version of the model, andreasjansson/wizardcoder-python-34b-v1-gguf:b357a162.

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

This model doesn't have a readme.