Official

ibm-granite / granite-20b-code-instruct-8k

Join the Granite community where you can find numerous recipe workbooks to help you get started with a wide variety of use cases using this model. https://github.com/ibm-granite-community

Cold

Public
110K runs
Priced per token
GitHub
Paper
License

Run with an API

Playground API Examples README

Input

prompt

string

Shift + Return to add a new line

Could you please explain what APR means?Could you please explain what APR means?

Prompt

Default: ""

system_prompt

string

Shift + Return to add a new line

You are an expert in finance that knows many concepts related to loans and credit.You are an expert in finance that knows many concepts related to loans and credit.

System prompt to send to the model. This is prepended to the prompt and helps guide system behavior. Ignored for non-chat models.

Default: "You are a helpful assistant."

min_tokens

integer

The minimum number of tokens the model should generate as output.

Default: 0

max_tokens

integer

The maximum number of tokens the model should generate as output.

Default: 512

temperature

number

The value used to modulate the next token probabilities.

Default: 0.6

top_p

number

A probability threshold for generating the output. If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751).

Default: 0.9

top_k

integer

The number of highest probability tokens to consider for generating the output. If > 0, only keep the top k tokens with highest probability (top-k filtering).

Default: 50

presence_penalty

number

Presence penalty

Default: 0

frequency_penalty

number

Frequency penalty

Default: 0

stop_sequences

string

Shift + Return to add a new line

A comma-separated list of sequences to stop generation at. For example, '<end>,<stop>' will stop generation at the first instance of 'end' or '<stop>'.

prompt_template

string

Shift + Return to add a new line

A template to format the prompt with. If not provided, the default prompt template will be used.

seed

integer

Random seed. Leave blank to randomize the seed.

Run this model in Node.js with one line of code:

npx create-replicate --model=ibm-granite/granite-20b-code-instruct-8k

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run ibm-granite/granite-20b-code-instruct-8k using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const input = {
  top_k: 50,
  top_p: 0.9,
  prompt: "Could you please explain what APR means?",
  max_tokens: 512,
  min_tokens: 0,
  temperature: 0.6,
  system_prompt: "You are an expert in finance that knows many concepts related to loans and credit.",
  presence_penalty: 0,
  frequency_penalty: 0
};

for await (const event of replicate.stream("ibm-granite/granite-20b-code-instruct-8k", { input })) {
  process.stdout.write(event.toString());
};

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run ibm-granite/granite-20b-code-instruct-8k using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

# The ibm-granite/granite-20b-code-instruct-8k model can stream output as it's running.
for event in replicate.stream(
    "ibm-granite/granite-20b-code-instruct-8k",
    input={
        "top_k": 50,
        "top_p": 0.9,
        "prompt": "Could you please explain what APR means?",
        "max_tokens": 512,
        "min_tokens": 0,
        "temperature": 0.6,
        "system_prompt": "You are an expert in finance that knows many concepts related to loans and credit.",
        "presence_penalty": 0,
        "frequency_penalty": 0
    },
):
    print(str(event), end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run ibm-granite/granite-20b-code-instruct-8k using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "input": {
      "top_k": 50,
      "top_p": 0.9,
      "prompt": "Could you please explain what APR means?",
      "max_tokens": 512,
      "min_tokens": 0,
      "temperature": 0.6,
      "system_prompt": "You are an expert in finance that knows many concepts related to loans and credit.",
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/models/ibm-granite/granite-20b-code-instruct-8k/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

APR stands for Annual Percentage Rate. It is a figure that represents the annual cost of borrowing, including fees and interest. It is used to help consumers compare the costs of different loans and credit products. The higher the APR, the more expensive the loan is.

{
  "completed_at": "2024-10-17T16:34:22.167598Z",
  "created_at": "2024-10-17T16:34:20.985000Z",
  "data_removed": false,
  "error": null,
  "id": "4rfr3bw6q5rm00cjkda88c6j6c",
  "input": {
    "top_p": 0.9,
    "prompt": "Could you please explain what APR means?",
    "max_tokens": 512,
    "min_tokens": 0,
    "temperature": 0.6,
    "system_prompt": "You are an expert in finance that knows many concepts related to loans and credit.",
    "presence_penalty": 0,
    "frequency_penalty": 0
  },
  "logs": "INFO 10-17 16:34:20 async_llm_engine.py:173] Added request f21156ed57234d8585b9b09777df8582.\nINFO 10-17 16:34:21 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.\nGeneration took 1729182595.16sFormatted prompt: System:\nYou are an expert in finance that knows many concepts related to loans and credit.\n\nQuestion:\nCould you please explain what APR means?\n\nAnswer:\nRandom seed used: `99346`\nNote: Random seed will not impact output if greedy decoding is used.\nINFO 10-17 16:34:22 async_llm_engine.py:140] Finished request f21156ed57234d8585b9b09777df8582.",
  "metrics": {
    "batch_size": 1.0006387106010313,
    "input_token_count": 39,
    "tokens_per_second": 49.61475799941566,
    "output_token_count": 58,
    "predict_time_share": 1.1756401062011719,
    "predict_time": 1.177441129,
    "total_time": 1.182598,
    "time_to_first_token": 0.013591117
  },
  "output": [
    "AP",
    "R",
    " stands",
    " for",
    " Ann",
    "ual",
    " Per",
    "centage",
    " Rate",
    ".",
    " It",
    " is",
    " a",
    " figure",
    " that",
    " represents",
    " the",
    " annual",
    " cost",
    " of",
    " borrow",
    "ing",
    ",",
    " including",
    " fees",
    " and",
    " interest",
    ".",
    " It",
    " is",
    " used",
    " to",
    " help",
    " consumers",
    " compare",
    " the",
    " costs",
    " of",
    " different",
    " lo",
    "ans",
    " and",
    " credit",
    " products",
    ".",
    " The",
    " higher",
    " the",
    " APR",
    ",",
    " the",
    " more",
    " expensive",
    " the",
    " loan",
    " is",
    ".",
    ""
  ],
  "started_at": "2024-10-17T16:34:20.990157Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://stream-b.svc.sea.v.replicate.net/v1/streams/wuruoxlkb3th2ds7dk5gjumk7mxjxrclvctwd6lajtfn25f4bg3q",
    "get": "https://api.replicate.com/v1/predictions/4rfr3bw6q5rm00cjkda88c6j6c",
    "cancel": "https://api.replicate.com/v1/predictions/4rfr3bw6q5rm00cjkda88c6j6c/cancel"
  },
  "version": "hidden"
}

Generated in

1.2 seconds

Input tokens

Output tokens

Tokens per second

49.61 tokens / second

Time to first token

14 milliseconds

Tweak it Share Report

Pricing

This language model is priced by how many input tokens are sent as inputs and how many output tokens are generated.

Type	Per unit	Per $1
Input	$0.10 / 1M tokens or 10M tokens / $1
Output	$0.50 / 1M tokens or 2M tokens / $1

For example, for $10 you can run around 28,571 predictions where the input is a sentence or two (15 tokens) and the output is a few paragraphs (700 tokens).

Check out our docs for more information about how per-token pricing works on Replicate.

Readme

Granite Code Models

Introduction to Granite Code Models

We introduce the Granite series of decoder-only code models for code generative tasks (e.g., fixing bugs, explaining code, documenting code), trained with code written in 116 programming languages. A comprehensive evaluation of the Granite Code model family on diverse tasks demonstrates that our models consistently reach state-of-the-art performance among available open source code LLMs.

The key advantages of Granite Code models include:

All-rounder Code LLM: Granite Code models achieve competitive or state-of-the-art performance on different kinds of code-related tasks, including code generation, explanation, fixing, editing, translation, and more. Demonstrating their ability to solve diverse coding tasks.
Trustworthy Enterprise-Grade LLM: All our models are trained on license-permissible data collected following IBM’s AI Ethics principles and guided by IBM’s Corporate Legal team for trustworthy enterprise usage. We release all our Granite Code models under an Apache 2.0 license license for research and commercial use.

The family of Granite Code Models comes in two main variants:

Granite Code Base Models: base foundational models designed for code-related tasks (e.g., code repair, code explanation, code synthesis).
Granite Code Instruct Models: instruction following models finetuned using a combination of Git commits paired with human instructions and open source synthetically generated code instruction datasets.

Both base and instruct models are available in sizes of 3B, 8B, 20B, and 34B parameters.

Data Collection

Our process to prepare code pretraining data involves several stages. First, we collect a combination of publicly available datasets (e.g., GitHub Code Clean, Starcoder data), public code repositories, and issues from GitHub. Second, we filter the code data collected based on the programming language in which data is written (which we determined based on file extension). Then, we also filter out data with low code quality. Third, we adopt an aggressive deduplication strategy that includes both exact and fuzzy deduplication to remove documents having (near) identical code content. Finally, we apply a HAP content filter that reduces models’ likelihood of generating hateful, abusive, or profane language. We also make sure to redact Personally Identifiable Information (PII) by replacing PII content (e.g., names, email addresses, keys, passwords) with corresponding tokens (e.g., ⟨NAME⟩, ⟨EMAIL⟩, ⟨KEY⟩, ⟨PASSWORD⟩). We also scan all datasets using ClamAV to identify and remove instances of malware in the source code. In addition to collecting code data for model training, we curate several publicly available high-quality natural language datasets for improving the model’s proficiency in language understanding and mathematical reasoning.

Pretraining

The Granite Code Base models are trained on 3-4T tokens of code data and natural language datasets related to code. Data is tokenized via byte pair encoding (BPE), employing the same tokenizer as StarCoder. We utilize high-quality data with two phases of training as follows:

Phase 1 (code only training): During phase 1, 3B and 8B models are trained for 4 trillion tokens of code data comprising 116 languages. The 20B parameter model is trained on 3 trillion tokens of code. The 34B model is trained on 1.4T tokens after the depth upscaling which is done on the 1.6T checkpoint of 20B model.
Phase 2 (code + language training): In phase 2, we include additional high-quality publicly available data from various domains, including technical, mathematics, and web documents, to further improve the model’s performance. We train all our models for 500B tokens (80% code-20% language mixture) in phase 2 training.

Instruction Tuning

Granite Code Instruct models are finetuned on the following types of instruction data: 1) code commits sourced from CommitPackFT, 2) high-quality math datasets, specifically we used MathInstruct and MetaMathQA, 3) Code instruction datasets such as Glaive-Code-Assistant-v3, Self-OSS-Instruct-SC2, Glaive-Function-Calling-v2, NL2SQL11 and a small collection of synthetic API calling datasets, and 4) high-quality language instruction datasets such as HelpSteer and an open license-filtered version of Platypus.

Evaluation Results

We conduct an extensive evaluation of our code models on a comprehensive list of benchmarks that includes but is not limited to HumanEvalPack, MBPP, and MBPP+. This set of benchmarks encompasses different coding tasks across commonly used programming languages (e.g., Python, JavaScript, Java, Go, C++, Rust).

Our findings reveal that Granite Code models outperform strong open source models across model sizes. We provide evaluation results in our paper.

Granite Community

Join the Granite community, where you can find numerous recipes to help you get started in a wide variety of use cases.

How to Contribute to this Project?

Plese check our Guidelines and Code of Conduct to contribute to our project.

License

All Granite Code Models are distributed under Apache 2.0 license.