Table of contents
Most language models are trained on huge datasets that make them very generalizable.
For example, Llama 1 was trained on a 1.4 trillion token dataset, including Wikipedia, millions of web pages via CommonCrawl, open source GitHub repos, and Stack Exchange posts1. Giant training datasets like this are a big part of why language models can do everything from writing code to answering historical questions to creating poems.
But generalized language models aren't good at everything, and that's where fine-tuning comes. Fine-tuning is the process of taking a pre-trained language model and training it on a smaller, more specific dataset. This allows you to customize the model to get better at a particular task.
For example, we might want to:
Some of these tasks can be accomplished by adjusting your prompt, but we'll always be limited by our context window. Fine-tuning allows us to improve our model without being limited by the context window.
The first step of fine-tuning is to collect your data. Generally, training data is in the format of a jsonl
text file, where each line is a JSON object with prompt
/completion
keys or text
key.
If you're building an instruction-tuned model, like a chat bot that answers questions, structure your data using an object with a prompt key and a completion key on each line:
If you’re building an autocompleting model to do tasks like completing a user’s writing, code completion, finishing lists, few-shotting specific tasks like classification, or if you want more control over the format of your training data, structure each JSON line as a single object with a text key and a string value:
For more on structuring your dataset for fine-tuning, check out our fine-tuning docs.
You can fine-tune open source models on lots of hosting providers, or on your own machine. We're biased, but we'd recommend Replicate 😉, but services like Google Colab and Brev.dev are also good options.
If you want to fine-tune a closed model like GPT-3.5, you'll need to use OpenAI's API.
On Replicate we have a fine-tuning guide that walks you through the process of fine-tuning a model on Replicate. If you want to fine tune on Colab, this notebook is a good starting point.
There are lots of parameters you can adjust when you fine-tune. On Replicate, every model that can be fine-tuned has a "Train" tab that lets you adjust these parameters. We try and set you with reasonable defaults.
For example, here are the parameters for Llama 2 7B:
{"text": ...}
or {"prompt": ..., "completion": ...}
. Must be JSONL.
num_train_epochs (optional, default=3): Number of epochs (iterations over the entire training dataset) to train for.{"text": ...}
or {"prompt": ..., "completion": ...}
or {"prompt": ..., "completion": ...}
. Must be JSONL.Want to try fine-tuning a model? Check out our fine-tuning guide to get started.