We're Jillian and Zeke, members of the team at Replicate. In this post, we’ll introduce some improvements we’re making to the way models are documented on Replicate.
In the world of open-source software, README files have become an indispensable part of every well-documented project. For most users, the README is the first thing you see. It's the starting point for learning about what a project is, how it works, and whether it's right for your needs.
In the sub-world of software that is machine learning, the plot thickens: Unlike traditional software, machine learning models are not just source code, but rather the result of a training process that includes both code and training data. With ML software, it's no longer possible to understand a program by simply "reading the source".
In 2019, researchers at Google published an academic paper proposing a framework called Model cards. Model cards are "short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, [phenotypic, or intersectional groups] that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information."
Model cards have become increasingly popular in recent years, and been adopted by organizations like Google, Meta, OpenAI, Hugging Face, and others.
Inspired by the concept of model cards, we've added a new feature to Replicate to make it easier for model authors to create better documentation. When you create a new model, the default README now includes a template with prompts for sections like "Intended use" and "Ethical considerations". The template is intended to be a standardized way for users to compare and understand the capabilities of models on Replicate.
The template consists of four sections in addition to a brief non-technical description. We’ll briefly describe each section here.
There are many choices made when designing and training a model. This section should include details about the model itself, including model architecture, training schemes, and model parameters.
There are certain uses that a model is designed to be used for, and there are other uses that a model isn’t designed to be used for. This section should enumerate those cases.
Related to intended uses, models can include biases and have certain limitations. This section should flag these limitations so that users can deploy the model appropriately.
This section includes various tips and tricks for working with the model, such as good parameter initialization schemes and properties of effective inputs.
We created this feature because we believe in making machine learning easily accessible and shareable. One important step to achieving our mission is to create documentation that is easy to understand.
This new template is live for new models on Replicate. Give it a try when you create your next model, and let us know what you think.
🤔 Do you have a model you'd like to publish to Replicate, but don't yet have publish access? Take this tiny survey to join our waitlist.