Earlier this month we announced a sync mode for our API, which makes it quicker and easier to get model output. On Oct 29, 2024, we made the difficult decision to roll back data URL responses as part of our sync API. This post explains why we've done this and what we're going to do next.
At the start of October, we began rolling out a sync mode for our API to make it easier for you to get model output quickly:
Prefer: wait HTTP header to the API, we wait for up to 60 seconds for the model to finish running, and send outputs straight back to the client if possible.From early October, any client could opt into the synchronous API (1) with a Prefer: wait HTTP header. The data URL responses (2) rolled out more slowly over the course of the month.
We added support for synchronous mode in the 1.0.0 releases of our JavaScript and Python client libraries, and made it the default behavior for the replicate.run methods in those libraries. We also added a FileOutput type that we hoped would make it easy to handle either https:// URLs or data URLs with the same code.
As we rolled out (2) later in October, many users told us that we'd broken their applications. Initially, models were returning https:// URLs and then later—as we turned on the relevant infrastructure—returning data URLs. Even people using our 1.x client libraries found things breaking, because it wasn't clear how to use the FileOutput objects, and some users were using the url properties of these objects to access the underlying HTTP URLs, which broke when those became data URLs.
Separately, people were also surprised when curl examples copied and pasted from our documentation resulted in swaths of base64-encoded data in their terminal. We hadn't given users the ability to opt out of this behavior.
We received these reports before we finished rolling out data URLs to all models. It was clear that continuing the rollout would create more problems for users.
On Oct 29, 2024, we rolled back data URL responses from the sync API. We knew this could break things for some customers, but it felt like our best option for minimizing breakage overall.
We had intended that all of these changes would be opt-in and not breaking changes. What we missed was the interaction between changes to our client libraries' default behavior (which opted into the new API) and the delayed rollout of data URLs in responses.
The following groups may have been impacted by the rollout:
1.x) versions of our JavaScript and Python client libraries and were using the replicate.run methods to run models that did not have data URLs enabled at the time of our 1.x releases.The following groups were not impacted by the rollout:
1.x client libraries.Prefer: wait HTTP header.replicate.predictions.create methods instead of replicate.run to run models.The following groups may have been impacted by the rollback:
First we want to be clear: we messed up this rollout. Sorry. We know this has to be better for a company that's all about APIs.
We also know some of you want data URL output in the sync API and we'll be working on how we can do this in a way that's better for everyone.
We've learned that even if our API changes are made in a way that is technically backwards compatible, we can still break customers' applications due to interactions between client libraries and the API. The promise made by our API was that it would return data URLs if it could (if the payload was under 5MB) and https:// URLs otherwise, but the actual behavior for most models throughout October was that they would only return https:// URLs, and our customers quite reasonably wrote code based on that behavior.
We also recognise that there is a broader challenge we need to address, which is how we communicate with our customers about changes to our APIs and client libraries. You can expect us to have things to share in future about how we version our API and keep you in the loop about what is changing.
Thank you for your understanding and your trust, and please get in touch if you have questions about any of this.