baaivision/emu3-chat – Run with an API on Replicate

baaivision / emu3-chat

Emu3-Chat for vision-language understanding

Cold

Public
28 runs
L40S
GitHub
Weights
Paper
License

Iterate in playground

Run with an API

Playground API Examples README Versions

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=baaivision/emu3-chat

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run baaivision/emu3-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "baaivision/emu3-chat:52034986454b08979c0fcb855175ec26a2fd8d40b3ac90913275917c504ea7a4",
  {
    input: {
      text: "Please describe the image.",
      image: "https://replicate.delivery/pbxt/Li3FLacLbDi0oQJNea3ijGncsfWeCVZrnHZgbffYu6k3WZ3v/arch.png",
      top_p: 0.9,
      temperature: 0.7,
      max_new_tokens: 1024
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run baaivision/emu3-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "baaivision/emu3-chat:52034986454b08979c0fcb855175ec26a2fd8d40b3ac90913275917c504ea7a4",
    input={
        "text": "Please describe the image.",
        "image": "https://replicate.delivery/pbxt/Li3FLacLbDi0oQJNea3ijGncsfWeCVZrnHZgbffYu6k3WZ3v/arch.png",
        "top_p": 0.9,
        "temperature": 0.7,
        "max_new_tokens": 1024
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run baaivision/emu3-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "baaivision/emu3-chat:52034986454b08979c0fcb855175ec26a2fd8d40b3ac90913275917c504ea7a4",
    "input": {
      "text": "Please describe the image.",
      "image": "https://replicate.delivery/pbxt/Li3FLacLbDi0oQJNea3ijGncsfWeCVZrnHZgbffYu6k3WZ3v/arch.png",
      "top_p": 0.9,
      "temperature": 0.7,
      "max_new_tokens": 1024
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

The image depicts a graphical representation of a process involving a "Next-Token Prediction" system. The process is encapsulated within a rectangular box with rounded corners, which is labeled "Next-Token Prediction (Transformer Decoder)." The box is divided into two main sections: the upper section contains the title, while the lower section shows the steps involved in the process. The upper section of the box has a light green background with black text. The title reads "Next-Token Prediction (Transformer Decoder)" and is centered within the box. Below the title, there is an icon representing a person, which likely symbolizes the human element or user interface of the system. To the right of the person icon, there is an upward arrow, indicating an upward trend or progression. The lower section of the box has a darker green background with white text and icons. The text reads "Tokenize" and is positioned to the right of the person icon. Below the text, there is an icon representing a document or file, which suggests that the process involves tokenizing text data. The overall layout of the image is clean and organized, with clear demarcations between the different components of the process. ### Analysis and Description: 1. **Title and Title**: The title "Next-Token Prediction (Transformer Decoder)" indicates that the system is designed to predict next tokens in a sequence, likely for a natural language processing (NLP) task. The term "Transformer Decoder" suggests that this system uses a transformer architecture, which is a type of neural network architecture that has become popular in NLP for its ability to process sequential data. 2. **User Interface**: The icons and text are placed in a way that guides the user through the process. The person icon likely represents the user interface, while the upward arrow and document icon suggest the steps involved in tokenization and document handling. 3. **Steps Involved**: - **Tokenization**: The process of converting text data into tokens, which are the basic units of text in NLP models. - **Document Handling**: The system likely handles documents or files, which are used to store and manage the tokens. 4. **Design Elements**: The use of a rectangular box with rounded corners and a clear, organized layout makes the process easy to understand and navigate. The light green background for the title and the darker green background for the steps provides a visual distinction between the title and the steps, ensuring that the user can focus on the process without confusion. 5. **Functionality**: The system seems to be designed for efficiency and ease of use, as indicated by the clear and concise labeling of the steps. The use of a transformer architecture suggests that it is capable of handling complex text data, which is essential for NLP tasks. ### Conclusion: The image represents a detailed process for predicting next tokens in a sequence using a transformer-based transformer decoder system. The clear and organized layout, combined with the use of icons and text, makes it easy for users to understand and interact with the system. The steps involved in tokenization and document handling are clearly outlined, indicating a well-thought-out design for ease of use and efficiency.

{
  "completed_at": "2024-09-29T20:50:31.360849Z",
  "created_at": "2024-09-29T20:48:40.360000Z",
  "data_removed": false,
  "error": null,
  "id": "3mxe1z75n1rgp0cj7y5a5329mg",
  "input": {
    "text": "Please describe the image.",
    "image": "https://replicate.delivery/pbxt/Li3FLacLbDi0oQJNea3ijGncsfWeCVZrnHZgbffYu6k3WZ3v/arch.png",
    "top_p": 0.9,
    "temperature": 0.7,
    "max_new_tokens": 1024
  },
  "logs": "/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:567: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.7` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\nwarnings.warn(\n/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:572: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\nwarnings.warn(\nThe `seen_tokens` attribute is deprecated and will be removed in v4.41. Use the `cache_position` model input instead.",
  "metrics": {
    "predict_time": 32.914005215,
    "total_time": 111.000849
  },
  "output": " The image depicts a graphical representation of a process involving a \"Next-Token Prediction\" system. The process is encapsulated within a rectangular box with rounded corners, which is labeled \"Next-Token Prediction (Transformer Decoder).\" The box is divided into two main sections: the upper section contains the title, while the lower section shows the steps involved in the process.\n\nThe upper section of the box has a light green background with black text. The title reads \"Next-Token Prediction (Transformer Decoder)\" and is centered within the box. Below the title, there is an icon representing a person, which likely symbolizes the human element or user interface of the system. To the right of the person icon, there is an upward arrow, indicating an upward trend or progression.\n\nThe lower section of the box has a darker green background with white text and icons. The text reads \"Tokenize\" and is positioned to the right of the person icon. Below the text, there is an icon representing a document or file, which suggests that the process involves tokenizing text data. The overall layout of the image is clean and organized, with clear demarcations between the different components of the process.\n\n### Analysis and Description:\n\n1. **Title and Title**: The title \"Next-Token Prediction (Transformer Decoder)\" indicates that the system is designed to predict next tokens in a sequence, likely for a natural language processing (NLP) task. The term \"Transformer Decoder\" suggests that this system uses a transformer architecture, which is a type of neural network architecture that has become popular in NLP for its ability to process sequential data.\n\n2. **User Interface**: The icons and text are placed in a way that guides the user through the process. The person icon likely represents the user interface, while the upward arrow and document icon suggest the steps involved in tokenization and document handling.\n\n3. **Steps Involved**:\n   - **Tokenization**: The process of converting text data into tokens, which are the basic units of text in NLP models.\n   - **Document Handling**: The system likely handles documents or files, which are used to store and manage the tokens.\n\n4. **Design Elements**: The use of a rectangular box with rounded corners and a clear, organized layout makes the process easy to understand and navigate. The light green background for the title and the darker green background for the steps provides a visual distinction between the title and the steps, ensuring that the user can focus on the process without confusion.\n\n5. **Functionality**: The system seems to be designed for efficiency and ease of use, as indicated by the clear and concise labeling of the steps. The use of a transformer architecture suggests that it is capable of handling complex text data, which is essential for NLP tasks.\n\n### Conclusion:\n\nThe image represents a detailed process for predicting next tokens in a sequence using a transformer-based transformer decoder system. The clear and organized layout, combined with the use of icons and text, makes it easy for users to understand and interact with the system. The steps involved in tokenization and document handling are clearly outlined, indicating a well-thought-out design for ease of use and efficiency.",
  "started_at": "2024-09-29T20:49:58.446844Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/3mxe1z75n1rgp0cj7y5a5329mg",
    "cancel": "https://api.replicate.com/v1/predictions/3mxe1z75n1rgp0cj7y5a5329mg/cancel"
  },
  "version": "52034986454b08979c0fcb855175ec26a2fd8d40b3ac90913275917c504ea7a4"
}

Generated in

32.9 seconds

Tweak it Share Report

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Emu3: Next-Token Prediction is All You Need

This is the demo for vision-language understanding. For image generation, visit: https://replicate.com/chenxwh/emu3-gen

arch.

We introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction! By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences.

Emu3 excels in both generation and perception

Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship open models such as SDXL, LLaVA-1.6 and OpenSora-1.2, while eliminating the need for diffusion or compositional architectures.

comparison.

Highlights

Emu3 is capable of generating high-quality images following the text input, by simply predicting the next vision token. The model naturally supports flexible resolutions and styles.
Emu3 shows strong vision-language understanding capabilities to see the physical world and provides coherent text responses. Notably, this capability is achieved without depending on a CLIP and a pretrained LLM.
Emu3 simply generates a video causally by predicting the next token in a video sequence, unlike the video diffusion model as in Sora. With a video in context, Emu3 can also naturally extend the video and predict what will happen next.