Input

Run this model in Node.js with one line of code:

npx create-replicate --model=lucataco/apollo-3b

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run lucataco/apollo-3b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "lucataco/apollo-3b:ed5ec3233f7a4331a2b0d33b8a02e62057a335e1a03107d56db9fac10cf3ce55",
  {
    input: {
      top_p: 0.7,
      video: "https://replicate.delivery/pbxt/M9jsPwn8UqBPUCM6xiFEGcyfaZWXDwJ6EwcVqLPjKzcmctgn/replicate-prediction-jv0zykaqvhrmc0ckt4vtepaep4.mp4",
      prompt: "Describe this video in detail",
      temperature: 0.4,
      max_new_tokens: 256
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run lucataco/apollo-3b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "lucataco/apollo-3b:ed5ec3233f7a4331a2b0d33b8a02e62057a335e1a03107d56db9fac10cf3ce55",
    input={
        "top_p": 0.7,
        "video": "https://replicate.delivery/pbxt/M9jsPwn8UqBPUCM6xiFEGcyfaZWXDwJ6EwcVqLPjKzcmctgn/replicate-prediction-jv0zykaqvhrmc0ckt4vtepaep4.mp4",
        "prompt": "Describe this video in detail",
        "temperature": 0.4,
        "max_new_tokens": 256
    }
)

print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run lucataco/apollo-3b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "lucataco/apollo-3b:ed5ec3233f7a4331a2b0d33b8a02e62057a335e1a03107d56db9fac10cf3ce55",
    "input": {
      "top_p": 0.7,
      "video": "https://replicate.delivery/pbxt/M9jsPwn8UqBPUCM6xiFEGcyfaZWXDwJ6EwcVqLPjKzcmctgn/replicate-prediction-jv0zykaqvhrmc0ckt4vtepaep4.mp4",
      "prompt": "Describe this video in detail",
      "temperature": 0.4,
      "max_new_tokens": 256
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

The video features a lone astronaut in a white spacesuit, equipped with a helmet and gloves, standing on the moon's surface. The backdrop is dominated by a large, detailed image of the moon, set against a starry space. The astronaut begins to run across the lunar terrain, leaving footprints behind. As he runs, the camera angle shifts to reveal more of the moon's rugged landscape. The astronaut continues his run until he reaches the edge of the frame, where he leaps into the vast expanse of space, floating away from the moon. Throughout the sequence, the moon remains a constant and prominent feature in the background, emphasizing the astronaut's journey into the cosmos. The video captures the astronaut's solitary trek across the moon's surface and his subsequent leap into the unknown, symbolizing humanity's boundless curiosity and spirit of exploration. The astronaut's actions are depicted with precision and grace, highlighting the beauty and isolation of space travel. The video concludes with the astronaut floating freely in space, surrounded by the endless void of space, underscoring the awe-inspiring scale and mystery of the universe. The astronaut's journey serves as a powerful metaphor for human ambition and the quest for knowledge, encapsulating the essence of space exploration. The

{
  "completed_at": "2024-12-16T20:53:39.619257Z",
  "created_at": "2024-12-16T20:52:41.083000Z",
  "data_removed": false,
  "error": null,
  "id": "65eyxmwszdrm80ckt4zajjt3rm",
  "input": {
    "top_p": 0.7,
    "video": "https://replicate.delivery/pbxt/M9jsPwn8UqBPUCM6xiFEGcyfaZWXDwJ6EwcVqLPjKzcmctgn/replicate-prediction-jv0zykaqvhrmc0ckt4vtepaep4.mp4",
    "prompt": "Describe this video in detail",
    "temperature": 0.4,
    "max_new_tokens": 256
  },
  "logs": "/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\nwarnings.warn(\n/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/torch/utils/checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None\nwarnings.warn(\nThe attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.",
  "metrics": {
    "predict_time": 4.283875997,
    "total_time": 58.536257
  },
  "output": "The video features a lone astronaut in a white spacesuit, equipped with a helmet and gloves, standing on the moon's surface. The backdrop is dominated by a large, detailed image of the moon, set against a starry space. The astronaut begins to run across the lunar terrain, leaving footprints behind. As he runs, the camera angle shifts to reveal more of the moon's rugged landscape. The astronaut continues his run until he reaches the edge of the frame, where he leaps into the vast expanse of space, floating away from the moon. Throughout the sequence, the moon remains a constant and prominent feature in the background, emphasizing the astronaut's journey into the cosmos. The video captures the astronaut's solitary trek across the moon's surface and his subsequent leap into the unknown, symbolizing humanity's boundless curiosity and spirit of exploration. The astronaut's actions are depicted with precision and grace, highlighting the beauty and isolation of space travel. The video concludes with the astronaut floating freely in space, surrounded by the endless void of space, underscoring the awe-inspiring scale and mystery of the universe. The astronaut's journey serves as a powerful metaphor for human ambition and the quest for knowledge, encapsulating the essence of space exploration. The",
  "started_at": "2024-12-16T20:53:35.335381Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/65eyxmwszdrm80ckt4zajjt3rm",
    "cancel": "https://api.replicate.com/v1/predictions/65eyxmwszdrm80ckt4zajjt3rm/cancel"
  },
  "version": "ed5ec3233f7a4331a2b0d33b8a02e62057a335e1a03107d56db9fac10cf3ce55"
}

Generated in

4.3 seconds

Tweak it Iterate in playground Share Report View full prediction

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Apollo: An Exploration of Video Understanding in Large Multimodal Models

Apollo is a family of Large Multimodal Models (LMMs) that push the state-of-the-art in video understanding. It supports tasks including: - Long-form video comprehension - Temporal reasoning - Complex video question-answering - Multi-turn conversations grounded in video content

Apollo models excel at handling hour-long videos, balancing speed and accuracy through strategic design decisions. Our models outperform most 7B competitors at just 3B parameters and even rival 30B-scale models.

Key Highlights: - Scaling Consistency: Design decisions validated on smaller models and datasets effectively transfer to larger scales, reducing computation and experimentation costs. - Efficient Video Sampling: fps sampling and advanced token resampling strategies (e.g., Perceiver) yield stronger temporal perception. - Encoder Synergies: Combining SigLIP-SO400M (image) with InternVideo2 (video) delivers a robust representation, outperforming single encoders on temporal tasks. - ApolloBench: A streamlined evaluation benchmark (41x faster) that focuses on true video understanding capabilities.

Citation

If you find this project useful, please consider citing:

@article{zohar2024apollo,
    title={Apollo: An Exploration of Video Understanding in Large Multimodal Models},
    author={Zohar, Orr and Wang, Xiaohan and Dubois, Yann and Mehta, Nikhil and Xiao, Tong and Hansen-Estruch, Philippe and Yu, Licheng and Wang, Xiaofang and Juefei-Xu, Felix and Zhang, Ning and Yeung-Levy, Serena and Xia, Xide},
    journal={arXiv preprint arXiv:2412.10360},
    year={2024}
}

For more details, visit the project website or check out the paper.