lucataco / bulk-video-caption

Video Preprocessing tool for captioning multiple videos using GPT, Claude or Gemini

Cold

Public
118 runs
CPU
GitHub
License

Iterate in the playground

Run with an API

Playground API Examples README Versions

Input

video_zip_archive

*file

ZIP archive containing videos to process

include_csv

boolean

Whether to include CSV in output

Default: true

caption_prefix

string

Shift + Return to add a new line

Optional prefix for video captions

Default: ""

caption_suffix

string

Shift + Return to add a new line

Optional suffix for video captions

Default: ""

frames_to_extract

integer

Number of frames to extract from each video for analysis

Default: 2

system_prompt

string

Shift + Return to add a new line

Analyze these frames from a video and write a detailed caption. Describe the type of video (e.g., animation, live-action footage, etc.). Focus on consistent elements across frames and any notable motion or action. Describe the main subjects, setting, and overall mood of the video. Use clear, descriptive language suitable for text-to-video generation.Analyze these frames from a video and write a detailed caption. Describe the type of video (e.g., animation, live-action footage, etc.). Focus on consistent elements across frames and any notable motion or action. Describe the main subjects, setting, and overall mood of the video. Use clear, descriptive language suitable for text-to-video generation.

System prompt for caption generation

Default: "\n Analyze these frames from a video and write a detailed caption. \n Describe the type of video (e.g., animation, live-action footage, etc.).\n Focus on consistent elements across frames and any notable motion or action.\n Describe the main subjects, setting, and overall mood of the video.\n Use clear, descriptive language suitable for text-to-video generation.\n "

model

string

AI model to use for captioning

Default: "gpt-4o"

openai_api_key

secret

API key for OpenAI

anthropic_api_key

secret

API key for Anthropic

google_generativeai_api_key

secret

API key for Google Generative AI

Run this model in Node.js with one line of code:

npx create-replicate --model=lucataco/bulk-video-caption

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run lucataco/bulk-video-caption using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "lucataco/bulk-video-caption:bd610b3c0ecd967e0528ff94d6d1b19cb067aaa12cf1516029ca3803d7e46791",
  {
    input: {
      model: "gpt-4o",
      include_csv: false,
      system_prompt: "Analyze these frames from a video and write a detailed caption. Describe the type of video (e.g., animation, live-action footage, etc.). Focus on consistent elements across frames and any notable motion or action. Describe the main subjects, setting, and overall mood of the video. Use clear, descriptive language suitable for text-to-video generation.",
      caption_prefix: "",
      caption_suffix: "melty.",
      openai_api_key: "",
      frames_to_extract: 1,
      video_zip_archive: "https://replicate.delivery/pbxt/M8gY3MV17leQVFCbrJo0CWRznIv6joAa20dfVfqCpwulNHKj/melty-seg-3.zip"
    }
  }
);

// To access the file URL:
console.log(output.url()); //=> "http://example.com"

// To write the file to disk:
fs.writeFile("my-image.png", output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run lucataco/bulk-video-caption using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "lucataco/bulk-video-caption:bd610b3c0ecd967e0528ff94d6d1b19cb067aaa12cf1516029ca3803d7e46791",
    input={
        "model": "gpt-4o",
        "include_csv": False,
        "system_prompt": "Analyze these frames from a video and write a detailed caption. Describe the type of video (e.g., animation, live-action footage, etc.). Focus on consistent elements across frames and any notable motion or action. Describe the main subjects, setting, and overall mood of the video. Use clear, descriptive language suitable for text-to-video generation.",
        "caption_prefix": "",
        "caption_suffix": "melty.",
        "openai_api_key": "",
        "frames_to_extract": 1,
        "video_zip_archive": "https://replicate.delivery/pbxt/M8gY3MV17leQVFCbrJo0CWRznIv6joAa20dfVfqCpwulNHKj/melty-seg-3.zip"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run lucataco/bulk-video-caption using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "lucataco/bulk-video-caption:bd610b3c0ecd967e0528ff94d6d1b19cb067aaa12cf1516029ca3803d7e46791",
    "input": {
      "model": "gpt-4o",
      "include_csv": false,
      "system_prompt": "Analyze these frames from a video and write a detailed caption. Describe the type of video (e.g., animation, live-action footage, etc.). Focus on consistent elements across frames and any notable motion or action. Describe the main subjects, setting, and overall mood of the video. Use clear, descriptive language suitable for text-to-video generation.",
      "caption_prefix": "",
      "caption_suffix": "melty.",
      "openai_api_key": "",
      "frames_to_extract": 1,
      "video_zip_archive": "https://replicate.delivery/pbxt/M8gY3MV17leQVFCbrJo0CWRznIv6joAa20dfVfqCpwulNHKj/melty-seg-3.zip"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

video_captions.zip

{
  "completed_at": "2024-12-13T21:38:20.454214Z",
  "created_at": "2024-12-13T21:38:03.631000Z",
  "data_removed": false,
  "error": null,
  "id": "ah7ykc0gxxrg80ckr7tty5xvc8",
  "input": {
    "model": "gpt-4o",
    "include_csv": false,
    "system_prompt": "Analyze these frames from a video and write a detailed caption. Describe the type of video (e.g., animation, live-action footage, etc.). Focus on consistent elements across frames and any notable motion or action. Describe the main subjects, setting, and overall mood of the video. Use clear, descriptive language suitable for text-to-video generation.",
    "caption_prefix": "",
    "caption_suffix": "melty.",
    "openai_api_key": "[REDACTED]",
    "frames_to_extract": 1,
    "video_zip_archive": "https://replicate.delivery/pbxt/M8gY3MV17leQVFCbrJo0CWRznIv6joAa20dfVfqCpwulNHKj/melty-seg-3.zip"
  },
  "logs": "Files extracted:\n/tmp/outputs/straws.mov\n/tmp/outputs/glove.mov\n/tmp/outputs/cat.mov\nNumber of videos to be captioned: 3\n===================================================\nProcessing straws.mov\nCaption: A vibrant cluster of colorful drinking straws is tightly packed together, showcasing a range of bright hues including orange, pink, blue, green, and yellow. The straws are slightly tilted, suggesting movement or rearrangement. The background is a neutral gray, emphasizing the vividness of the straws. The mood is playful and dynamic. Melty.\n===================================================\nProcessing glove.mov\nCaption: A pale yellow rubber glove is placed against a smooth, softly-lit background. The material appears shiny and slightly wrinkled, suggesting a relaxed and casual setting. The overall mood is calm and minimalistic.\nmelty.\n===================================================\nProcessing cat.mov\nCaption: The frame depicts an adorable kitten with large, expressive eyes and soft, striped fur, looking intently at something off-camera while nestled on a cozy surface. The kitten's curious expression is captivating, adding to the overall warm and endearing mood of the scene. melty.\n===================================================",
  "metrics": {
    "predict_time": 10.573291217,
    "total_time": 16.823214
  },
  "output": "https://replicate.delivery/czjl/4rU99k2uoTr6PFHn0fQCtw2BtMSG7F2ekCwdoQBlBkJMtl6TA/video_captions.zip",
  "started_at": "2024-12-13T21:38:09.880923Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://stream.replicate.com/v1/files/fddq-s3okqivpofyzbvqriiomt27rwbvkvv7g65b6yadjzz6qv2f6qgxq",
    "get": "https://api.replicate.com/v1/predictions/ah7ykc0gxxrg80ckr7tty5xvc8",
    "cancel": "https://api.replicate.com/v1/predictions/ah7ykc0gxxrg80ckr7tty5xvc8/cancel"
  },
  "version": "bd610b3c0ecd967e0528ff94d6d1b19cb067aaa12cf1516029ca3803d7e46791"
}

Generated in

10.6 seconds

Tweak it Share Report

Run time and cost

This model runs on CPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Batch video caption

A cog model for batch video captioning using various AI from OpenAI, Anthropic, and Google

Features

Process multiple images from a ZIP archive
supports mov, mp4
Customizable caption prefixes and suffixes
Support for multiple AI models:
- OpenAI: GPT-4 and variants
- Anthropic: Claude-3.5, Claude-3 variants
- Google: Gemini-1.5 variants
Flexible system prompts
Error handling and retry mechanism
Output as a ZIP file containing captions that match image filenames as well as an optional CSV summary