ALPHA VERSION

Turn a robot manipulation video into timestamped subtasks.

Send an episode and a sentence describing the task. Get back structured labels: start and end times, a name for each completed manipulation event, pass or fail, and failure attributes such as retry or missed grasp.

Robot teams hold thousands of teleoperation episodes that are effectively unsearchable. To find every failed grasp of the yellow block, someone has to watch them. This makes episodes queryable.

Example

Input: a 17 second DROID episode, with the instruction Put the blue block in the green bowl.

Output, verbatim from a real run (not edited):

{
  "task": "Put the blue block in the green bowl",
  "duration_seconds": 17.73,
  "segments": [
    {
      "start_seconds": 0.0,
      "end_seconds": 11.0,
      "label": "place_blue_block_in_green_bowl",
      "result": "fail",
      "attributes": [],
      "description": "The robot attempts to grasp the blue block to place it into the green bowl, but fails to pick it up from the table.",
      "confidence": "low",
      "flags": [
        "label_disagreement"
      ]
    },
    {
      "start_seconds": 11.0,
      "end_seconds": 15.5,
      "label": "place_blue_block_in_green_bowl",
      "result": "pass",
      "attributes": [],
      "description": "The robot carries the blue block over to the green bowl and successfully places it inside.",
      "confidence": "medium",
      "flags": []
    }
  ]
}

A failed attempt followed by a successful one, located in time. That is the shape of query most teams want and cannot run today.

Inputs

Schema mode

Supplying subtasks switches on schema mode. Labels are then constrained to your vocabulary and snapped to it in code. If the model returns something outside the list, it is mapped to the nearest allowed label and the segment is flagged.

Quality modes

How accurate is it

Measured on WGO-Bench, a public benchmark of 100 manually annotated robot and egocentric episodes with 743 gold subtask segments, using balanced:

Scoring protocol: greedy one to one matching at IoU 0.5 or above, pooled across the corpus. Boundary recall counts interior boundaries only, since the first start and last end are dictated by the episode rather than discovered.

Limitations

Events under one second are close to invisible. Recall below one second is 0.038. Timestamps have sub second resolution, but that is not the same as sub second accuracy, and we do not claim the latter.

Recall depends strongly on event duration. Roughly 0.75 for events over eight seconds, 0.37 between one and two seconds. At coarse sampling a short event is only a frame or two, so it is a visibility limit rather than a reasoning one.

It over splits. Precision is 0.68. Finding more real events costs some spurious boundaries. For search that is usually the right trade, and low confidence splits carry flags you can filter on.

Label accuracy is uneven across robots. 0.80 on some sources, 0.46 on DROID, whose reference labels read like task instructions rather than event descriptions.

These are automatically generated labels with no human review.

Confidence and flags

confidence is derived from observable disagreement between pipeline stages. A self reported 0.97 might look precise but is not calibrated.

Flags:

• label_disagreement: the segmentation pass and the labeling pass named the event differently
• boundary_moved_1.25s: refinement moved this boundary a long way
• segment_count_unstable: a repeat pass found a different number of events
• label_outside_vocabulary: the label had to be snapped into your subtasks list

Filter to unflagged segments when you need higher precision.

Bring your own key

This model calls the Gemini API. Get one at aistudio.google.com/apikey. The key is write only, scoped to your call, and never written into the response.

Typical cost on your Gemini account is about 2.40 USD per hour of video in balanced mode.

Data handling

The model container keeps nothing. Frames are extracted to a temporary directory and deleted when the call returns, and no video, frame or annotation is written to storage we control. Your video is sent to the Gemini API as contact sheet images under your own API key, so Google’s terms for your account apply, not ours. Your key is write only and is never returned in the response.

Replicate itself stores prediction inputs and outputs under its own retention policy. If your footage is sensitive, review that before uploading.

Example footage

The example episodes come from DROID via lerobot/droid_1.0.1 (Apache 2.0, DROID itself CC BY 4.0), used with attribution.

Built by

Manda - evaluation infrastructure for robot learning.