lucataco / segment-anything-2

Segment Anything 2 (SAM2) by Meta - Automatic mask generation

Cold

Public
19.4K runs
L40S
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

image

*file

Input image

mask_limit

integer

maximum number of masks to return. If -1 or None, all masks will be returned. NOTE: The masks are sorted by predicted_iou.

Default: -1

points_per_side

integer

The number of points to be sampled along one side of the image.

Default: 64

points_per_batch

integer

Sets the number of points run simultaneously by the model

Default: 128

pred_iou_thresh

number

A filtering threshold in [0,1], using the model's predicted mask quality.

Default: 0.7

stability_score_thresh

number

A filtering threshold in [0,1], using the stability of the mask under changes to the cutoff used to binarize the model's mask predictions.

Default: 0.92

stability_score_offset

number

The amount to shift the cutoff when calculated the stability score.

Default: 0.7

crop_n_layers

integer

If >0, mask prediction will be run again on crops of the image

Default: 1

box_nms_thresh

number

The box IoU cutoff used by non-maximal suppression to filter duplicate masks.

Default: 0.7

crop_n_points_downscale_factor

integer

The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n.

Default: 2

min_mask_region_area

number

If >0, postprocessing will be applied to remove disconnected regions and holes in masks with area smaller than min_mask_region_area.

Default: 25

mask_2_mask

boolean

Whether to add a one step refinement using previous mask predictions.

Default: true

multimask_output

boolean

Whether to output multimask at each point of the grid.

Default: false

Run this model in Node.js with one line of code:

npx create-replicate --model=lucataco/segment-anything-2

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run lucataco/segment-anything-2 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "lucataco/segment-anything-2:be7cbde9fdf0eecdc8b20ffec9dd0d1cfeace0832d4d0b58a071d993182e1be0",
  {
    input: {
      image: "https://replicate.delivery/pbxt/LMUHQoCnNzN15MpwWLPJUVF96g3zxer6p3dNTHtVrNxqMMe0/socer.png",
      mask_limit: 2,
      mask_2_mask: true,
      crop_n_layers: 1,
      box_nms_thresh: 0.7,
      points_per_side: 64,
      pred_iou_thresh: 0.7,
      multimask_output: false,
      points_per_batch: 128,
      min_mask_region_area: 25,
      stability_score_offset: 0.7,
      stability_score_thresh: 0.92,
      crop_n_points_downscale_factor: 2
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run lucataco/segment-anything-2 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "lucataco/segment-anything-2:be7cbde9fdf0eecdc8b20ffec9dd0d1cfeace0832d4d0b58a071d993182e1be0",
    input={
        "image": "https://replicate.delivery/pbxt/LMUHQoCnNzN15MpwWLPJUVF96g3zxer6p3dNTHtVrNxqMMe0/socer.png",
        "mask_limit": 2,
        "mask_2_mask": True,
        "crop_n_layers": 1,
        "box_nms_thresh": 0.7,
        "points_per_side": 64,
        "pred_iou_thresh": 0.7,
        "multimask_output": False,
        "points_per_batch": 128,
        "min_mask_region_area": 25,
        "stability_score_offset": 0.7,
        "stability_score_thresh": 0.92,
        "crop_n_points_downscale_factor": 2
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run lucataco/segment-anything-2 using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "be7cbde9fdf0eecdc8b20ffec9dd0d1cfeace0832d4d0b58a071d993182e1be0",
    "input": {
      "image": "https://replicate.delivery/pbxt/LMUHQoCnNzN15MpwWLPJUVF96g3zxer6p3dNTHtVrNxqMMe0/socer.png",
      "mask_limit": 2,
      "mask_2_mask": true,
      "crop_n_layers": 1,
      "box_nms_thresh": 0.7,
      "points_per_side": 64,
      "pred_iou_thresh": 0.7,
      "multimask_output": false,
      "points_per_batch": 128,
      "min_mask_region_area": 25,
      "stability_score_offset": 0.7,
      "stability_score_thresh": 0.92,
      "crop_n_points_downscale_factor": 2
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Generated in

16.2 seconds

Tweak it ShareReport

Examples

View more examples

Run time and cost

This model costs approximately $0.032 to run on Replicate, or 31 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 33 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Note: Currently, at this time this model only support image inputs (not video yet) and runs the large variant of the model.

SAM 2: Segment Anything in Images and Videos

About

Implementation of SAM 2, a model for segmenting objects in images and videos using various prompts.

Limitations

Performance may vary depending on image/video quality and complexity.
Very fast or complex motions in videos might be challenging.
Higher resolutions provide more detail but require more processing time.

SAM 2 is a 🔥 model developed by Meta AI Research. It excels at segmenting objects in both images and videos with various types of prompts.

Core Model

model architecture
An overview of the SAM 2 framework.

SAM 2 uses a transformer architecture with streaming memory for real-time video processing. It builds on the original SAM model, extending its capabilities to video.

For more technical details, check out the Research paper.

Safety

⚠️ Users should be aware of potential ethical implications: - Ensure you have the right to use input images and videos, especially those featuring identifiable individuals. - Be responsible about generated content to avoid potential misuse. - Be cautious about using copyrighted material as inputs without permission.

Support

All credit goes to the Meta AI Research team

Citation

@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint},
  year={2024}
}