nvidia/prismer:569a81a5 – Run with an API on Replicate

Version

You're looking at a specific version of this model. Jump to the model overview.

nvidia /prismer:569a81a5

Playground API

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=nvidia/prismer

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "nvidia/prismer:569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab",
  {
    input: {
      task: "caption",
      model_size: "base",
      input_image: "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg",
      use_experts: true,
      output_expert_labels: true
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "nvidia/prismer:569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab",
    input={
        "task": "caption",
        "model_size": "base",
        "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg",
        "use_experts": True,
        "output_expert_labels": True
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab",
    "input": {
      "task": "caption",
      "model_size": "base",
      "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg",
      "use_experts": true,
      "output_expert_labels": true
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

edge

depth

answer

a man riding a skateboard across a cross walk

ocr_pt

ocr.pt

ocr_png

object_png

segmentation

object_labels

{ "0": 134, "1": 151, "2": 607, "3": 607, "4": 134, "5": 607, "6": 136, "7": 607, "8": 15, "9": 176, "10": 15, "11": 15, "12": 219, "13": 201 }

surface_normal

{
  "completed_at": "2023-03-13T09:22:34.460617Z",
  "created_at": "2023-03-13T09:20:55.643640Z",
  "data_removed": false,
  "error": null,
  "id": "gfzyqg7ubjfmvkqfx7ccs6lhfy",
  "input": {
    "task": "caption",
    "model_size": "base",
    "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg",
    "use_experts": true,
    "output_expert_labels": true
  },
  "logs": "***** Generating edge *****\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.67s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.67s/it]\n***** Generating depth *****\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.52s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.52s/it]\n***** Generating normal *****\nUsing cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.86s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.86s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n  0%|          | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00,  2.79s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.79s/it]\n***** Generating ocrdet *****\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.93s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.93s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.46s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.46s/it]",
  "metrics": {
    "predict_time": 98.735322,
    "total_time": 98.816977
  },
  "output": {
    "edge": "https://replicate.delivery/pbxt/bApG0ru2GCIMFl0UgHffoBh4ecz4M4Nlkxsp1BJ27hGzqTOhA/edge.png",
    "depth": "https://replicate.delivery/pbxt/j1z1NewYQZzhFqdBOKP8YmyzK8VKMIEXTAtOSvJtrj5s6kTIA/depth.png",
    "answer": "a man riding a skateboard across a cross walk",
    "ocr_pt": "https://replicate.delivery/pbxt/40FrffsR1zu0pk25gBZQhruAMA4WbZNe8baeHjgvgEXqVncCB/ocr.pt",
    "ocr_png": "https://replicate.delivery/pbxt/t6w6UkYYOO7vDxEYr3MIGhcL81AemABmMH7rylpJfm3a1JnQA/ocr.png",
    "object_png": "https://replicate.delivery/pbxt/UvnsWUAnNxobGhx7gm2UFa28jnyNHfkKh03adzt0NuTt6kTIA/obj.png",
    "segmentation": "https://replicate.delivery/pbxt/6lDjLet8lAU7PqfHK7whquh68vMTMirtxzEpfzYpjhrzqTOhA/segmentation.png",
    "object_labels": {
      "0": 134,
      "1": 151,
      "2": 607,
      "3": 607,
      "4": 134,
      "5": 607,
      "6": 136,
      "7": 607,
      "8": 15,
      "9": 176,
      "10": 15,
      "11": 15,
      "12": 219,
      "13": 201
    },
    "surface_normal": "https://replicate.delivery/pbxt/j62BcIff8YufyJEM5lkAXERAz4Zwsx0DiZsXd8mOKilyqTOhA/normal.png"
  },
  "started_at": "2023-03-13T09:20:55.725295Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/gfzyqg7ubjfmvkqfx7ccs6lhfy",
    "cancel": "https://api.replicate.com/v1/predictions/gfzyqg7ubjfmvkqfx7ccs6lhfy/cancel"
  },
  "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab"
}

Generated in

1 minute 39 seconds

Tweak itReport

***** Generating edge *****
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.67s/it]
100%|██████████| 1/1 [00:02<00:00,  2.67s/it]
***** Generating depth *****
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.52s/it]
100%|██████████| 1/1 [00:02<00:00,  2.52s/it]
***** Generating normal *****
Using cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master
Loading base model ()...Done.
Removing last two layers (global_pool & classifier).
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.86s/it]
100%|██████████| 1/1 [00:02<00:00,  2.86s/it]
***** Generating objdet *****
Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.
  0%|          | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
100%|██████████| 1/1 [00:02<00:00,  2.79s/it]
100%|██████████| 1/1 [00:02<00:00,  2.79s/it]
***** Generating ocrdet *****
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.93s/it]
100%|██████████| 1/1 [00:02<00:00,  2.93s/it]
***** Generating segmentation *****
/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.46s/it]
100%|██████████| 1/1 [00:02<00:00,  2.46s/it]