nvidia/prismer:569a81a5 – Run with an API on Replicate

Version

You're looking at a specific version of this model. Jump to the model overview.

nvidia /prismer:569a81a5

Playground API

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=nvidia/prismer

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "nvidia/prismer:569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab",
  {
    input: {
      task: "caption",
      model_size: "base",
      input_image: "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg",
      use_experts: true,
      output_expert_labels: true
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "nvidia/prismer:569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab",
    input={
        "task": "caption",
        "model_size": "base",
        "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg",
        "use_experts": True,
        "output_expert_labels": True
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab",
    "input": {
      "task": "caption",
      "model_size": "base",
      "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg",
      "use_experts": true,
      "output_expert_labels": true
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

edge

depth

answer

a baseball player swinging a bat at a ball

ocr_pt

ocr.pt

ocr_png

object_png

segmentation

object_labels

{ "0": 134, "1": 10, "2": 10, "3": 129, "4": 175, "5": 204, "6": 134, "7": 189, "8": 196, "9": 196 }

surface_normal

{
  "completed_at": "2023-03-13T09:20:24.984666Z",
  "created_at": "2023-03-13T09:11:18.148127Z",
  "data_removed": false,
  "error": null,
  "id": "gchbvguxovbkjhb7rgd74psnce",
  "input": {
    "task": "caption",
    "model_size": "base",
    "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg",
    "use_experts": true,
    "output_expert_labels": true
  },
  "logs": "***** Generating edge *****\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.27s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.27s/it]\n***** Generating depth *****\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.17s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.17s/it]\n***** Generating normal *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/hub.py:267: UserWarning: You are about to download and run code from an untrusted repository. In a future release, this won't be allowed. To add the repository to your trusted list, change the command to {calling_fn}(..., trust_repo=False) and a command prompt will appear asking for an explicit confirmation of trust, or load(..., trust_repo=True), which will assume that the prompt is to be answered with 'yes'. You can also use load(..., trust_repo='check') which will only prompt for confirmation if the repo is not already trusted. This will eventually be the default behaviour\nwarnings.warn(\nDownloading: \"https://github.com/rwightman/gen-efficientnet-pytorch/zipball/master\" to /root/.cache/torch/hub/master.zip\nDownloading: \"https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b5_ap-9e82fae8.pth\" to /root/.cache/torch/hub/checkpoints/tf_efficientnet_b5_ap-9e82fae8.pth\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.29s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.30s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n  0%|          | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00,  2.36s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.36s/it]\n***** Generating ocrdet *****\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00,  2.01s/it]\n100%|██████████| 1/1 [00:02<00:00,  2.01s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\n  0%|          | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:03<00:00,  3.34s/it]\n100%|██████████| 1/1 [00:03<00:00,  3.34s/it]",
  "metrics": {
    "predict_time": 89.336026,
    "total_time": 546.836539
  },
  "output": {
    "edge": "https://replicate.delivery/pbxt/wTF62ikSr96AIVScIoljKau7696y15zqdSsL9amfjefxmTOhA/edge.png",
    "depth": "https://replicate.delivery/pbxt/DGepUblNcnwoYaZjzvaS16pslElmkicR8QvwteHYvUXXzJnQA/depth.png",
    "answer": "a baseball player swinging a bat at a ball",
    "ocr_pt": "https://replicate.delivery/pbxt/AaCVyvoznm7hOVKJ08j97DvRLWMAs3QnrVLkzTPmQNN2cyJE/ocr.pt",
    "ocr_png": "https://replicate.delivery/pbxt/XqgXt56ILWKQGxMVgKS0xzyYeaseGwfCR9QSzCQ34t6xmTOhA/ocr.png",
    "object_png": "https://replicate.delivery/pbxt/ZpiIBbTSCF72DRv7Xcsn3j83EkmKiO0DN52Xe8wAEbUs5kTIA/obj.png",
    "segmentation": "https://replicate.delivery/pbxt/Cx0v5RkBKEZBGxJcySahp8pICuoj0yyfbI6xFYA9PtHs5kTIA/segmentation.png",
    "object_labels": {
      "0": 134,
      "1": 10,
      "2": 10,
      "3": 129,
      "4": 175,
      "5": 204,
      "6": 134,
      "7": 189,
      "8": 196,
      "9": 196
    },
    "surface_normal": "https://replicate.delivery/pbxt/tHGMlbpYourPMtzl2ES7K9dtUpPCbemc72Yd93FtxXEs5kTIA/normal.png"
  },
  "started_at": "2023-03-13T09:18:55.648640Z",
  "status": "succeeded",
  "urls": {
    "get": "https://api.replicate.com/v1/predictions/gchbvguxovbkjhb7rgd74psnce",
    "cancel": "https://api.replicate.com/v1/predictions/gchbvguxovbkjhb7rgd74psnce/cancel"
  },
  "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab"
}

Generated in

89.3 seconds

Tweak itReport

***** Generating edge *****
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.27s/it]
100%|██████████| 1/1 [00:02<00:00,  2.27s/it]
***** Generating depth *****
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.17s/it]
100%|██████████| 1/1 [00:02<00:00,  2.17s/it]
***** Generating normal *****
/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/hub.py:267: UserWarning: You are about to download and run code from an untrusted repository. In a future release, this won't be allowed. To add the repository to your trusted list, change the command to {calling_fn}(..., trust_repo=False) and a command prompt will appear asking for an explicit confirmation of trust, or load(..., trust_repo=True), which will assume that the prompt is to be answered with 'yes'. You can also use load(..., trust_repo='check') which will only prompt for confirmation if the repo is not already trusted. This will eventually be the default behaviour
warnings.warn(
Downloading: "https://github.com/rwightman/gen-efficientnet-pytorch/zipball/master" to /root/.cache/torch/hub/master.zip
Downloading: "https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b5_ap-9e82fae8.pth" to /root/.cache/torch/hub/checkpoints/tf_efficientnet_b5_ap-9e82fae8.pth
Loading base model ()...Done.
Removing last two layers (global_pool & classifier).
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.29s/it]
100%|██████████| 1/1 [00:02<00:00,  2.30s/it]
***** Generating objdet *****
Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.
  0%|          | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
100%|██████████| 1/1 [00:02<00:00,  2.36s/it]
100%|██████████| 1/1 [00:02<00:00,  2.36s/it]
***** Generating ocrdet *****
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00,  2.01s/it]
100%|██████████| 1/1 [00:02<00:00,  2.01s/it]
***** Generating segmentation *****
/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...
  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:03<00:00,  3.34s/it]
100%|██████████| 1/1 [00:03<00:00,  3.34s/it]