nvidia
/
prismer
A Vision-Language Model with An Ensemble of Experts
Prediction
nvidia/prismer:e604611dID4x5slvr76fbmhbb5nsuwxau7riStatusSucceededSourceWebHardware–Total durationCreatedInput
- task
- vqa
- question
- what is the man sitting down doing?
- input_image
- use_experts
- output_expert_labels
{ "task": "vqa", "question": "what is the man sitting down doing?", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false }Install Replicate’s Node.js client library:npm install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", { input: { task: "vqa", question: "what is the man sitting down doing?", input_image: "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", use_experts: true, output_expert_labels: false } } ); console.log(output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:import replicateRun nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", input={ "task": "vqa", "question": "what is the man sitting down doing?", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": True, "output_expert_labels": False } ) print(output)To learn more, take a look at the guide on getting started with Python.
Set theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", "input": { "task": "vqa", "question": "what is the man sitting down doing?", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
You can run this model locally using Cog. First, install Cog:brew install cogIf you don’t have Homebrew, there are other installation options available.
Run this to download the model and run it in your local environment:
cog predict r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1 \ -i 'task="vqa"' \ -i 'question="what is the man sitting down doing?"' \ -i 'input_image="https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg"' \ -i 'use_experts=true' \ -i 'output_expert_labels=false'To learn more, take a look at the Cog documentation.
Run this to download the model and run it in your local environment:
docker run -d -p 5000:5000 --gpus=all r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1curl -s -X POST \ -H "Content-Type: application/json" \ -d $'{ "input": { "task": "vqa", "question": "what is the man sitting down doing?", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false } }' \ http://localhost:5000/predictionsTo learn more, take a look at the Cog documentation.
Output
riding bike{ "completed_at": "2023-03-11T22:26:46.233154Z", "created_at": "2023-03-11T22:25:12.647330Z", "data_removed": false, "error": null, "id": "4x5slvr76fbmhbb5nsuwxau7ri", "input": { "task": "vqa", "question": "what is the man sitting down doing?", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false }, "logs": "***** Generating edge *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.49s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.49s/it]\n***** Generating depth *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.50s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.50s/it]\n***** Generating normal *****\nUsing cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.69s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.69s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00, 2.58s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.58s/it]\n***** Generating ocrdet *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.94s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.95s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.56s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.56s/it]", "metrics": { "predict_time": 93.519066, "total_time": 93.585824 }, "output": { "answer": "riding bike" }, "started_at": "2023-03-11T22:25:12.714088Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/4x5slvr76fbmhbb5nsuwxau7ri", "cancel": "https://api.replicate.com/v1/predictions/4x5slvr76fbmhbb5nsuwxau7ri/cancel" }, "version": "ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017" }Generated in***** Generating edge ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.49s/it] 100%|██████████| 1/1 [00:02<00:00, 2.49s/it] ***** Generating depth ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.50s/it] 100%|██████████| 1/1 [00:02<00:00, 2.50s/it] ***** Generating normal ***** Using cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master Loading base model ()...Done. Removing last two layers (global_pool & classifier). 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.69s/it] 100%|██████████| 1/1 [00:02<00:00, 2.69s/it] ***** Generating objdet ***** Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content. 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 100%|██████████| 1/1 [00:02<00:00, 2.58s/it] 100%|██████████| 1/1 [00:02<00:00, 2.58s/it] ***** Generating ocrdet ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.94s/it] 100%|██████████| 1/1 [00:02<00:00, 2.95s/it] ***** Generating segmentation ***** /root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ... error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain. error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain. error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain. error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain. error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain. error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain. 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.56s/it] 100%|██████████| 1/1 [00:02<00:00, 2.56s/it]Prediction
nvidia/prismer:e604611dIDgchbvguxovbkjhb7rgd74psnceStatusSucceededSourceWebHardware–Total durationCreatedInput
- task
- caption
- model_size
- base
- input_image
- use_experts
- output_expert_labels
{ "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg", "use_experts": true, "output_expert_labels": true }Install Replicate’s Node.js client library:npm install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", { input: { task: "caption", model_size: "base", input_image: "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg", use_experts: true, output_expert_labels: true } } ); console.log(output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:import replicateRun nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", input={ "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg", "use_experts": True, "output_expert_labels": True } ) print(output)To learn more, take a look at the guide on getting started with Python.
Set theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", "input": { "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg", "use_experts": true, "output_expert_labels": true } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
You can run this model locally using Cog. First, install Cog:brew install cogIf you don’t have Homebrew, there are other installation options available.
Run this to download the model and run it in your local environment:
cog predict r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1 \ -i 'task="caption"' \ -i 'model_size="base"' \ -i 'input_image="https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg"' \ -i 'use_experts=true' \ -i 'output_expert_labels=true'To learn more, take a look at the Cog documentation.
Run this to download the model and run it in your local environment:
docker run -d -p 5000:5000 --gpus=all r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1curl -s -X POST \ -H "Content-Type: application/json" \ -d $'{ "input": { "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg", "use_experts": true, "output_expert_labels": true } }' \ http://localhost:5000/predictionsTo learn more, take a look at the Cog documentation.
Output
edge
depth
answer
a baseball player swinging a bat at a ballocr_pt
ocr.ptocr_png
object_png
segmentation
object_labels
{ "0": 134, "1": 10, "2": 10, "3": 129, "4": 175, "5": 204, "6": 134, "7": 189, "8": 196, "9": 196 }surface_normal
{ "completed_at": "2023-03-13T09:20:24.984666Z", "created_at": "2023-03-13T09:11:18.148127Z", "data_removed": false, "error": null, "id": "gchbvguxovbkjhb7rgd74psnce", "input": { "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSOKYXqqeSKnOB5SneFCDZaCr3JbJSSAofz6tH6Ai5BCZOu/2.jpeg", "use_experts": true, "output_expert_labels": true }, "logs": "***** Generating edge *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.27s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.27s/it]\n***** Generating depth *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.17s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.17s/it]\n***** Generating normal *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/hub.py:267: UserWarning: You are about to download and run code from an untrusted repository. In a future release, this won't be allowed. To add the repository to your trusted list, change the command to {calling_fn}(..., trust_repo=False) and a command prompt will appear asking for an explicit confirmation of trust, or load(..., trust_repo=True), which will assume that the prompt is to be answered with 'yes'. You can also use load(..., trust_repo='check') which will only prompt for confirmation if the repo is not already trusted. This will eventually be the default behaviour\nwarnings.warn(\nDownloading: \"https://github.com/rwightman/gen-efficientnet-pytorch/zipball/master\" to /root/.cache/torch/hub/master.zip\nDownloading: \"https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b5_ap-9e82fae8.pth\" to /root/.cache/torch/hub/checkpoints/tf_efficientnet_b5_ap-9e82fae8.pth\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.29s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.30s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00, 2.36s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.36s/it]\n***** Generating ocrdet *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.01s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.01s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:03<00:00, 3.34s/it]\n100%|██████████| 1/1 [00:03<00:00, 3.34s/it]", "metrics": { "predict_time": 89.336026, "total_time": 546.836539 }, "output": { "edge": "https://replicate.delivery/pbxt/wTF62ikSr96AIVScIoljKau7696y15zqdSsL9amfjefxmTOhA/edge.png", "depth": "https://replicate.delivery/pbxt/DGepUblNcnwoYaZjzvaS16pslElmkicR8QvwteHYvUXXzJnQA/depth.png", "answer": "a baseball player swinging a bat at a ball", "ocr_pt": "https://replicate.delivery/pbxt/AaCVyvoznm7hOVKJ08j97DvRLWMAs3QnrVLkzTPmQNN2cyJE/ocr.pt", "ocr_png": "https://replicate.delivery/pbxt/XqgXt56ILWKQGxMVgKS0xzyYeaseGwfCR9QSzCQ34t6xmTOhA/ocr.png", "object_png": "https://replicate.delivery/pbxt/ZpiIBbTSCF72DRv7Xcsn3j83EkmKiO0DN52Xe8wAEbUs5kTIA/obj.png", "segmentation": "https://replicate.delivery/pbxt/Cx0v5RkBKEZBGxJcySahp8pICuoj0yyfbI6xFYA9PtHs5kTIA/segmentation.png", "object_labels": { "0": 134, "1": 10, "2": 10, "3": 129, "4": 175, "5": 204, "6": 134, "7": 189, "8": 196, "9": 196 }, "surface_normal": "https://replicate.delivery/pbxt/tHGMlbpYourPMtzl2ES7K9dtUpPCbemc72Yd93FtxXEs5kTIA/normal.png" }, "started_at": "2023-03-13T09:18:55.648640Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/gchbvguxovbkjhb7rgd74psnce", "cancel": "https://api.replicate.com/v1/predictions/gchbvguxovbkjhb7rgd74psnce/cancel" }, "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab" }Generated in***** Generating edge ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.27s/it] 100%|██████████| 1/1 [00:02<00:00, 2.27s/it] ***** Generating depth ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.17s/it] 100%|██████████| 1/1 [00:02<00:00, 2.17s/it] ***** Generating normal ***** /root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/hub.py:267: UserWarning: You are about to download and run code from an untrusted repository. In a future release, this won't be allowed. To add the repository to your trusted list, change the command to {calling_fn}(..., trust_repo=False) and a command prompt will appear asking for an explicit confirmation of trust, or load(..., trust_repo=True), which will assume that the prompt is to be answered with 'yes'. You can also use load(..., trust_repo='check') which will only prompt for confirmation if the repo is not already trusted. This will eventually be the default behaviour warnings.warn( Downloading: "https://github.com/rwightman/gen-efficientnet-pytorch/zipball/master" to /root/.cache/torch/hub/master.zip Downloading: "https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b5_ap-9e82fae8.pth" to /root/.cache/torch/hub/checkpoints/tf_efficientnet_b5_ap-9e82fae8.pth Loading base model ()...Done. Removing last two layers (global_pool & classifier). 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.29s/it] 100%|██████████| 1/1 [00:02<00:00, 2.30s/it] ***** Generating objdet ***** Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content. 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 100%|██████████| 1/1 [00:02<00:00, 2.36s/it] 100%|██████████| 1/1 [00:02<00:00, 2.36s/it] ***** Generating ocrdet ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.01s/it] 100%|██████████| 1/1 [00:02<00:00, 2.01s/it] ***** Generating segmentation ***** /root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ... 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:03<00:00, 3.34s/it] 100%|██████████| 1/1 [00:03<00:00, 3.34s/it]Prediction
nvidia/prismer:e604611dIDgfzyqg7ubjfmvkqfx7ccs6lhfyStatusSucceededSourceWebHardware–Total durationCreatedInput
- task
- caption
- model_size
- base
- input_image
- use_experts
- output_expert_labels
{ "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg", "use_experts": true, "output_expert_labels": true }Install Replicate’s Node.js client library:npm install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", { input: { task: "caption", model_size: "base", input_image: "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg", use_experts: true, output_expert_labels: true } } ); console.log(output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:import replicateRun nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", input={ "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg", "use_experts": True, "output_expert_labels": True } ) print(output)To learn more, take a look at the guide on getting started with Python.
Set theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", "input": { "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg", "use_experts": true, "output_expert_labels": true } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
You can run this model locally using Cog. First, install Cog:brew install cogIf you don’t have Homebrew, there are other installation options available.
Run this to download the model and run it in your local environment:
cog predict r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1 \ -i 'task="caption"' \ -i 'model_size="base"' \ -i 'input_image="https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg"' \ -i 'use_experts=true' \ -i 'output_expert_labels=true'To learn more, take a look at the Cog documentation.
Run this to download the model and run it in your local environment:
docker run -d -p 5000:5000 --gpus=all r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1curl -s -X POST \ -H "Content-Type: application/json" \ -d $'{ "input": { "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg", "use_experts": true, "output_expert_labels": true } }' \ http://localhost:5000/predictionsTo learn more, take a look at the Cog documentation.
Output
edge
depth
answer
a man riding a skateboard across a cross walkocr_pt
ocr.ptocr_png
object_png
segmentation
object_labels
{ "0": 134, "1": 151, "2": 607, "3": 607, "4": 134, "5": 607, "6": 136, "7": 607, "8": 15, "9": 176, "10": 15, "11": 15, "12": 219, "13": 201 }surface_normal
{ "completed_at": "2023-03-13T09:22:34.460617Z", "created_at": "2023-03-13T09:20:55.643640Z", "data_removed": false, "error": null, "id": "gfzyqg7ubjfmvkqfx7ccs6lhfy", "input": { "task": "caption", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSVBaSQmIkWaATv0d1nKDXGDyISFHB5USLbExTmDvdBXkPA/1.jpeg", "use_experts": true, "output_expert_labels": true }, "logs": "***** Generating edge *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.67s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.67s/it]\n***** Generating depth *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.52s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.52s/it]\n***** Generating normal *****\nUsing cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.86s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.86s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00, 2.79s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.79s/it]\n***** Generating ocrdet *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.93s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.93s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.46s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.46s/it]", "metrics": { "predict_time": 98.735322, "total_time": 98.816977 }, "output": { "edge": "https://replicate.delivery/pbxt/bApG0ru2GCIMFl0UgHffoBh4ecz4M4Nlkxsp1BJ27hGzqTOhA/edge.png", "depth": "https://replicate.delivery/pbxt/j1z1NewYQZzhFqdBOKP8YmyzK8VKMIEXTAtOSvJtrj5s6kTIA/depth.png", "answer": "a man riding a skateboard across a cross walk", "ocr_pt": "https://replicate.delivery/pbxt/40FrffsR1zu0pk25gBZQhruAMA4WbZNe8baeHjgvgEXqVncCB/ocr.pt", "ocr_png": "https://replicate.delivery/pbxt/t6w6UkYYOO7vDxEYr3MIGhcL81AemABmMH7rylpJfm3a1JnQA/ocr.png", "object_png": "https://replicate.delivery/pbxt/UvnsWUAnNxobGhx7gm2UFa28jnyNHfkKh03adzt0NuTt6kTIA/obj.png", "segmentation": "https://replicate.delivery/pbxt/6lDjLet8lAU7PqfHK7whquh68vMTMirtxzEpfzYpjhrzqTOhA/segmentation.png", "object_labels": { "0": 134, "1": 151, "2": 607, "3": 607, "4": 134, "5": 607, "6": 136, "7": 607, "8": 15, "9": 176, "10": 15, "11": 15, "12": 219, "13": 201 }, "surface_normal": "https://replicate.delivery/pbxt/j62BcIff8YufyJEM5lkAXERAz4Zwsx0DiZsXd8mOKilyqTOhA/normal.png" }, "started_at": "2023-03-13T09:20:55.725295Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/gfzyqg7ubjfmvkqfx7ccs6lhfy", "cancel": "https://api.replicate.com/v1/predictions/gfzyqg7ubjfmvkqfx7ccs6lhfy/cancel" }, "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab" }Generated in***** Generating edge ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.67s/it] 100%|██████████| 1/1 [00:02<00:00, 2.67s/it] ***** Generating depth ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.52s/it] 100%|██████████| 1/1 [00:02<00:00, 2.52s/it] ***** Generating normal ***** Using cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master Loading base model ()...Done. Removing last two layers (global_pool & classifier). 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.86s/it] 100%|██████████| 1/1 [00:02<00:00, 2.86s/it] ***** Generating objdet ***** Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content. 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 100%|██████████| 1/1 [00:02<00:00, 2.79s/it] 100%|██████████| 1/1 [00:02<00:00, 2.79s/it] ***** Generating ocrdet ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.93s/it] 100%|██████████| 1/1 [00:02<00:00, 2.93s/it] ***** Generating segmentation ***** /root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ... 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.46s/it] 100%|██████████| 1/1 [00:02<00:00, 2.46s/it]Prediction
nvidia/prismer:e604611dID6ied42qfhjhwbhsiffcxehc54eStatusSucceededSourceWebHardware–Total durationCreatedInput
- task
- vqa
- question
- what is origin of the animal in the picture?
- model_size
- large
- input_image
- use_experts
- output_expert_labels
{ "task": "vqa", "question": "what is origin of the animal in the picture?", "model_size": "large", "input_image": "https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg", "use_experts": true, "output_expert_labels": true }Install Replicate’s Node.js client library:npm install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", { input: { task: "vqa", question: "what is origin of the animal in the picture?", model_size: "large", input_image: "https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg", use_experts: true, output_expert_labels: true } } ); console.log(output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:import replicateRun nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", input={ "task": "vqa", "question": "what is origin of the animal in the picture?", "model_size": "large", "input_image": "https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg", "use_experts": True, "output_expert_labels": True } ) print(output)To learn more, take a look at the guide on getting started with Python.
Set theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", "input": { "task": "vqa", "question": "what is origin of the animal in the picture?", "model_size": "large", "input_image": "https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg", "use_experts": true, "output_expert_labels": true } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
You can run this model locally using Cog. First, install Cog:brew install cogIf you don’t have Homebrew, there are other installation options available.
Run this to download the model and run it in your local environment:
cog predict r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1 \ -i 'task="vqa"' \ -i 'question="what is origin of the animal in the picture?"' \ -i 'model_size="large"' \ -i 'input_image="https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg"' \ -i 'use_experts=true' \ -i 'output_expert_labels=true'To learn more, take a look at the Cog documentation.
Run this to download the model and run it in your local environment:
docker run -d -p 5000:5000 --gpus=all r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1curl -s -X POST \ -H "Content-Type: application/json" \ -d $'{ "input": { "task": "vqa", "question": "what is origin of the animal in the picture?", "model_size": "large", "input_image": "https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg", "use_experts": true, "output_expert_labels": true } }' \ http://localhost:5000/predictionsTo learn more, take a look at the Cog documentation.
Output
edge
depth
answer
african elephantobject_png
segmentation
object_labels
{ "0": 173, "1": 209 }surface_normal
{ "completed_at": "2023-03-13T09:30:01.208145Z", "created_at": "2023-03-13T09:27:52.454439Z", "data_removed": false, "error": null, "id": "6ied42qfhjhwbhsiffcxehc54e", "input": { "task": "vqa", "question": "what is origin of the animal in the picture?", "model_size": "large", "input_image": "https://replicate.delivery/pbxt/ISyiNHxsjFNhpZKhXEOpGpRjw2lT5IoX0wsHOuPA1eDbCCN5/4.jpeg", "use_experts": true, "output_expert_labels": true }, "logs": "***** Generating edge *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.66s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.66s/it]\n***** Generating depth *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.52s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.52s/it]\n***** Generating normal *****\nUsing cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:03<00:00, 3.07s/it]\n100%|██████████| 1/1 [00:03<00:00, 3.07s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00, 2.89s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.89s/it]\n***** Generating ocrdet *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.30s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.30s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.62s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.62s/it]", "metrics": { "predict_time": 128.684256, "total_time": 128.753706 }, "output": { "edge": "https://replicate.delivery/pbxt/5s9zZPt7Tv7QNlQYagX9wmoUVWkAsGePKLopavZAU0TMeJnQA/edge.png", "depth": "https://replicate.delivery/pbxt/4vnfhZ21Dd04bCpfteciZWZGdViY1R6prX8ve6KeRuYEjP5EC/depth.png", "answer": "african elephant", "ocr_pt": null, "ocr_png": null, "object_png": "https://replicate.delivery/pbxt/A7bHYi2WbpboF5CeYANdcNfybfpP08J40iklPl7yHCnz4TOhA/obj.png", "segmentation": "https://replicate.delivery/pbxt/Blyb7e101kRFYKKpbftiBYEvWsDSIzXexCvaeqK5DlWhxncCB/segmentation.png", "object_labels": { "0": 173, "1": 209 }, "surface_normal": "https://replicate.delivery/pbxt/g8rNRajcYba6HxRJa9gstALuqBhguWz99UsyVZMZPffY8JnQA/normal.png" }, "started_at": "2023-03-13T09:27:52.523889Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/6ied42qfhjhwbhsiffcxehc54e", "cancel": "https://api.replicate.com/v1/predictions/6ied42qfhjhwbhsiffcxehc54e/cancel" }, "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab" }Generated in***** Generating edge ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.66s/it] 100%|██████████| 1/1 [00:02<00:00, 2.66s/it] ***** Generating depth ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.52s/it] 100%|██████████| 1/1 [00:02<00:00, 2.52s/it] ***** Generating normal ***** Using cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master Loading base model ()...Done. Removing last two layers (global_pool & classifier). 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:03<00:00, 3.07s/it] 100%|██████████| 1/1 [00:03<00:00, 3.07s/it] ***** Generating objdet ***** Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content. 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 100%|██████████| 1/1 [00:02<00:00, 2.89s/it] 100%|██████████| 1/1 [00:02<00:00, 2.89s/it] ***** Generating ocrdet ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.30s/it] 100%|██████████| 1/1 [00:02<00:00, 2.30s/it] ***** Generating segmentation ***** /root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ... 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.62s/it] 100%|██████████| 1/1 [00:02<00:00, 2.62s/it]Prediction
nvidia/prismer:e604611dInput
- task
- caption
- question
- model_size
- base
- input_image
- use_experts
- output_expert_labels
{ "task": "caption", "question": "", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false }Install Replicate’s Node.js client library:npm install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", { input: { task: "caption", question: "", model_size: "base", input_image: "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", use_experts: true, output_expert_labels: false } } ); console.log(output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:pip install replicateSet theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:import replicateRun nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run( "nvidia/prismer:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", input={ "task": "caption", "question": "", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": True, "output_expert_labels": False } ) print(output)To learn more, take a look at the guide on getting started with Python.
Set theREPLICATE_API_TOKENenvironment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1", "input": { "task": "caption", "question": "", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false } }' \ https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
You can run this model locally using Cog. First, install Cog:brew install cogIf you don’t have Homebrew, there are other installation options available.
Run this to download the model and run it in your local environment:
cog predict r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1 \ -i 'task="caption"' \ -i 'question=""' \ -i 'model_size="base"' \ -i 'input_image="https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg"' \ -i 'use_experts=true' \ -i 'output_expert_labels=false'To learn more, take a look at the Cog documentation.
Run this to download the model and run it in your local environment:
docker run -d -p 5000:5000 --gpus=all r8.im/nvidia/prismer@sha256:e604611dc43bfabc4eb5cda01eab65a491d74910cf5545da2a189718320873b1curl -s -X POST \ -H "Content-Type: application/json" \ -d $'{ "input": { "task": "caption", "question": "", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false } }' \ http://localhost:5000/predictionsTo learn more, take a look at the Cog documentation.
Output
a man riding a skateboard across a cross walk{ "completed_at": "2023-03-13T20:33:53.752602Z", "created_at": "2023-03-13T20:32:10.928556Z", "data_removed": false, "error": null, "id": "clyqudkpy5fzric3mbmcnyyysa", "input": { "task": "caption", "question": "", "model_size": "base", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false }, "logs": "***** Generating edge *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.46s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.46s/it]\n***** Generating depth *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.32s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.32s/it]\n***** Generating normal *****\nUsing cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.61s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.62s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00, 2.73s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.73s/it]\n***** Generating ocrdet *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.80s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.81s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.58s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.58s/it]", "metrics": { "predict_time": 102.749858, "total_time": 102.824046 }, "output": { "answer": "a man riding a skateboard across a cross walk" }, "started_at": "2023-03-13T20:32:11.002744Z", "status": "succeeded", "urls": { "get": "https://api.replicate.com/v1/predictions/clyqudkpy5fzric3mbmcnyyysa", "cancel": "https://api.replicate.com/v1/predictions/clyqudkpy5fzric3mbmcnyyysa/cancel" }, "version": "569a81a5da2233401dc05a37b2f0d17855eb953623c648956c823136e7f6c3ab" }Generated in***** Generating edge ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.46s/it] 100%|██████████| 1/1 [00:02<00:00, 2.46s/it] ***** Generating depth ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.32s/it] 100%|██████████| 1/1 [00:02<00:00, 2.32s/it] ***** Generating normal ***** Using cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master Loading base model ()...Done. Removing last two layers (global_pool & classifier). 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.61s/it] 100%|██████████| 1/1 [00:02<00:00, 2.62s/it] ***** Generating objdet ***** Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content. 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 100%|██████████| 1/1 [00:02<00:00, 2.73s/it] 100%|██████████| 1/1 [00:02<00:00, 2.73s/it] ***** Generating ocrdet ***** 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.80s/it] 100%|██████████| 1/1 [00:02<00:00, 2.81s/it] ***** Generating segmentation ***** /root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ... 0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:02<00:00, 2.58s/it] 100%|██████████| 1/1 [00:02<00:00, 2.58s/it]
Want to make some of these yourself?
Run this model