Versions
e604611d
Latest
ef479e09
308eb68a
You're looking at a specific version of this model. Jump to the model overview.
Input
Run this model in Node.js with one line of code:
npx create-replicate --model=nvidia/prismer
or set up a project from scratch
Install Replicate’s Node.js client library:
npm install replicateSet the
REPLICATE_API_TOKEN environment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import and set up the client:
import Replicate from "replicate";
const replicate = new Replicate({
auth: process.env.REPLICATE_API_TOKEN,
});Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
const output = await replicate.run(
"nvidia/prismer:ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017",
{
input: {
task: "vqa",
question: "what is the man sitting down doing?",
input_image: "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg",
use_experts: true,
output_expert_labels: false
}
}
);
console.log(output);To learn more, take a look at the guide on getting started with Node.js.
Install Replicate’s Python client library:
pip install replicateSet the
REPLICATE_API_TOKEN environment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Import the client:
import replicateRun nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
output = replicate.run(
"nvidia/prismer:ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017",
input={
"task": "vqa",
"question": "what is the man sitting down doing?",
"input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg",
"use_experts": True,
"output_expert_labels": False
}
)
print(output)To learn more, take a look at the guide on getting started with Python.
Set the
REPLICATE_API_TOKEN environment variable:export REPLICATE_API_TOKEN=<paste-your-token-here>Find your API token in your account settings.
Run nvidia/prismer using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.
curl -s -X POST \
-H "Authorization: Bearer $REPLICATE_API_TOKEN" \
-H "Content-Type: application/json" \
-H "Prefer: wait" \
-d $'{
"version": "ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017",
"input": {
"task": "vqa",
"question": "what is the man sitting down doing?",
"input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg",
"use_experts": true,
"output_expert_labels": false
}
}' \
https://api.replicate.com/v1/predictionsTo learn more, take a look at Replicate’s HTTP API reference docs.
You can run this model locally using Cog. First, install Cog:
brew install cogIf you don’t have Homebrew, there are other installation options available.
Run this to download the model and run it in your local environment:
cog predict r8.im/nvidia/prismer@sha256:ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017 \
-i 'task="vqa"' \
-i 'question="what is the man sitting down doing?"' \
-i 'input_image="https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg"' \
-i 'use_experts=true' \
-i 'output_expert_labels=false'To learn more, take a look at the Cog documentation.
Run this to download the model and run it in your local environment:
docker run -d -p 5000:5000 --gpus=all r8.im/nvidia/prismer@sha256:ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017curl -s -X POST \ -H "Content-Type: application/json" \ -d $'{ "input": { "task": "vqa", "question": "what is the man sitting down doing?", "input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg", "use_experts": true, "output_expert_labels": false } }' \ http://localhost:5000/predictions
To learn more, take a look at the Cog documentation.
Add a payment method to run this model.
Sign in with GitHub
By signing in, you agree to our
terms of service and privacy policy
Output
riding bike
{
"completed_at": "2023-03-11T22:26:46.233154Z",
"created_at": "2023-03-11T22:25:12.647330Z",
"data_removed": false,
"error": null,
"id": "4x5slvr76fbmhbb5nsuwxau7ri",
"input": {
"task": "vqa",
"question": "what is the man sitting down doing?",
"input_image": "https://replicate.delivery/pbxt/ISSa1VolSjpqROlBZm9FSrkC3PL0mJjwIQfeYNLYO8GowGuP/1.jpeg",
"use_experts": true,
"output_expert_labels": false
},
"logs": "***** Generating edge *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.49s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.49s/it]\n***** Generating depth *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.50s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.50s/it]\n***** Generating normal *****\nUsing cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master\nLoading base model ()...Done.\nRemoving last two layers (global_pool & classifier).\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.69s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.69s/it]\n***** Generating objdet *****\nLoading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.\n 0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\n100%|██████████| 1/1 [00:02<00:00, 2.58s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.58s/it]\n***** Generating ocrdet *****\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.94s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.95s/it]\n***** Generating segmentation *****\n/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)\nreturn _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]\nWeight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\nerror in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.\n 0%| | 0/1 [00:00<?, ?it/s]\n100%|██████████| 1/1 [00:02<00:00, 2.56s/it]\n100%|██████████| 1/1 [00:02<00:00, 2.56s/it]",
"metrics": {
"predict_time": 93.519066,
"total_time": 93.585824
},
"output": {
"answer": "riding bike"
},
"started_at": "2023-03-11T22:25:12.714088Z",
"status": "succeeded",
"urls": {
"get": "https://api.replicate.com/v1/predictions/4x5slvr76fbmhbb5nsuwxau7ri",
"cancel": "https://api.replicate.com/v1/predictions/4x5slvr76fbmhbb5nsuwxau7ri/cancel"
},
"version": "ef479e09f64a641964c3d573592f8a8964b898444f0b18bce4bb6fac2b0e7017"
}Generated in
***** Generating edge *****
0%| | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00, 2.49s/it]
100%|██████████| 1/1 [00:02<00:00, 2.49s/it]
***** Generating depth *****
0%| | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00, 2.50s/it]
100%|██████████| 1/1 [00:02<00:00, 2.50s/it]
***** Generating normal *****
Using cache found in /root/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master
Loading base model ()...Done.
Removing last two layers (global_pool & classifier).
0%| | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00, 2.69s/it]
100%|██████████| 1/1 [00:02<00:00, 2.69s/it]
***** Generating objdet *****
Loading config experts/obj_detection/configs/Base-CRCNN-COCO.yaml with yaml.unsafe_load. Your machine may be at risk if the file contains malicious content.
0%| | 0/1 [00:00<?, ?it/s]/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
100%|██████████| 1/1 [00:02<00:00, 2.58s/it]
100%|██████████| 1/1 [00:02<00:00, 2.58s/it]
***** Generating ocrdet *****
0%| | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00, 2.94s/it]
100%|██████████| 1/1 [00:02<00:00, 2.95s/it]
***** Generating segmentation *****
/root/.pyenv/versions/3.10.10/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Weight format of MultiScaleMaskedTransformerDecoder have changed! Please upgrade your models. Applying automatic conversion now ...
error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.
error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.
error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.
error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.
error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.
error in ms_deformable_im2col_cuda: the provided PTX was compiled with an unsupported toolchain.
0%| | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:02<00:00, 2.56s/it]
100%|██████████| 1/1 [00:02<00:00, 2.56s/it]