yorickvp / llava-v1.6-vicuna-13b

LLaVA v1.6: Large Language and Vision Assistant (Vicuna-13B)

Cold

Public
3.4M runs
L40S
GitHub
License

Run with an API

Playground API Examples README Versions

Input

image

file

Input image

prompt

*string

Shift + Return to add a new line

What should I take into account when visiting this place?What should I take into account when visiting this place?

Prompt to use for text generation

top_p

number

(minimum: 0, maximum: 1)

When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens

Default: 1

temperature

number

(minimum: 0)

Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic

Default: 0.2

max_tokens

integer

(minimum: 0)

Maximum number of tokens to generate. A word is generally 2-3 tokens

Default: 1024

history

string[]

List of earlier chat messages, alternating roles, starting with user input. Include <image> to specify which message to attach the image to.

Run this model in Node.js with one line of code:

npx create-replicate --model=yorickvp/llava-v1.6-vicuna-13b

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run yorickvp/llava-v1.6-vicuna-13b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "yorickvp/llava-v1.6-vicuna-13b:0603dec596080fa084e26f0ae6d605fc5788ed2b1a0358cd25010619487eae63",
  {
    input: {
      image: "https://replicate.delivery/pbxt/KKRcRQnPriwGsRJVCZTysuIWqHk2zQEJO6z7b1iAhyDjvLJb/waterview.jpg",
      top_p: 1,
      prompt: "What should I take into account when visiting this place?",
      max_tokens: 1024,
      temperature: 0.2
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run yorickvp/llava-v1.6-vicuna-13b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "yorickvp/llava-v1.6-vicuna-13b:0603dec596080fa084e26f0ae6d605fc5788ed2b1a0358cd25010619487eae63",
    input={
        "image": "https://replicate.delivery/pbxt/KKRcRQnPriwGsRJVCZTysuIWqHk2zQEJO6z7b1iAhyDjvLJb/waterview.jpg",
        "top_p": 1,
        "prompt": "What should I take into account when visiting this place?",
        "max_tokens": 1024,
        "temperature": 0.2
    }
)

# The yorickvp/llava-v1.6-vicuna-13b model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/yorickvp/llava-v1.6-vicuna-13b/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run yorickvp/llava-v1.6-vicuna-13b using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "0603dec596080fa084e26f0ae6d605fc5788ed2b1a0358cd25010619487eae63",
    "input": {
      "image": "https://replicate.delivery/pbxt/KKRcRQnPriwGsRJVCZTysuIWqHk2zQEJO6z7b1iAhyDjvLJb/waterview.jpg",
      "top_p": 1,
      "prompt": "What should I take into account when visiting this place?",
      "max_tokens": 1024,
      "temperature": 0.2
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/yorickvp/llava-v1.6-vicuna-13b@sha256:0603dec596080fa084e26f0ae6d605fc5788ed2b1a0358cd25010619487eae63 \
  -i 'image="https://replicate.delivery/pbxt/KKRcRQnPriwGsRJVCZTysuIWqHk2zQEJO6z7b1iAhyDjvLJb/waterview.jpg"' \
  -i 'top_p=1' \
  -i 'prompt="What should I take into account when visiting this place?"' \
  -i 'max_tokens=1024' \
  -i 'temperature=0.2'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/yorickvp/llava-v1.6-vicuna-13b@sha256:0603dec596080fa084e26f0ae6d605fc5788ed2b1a0358cd25010619487eae63
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "image": "https://replicate.delivery/pbxt/KKRcRQnPriwGsRJVCZTysuIWqHk2zQEJO6z7b1iAhyDjvLJb/waterview.jpg",
      "top_p": 1,
      "prompt": "What should I take into account when visiting this place?",
      "max_tokens": 1024,
      "temperature": 0.2
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

When visiting a place like the one shown in the image, which appears to be a serene lake with a dock and surrounded by mountains and forests, you should consider the following: 1. **Weather Conditions**: Be prepared for the local weather. It can change rapidly, especially in mountainous areas, so check the forecast and dress in layers. 2. **Safety**: If you plan to go on the water, make sure you have a life jacket and know how to swim. If you're not a strong swimmer, consider wearing a life jacket even if you're just walking along the shore. 3. **Wildlife**: Be aware of the local wildlife and maintain a safe distance. Do not feed the animals, as this can disrupt their natural behavior and diet. 4. **Leave No Trace**: Respect the environment by not littering and by staying on designated paths or trails. 5. **Emergency Preparedness**: Have a first aid kit and know basic first aid. It's also a good idea to have a way to contact emergency services if needed. 6. **Navigation**: If you're going into the woods or on the water, have a map or GPS device and know how to use it. 7. **Hydration and Nutrition**: Bring enough water and food for your visit, especially if you'll be active. 8. **Clothing and Gear**: Wear appropriate clothing for the weather and activities you plan to do. Bring sturdy footwear if you plan to hike or walk on uneven terrain. 9. **Local Regulations**: Follow any posted rules or regulations, which might include fishing regulations, fire restrictions, or camping rules. 10. **Respect for Others**: Be considerate of other visitors. Keep noise to a minimum and be mindful of others who may be trying to enjoy the same natural beauty as you. Enjoy your visit and remember to take care of the environment and your fellow visitors.

{
  "completed_at": "2024-02-01T16:37:38.082678Z",
  "created_at": "2024-02-01T16:35:26.604812Z",
  "data_removed": false,
  "error": null,
  "id": "puarwfdbp5xfjtsnhxw6ju5ety",
  "input": {
    "image": "https://replicate.delivery/pbxt/KKRcRQnPriwGsRJVCZTysuIWqHk2zQEJO6z7b1iAhyDjvLJb/waterview.jpg",
    "top_p": 1,
    "prompt": "What should I take into account when visiting this place?",
    "max_tokens": 1024,
    "temperature": 0.2
  },
  "logs": null,
  "metrics": {
    "predict_time": 29.647259,
    "total_time": 131.477866
  },
  "output": [
    "When ",
    "visiting ",
    "a ",
    "place ",
    "like ",
    "the ",
    "one ",
    "shown ",
    "in ",
    "the ",
    "image, ",
    "which ",
    "appears ",
    "to ",
    "be ",
    "a ",
    "serene ",
    "lake ",
    "with ",
    "a ",
    "dock ",
    "and ",
    "surrounded ",
    "by ",
    "mountains ",
    "and ",
    "forests, ",
    "you ",
    "should ",
    "consider ",
    "the ",
    "following:\n",
    "\n",
    "1. ",
    "**Weather ",
    "Conditions**: ",
    "Be ",
    "prepared ",
    "for ",
    "the ",
    "local ",
    "weather. ",
    "It ",
    "can ",
    "change ",
    "rapidly, ",
    "especially ",
    "in ",
    "mountainous ",
    "areas, ",
    "so ",
    "check ",
    "the ",
    "forecast ",
    "and ",
    "dress ",
    "in ",
    "layers.\n",
    "\n",
    "2. ",
    "**Safety**: ",
    "If ",
    "you ",
    "plan ",
    "to ",
    "go ",
    "on ",
    "the ",
    "water, ",
    "make ",
    "sure ",
    "you ",
    "have ",
    "a ",
    "life ",
    "jacket ",
    "and ",
    "know ",
    "how ",
    "to ",
    "swim. ",
    "If ",
    "you're ",
    "not ",
    "a ",
    "strong ",
    "swimmer, ",
    "consider ",
    "wearing ",
    "a ",
    "life ",
    "jacket ",
    "even ",
    "if ",
    "you're ",
    "just ",
    "walking ",
    "along ",
    "the ",
    "shore.\n",
    "\n",
    "3. ",
    "**Wildlife**: ",
    "Be ",
    "aware ",
    "of ",
    "the ",
    "local ",
    "wildlife ",
    "and ",
    "maintain ",
    "a ",
    "safe ",
    "distance. ",
    "Do ",
    "not ",
    "feed ",
    "the ",
    "animals, ",
    "as ",
    "this ",
    "can ",
    "disrupt ",
    "their ",
    "natural ",
    "behavior ",
    "and ",
    "diet.\n",
    "\n",
    "4. ",
    "**Leave ",
    "No ",
    "Trace**: ",
    "Respect ",
    "the ",
    "environment ",
    "by ",
    "not ",
    "littering ",
    "and ",
    "by ",
    "staying ",
    "on ",
    "designated ",
    "paths ",
    "or ",
    "trails.\n",
    "\n",
    "5. ",
    "**Emergency ",
    "Preparedness**: ",
    "Have ",
    "a ",
    "first ",
    "aid ",
    "kit ",
    "and ",
    "know ",
    "basic ",
    "first ",
    "aid. ",
    "It's ",
    "also ",
    "a ",
    "good ",
    "idea ",
    "to ",
    "have ",
    "a ",
    "way ",
    "to ",
    "contact ",
    "emergency ",
    "services ",
    "if ",
    "needed.\n",
    "\n",
    "6. ",
    "**Navigation**: ",
    "If ",
    "you're ",
    "going ",
    "into ",
    "the ",
    "woods ",
    "or ",
    "on ",
    "the ",
    "water, ",
    "have ",
    "a ",
    "map ",
    "or ",
    "GPS ",
    "device ",
    "and ",
    "know ",
    "how ",
    "to ",
    "use ",
    "it.\n",
    "\n",
    "7. ",
    "**Hydration ",
    "and ",
    "Nutrition**: ",
    "Bring ",
    "enough ",
    "water ",
    "and ",
    "food ",
    "for ",
    "your ",
    "visit, ",
    "especially ",
    "if ",
    "you'll ",
    "be ",
    "active.\n",
    "\n",
    "8. ",
    "**Clothing ",
    "and ",
    "Gear**: ",
    "Wear ",
    "appropriate ",
    "clothing ",
    "for ",
    "the ",
    "weather ",
    "and ",
    "activities ",
    "you ",
    "plan ",
    "to ",
    "do. ",
    "Bring ",
    "sturdy ",
    "footwear ",
    "if ",
    "you ",
    "plan ",
    "to ",
    "hike ",
    "or ",
    "walk ",
    "on ",
    "uneven ",
    "terrain.\n",
    "\n",
    "9. ",
    "**Local ",
    "Regulations**: ",
    "Follow ",
    "any ",
    "posted ",
    "rules ",
    "or ",
    "regulations, ",
    "which ",
    "might ",
    "include ",
    "fishing ",
    "regulations, ",
    "fire ",
    "restrictions, ",
    "or ",
    "camping ",
    "rules.\n",
    "\n",
    "10. ",
    "**Respect ",
    "for ",
    "Others**: ",
    "Be ",
    "considerate ",
    "of ",
    "other ",
    "visitors. ",
    "Keep ",
    "noise ",
    "to ",
    "a ",
    "minimum ",
    "and ",
    "be ",
    "mindful ",
    "of ",
    "others ",
    "who ",
    "may ",
    "be ",
    "trying ",
    "to ",
    "enjoy ",
    "the ",
    "same ",
    "natural ",
    "beauty ",
    "as ",
    "you.\n",
    "\n",
    "Enjoy ",
    "your ",
    "visit ",
    "and ",
    "remember ",
    "to ",
    "take ",
    "care ",
    "of ",
    "the ",
    "environment ",
    "and ",
    "your ",
    "fellow ",
    "visitors."
  ],
  "started_at": "2024-02-01T16:37:08.435419Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://streaming-api.svc.us.c.replicate.net/v1/streams/g347d3dyff644f6ao5rh5v3k3upd5wk5ycgjd7jr65rrqpws3zmq",
    "get": "https://api.replicate.com/v1/predictions/puarwfdbp5xfjtsnhxw6ju5ety",
    "cancel": "https://api.replicate.com/v1/predictions/puarwfdbp5xfjtsnhxw6ju5ety/cancel"
  },
  "version": "ba573f1c46b08ce9ea1a593048dad05a2309b05bd867ebb700009af2c61ee2d6"
}

Generated in

29.6 seconds

Tweak it Report

This example was created by a different version, yorickvp/llava-v1.6-vicuna-13b:ba573f1c.

Run time and cost

This model costs approximately $0.012 to run on Replicate, or 83 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 13 seconds. The predict time for this model varies significantly based on the inputs.

Readme

Check out the different LLaVA’s on Replicate:

Name	Version	Base	Size	Finetunable
v1.5 - Vicuna-13B	v1.5	Vicuna	13B	Yes
v1.6 - Vicuna-13B	v1.6	Vicuna	13B	No
v1.6 - Vicuna-7B	v1.6	Vicuna	7B	No
v1.6 - Mistral-7B	v1.6	Mistral	7B	No
v1.6 - Nous-Hermes-2-34B	v1.6	Nous-Hermes-2	34B	No

🌋 LLaVA v1.6: Large Language and Vision Assistant

Visual instruction tuning towards large language and vision models with GPT-4 level capabilities.

[Project Page] [Demo] [Data] [Model Zoo]

Improved Baselines with Visual Instruction Tuning [Paper]
Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee

Visual Instruction Tuning (NeurIPS 2023, Oral) [Paper]
Haotian Liu*, Chunyuan Li*, Qingyang Wu, Yong Jae Lee (*Equal Contribution)

LLaVA v1.6 changes

LLaVA-1.6 is out! With additional scaling to LLaVA-1.5, LLaVA-1.6-34B outperforms Gemini Pro on some benchmarks. It can now process 4x more pixels and perform more tasks/applications than before. Check out the blog post!

Summary

LLaVA represents a novel end-to-end trained large multimodal model that combines a vision encoder and Vicuna for general-purpose visual and language understanding, achieving impressive chat capabilities mimicking spirits of the multimodal GPT-4 and setting a new state-of-the-art accuracy on Science QA.