pku-yuangroup/llava-cot – Run with an API on Replicate

pku-yuangroup / llava-cot

Let Vision Language Models Reason Step-by-Step

Cold

Public
35 runs
L40S
GitHub
Weights
Paper
License

Run with an API

Playground API Examples README Versions

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=pku-yuangroup/llava-cot

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run pku-yuangroup/llava-cot using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "pku-yuangroup/llava-cot:40c17578a6b98679fc08e1a92fcef1b8000f857bf300a381091a13a67c28754d",
  {
    input: {
      image: "https://replicate.delivery/pbxt/M4VFa6E18it1vazUahiTB5RjNjDoajLbHcpMgMFBhJvmgGdh/Baklava%281%29.png",
      top_p: 0.95,
      prompt: "how to make this pastry",
      temperature: 0.9,
      max_new_tokens: 1024
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run pku-yuangroup/llava-cot using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "pku-yuangroup/llava-cot:40c17578a6b98679fc08e1a92fcef1b8000f857bf300a381091a13a67c28754d",
    input={
        "image": "https://replicate.delivery/pbxt/M4VFa6E18it1vazUahiTB5RjNjDoajLbHcpMgMFBhJvmgGdh/Baklava%281%29.png",
        "top_p": 0.95,
        "prompt": "how to make this pastry",
        "temperature": 0.9,
        "max_new_tokens": 1024
    }
)

# The pku-yuangroup/llava-cot model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/pku-yuangroup/llava-cot/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run pku-yuangroup/llava-cot using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "40c17578a6b98679fc08e1a92fcef1b8000f857bf300a381091a13a67c28754d",
    "input": {
      "image": "https://replicate.delivery/pbxt/M4VFa6E18it1vazUahiTB5RjNjDoajLbHcpMgMFBhJvmgGdh/Baklava%281%29.png",
      "top_p": 0.95,
      "prompt": "how to make this pastry",
      "temperature": 0.9,
      "max_new_tokens": 1024
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

<SUMMARY> To solve the problem, I will analyze the image to identify the key characteristics of the pastry, then describe how to make it by following standard steps for preparation. </SUMMARY> <CAPTION> The image shows a tray of baklava, a pastry made with layers of phyllo dough, filled with chopped nuts, and topped with a sweet syrup. The pastry is golden brown, indicating it's been baked. </CAPTION> <REASONING> To make baklava, start by preparing the nuts by roasting them. Then, assemble the layers by placing phyllo dough sheets in a tray, brushing them with butter. Spread the nut mixture evenly over the phyllo layers. Repeat this process until the layers are complete. Cut the baklava into diamond shapes and bake until golden. Finally, drizzle with syrup. </REASONING> <CONCLUSION> To make this delicious baklava, follow these steps: 1. Begin by preparing the nuts. You can use either walnuts, pistachios, or a combination of both. Roast the nuts in a 350°F oven for about 10 minutes until they are fragrant and slightly toasted. Let them cool down. 2. In a large bowl, combine the cooled nuts with the sugar and cinnamon. Mix well until the nuts are evenly coated with the sugar and cinnamon mixture. 3. To assemble the baklava, preheat your oven to 350°F. Take a rectangular tray or baking dish and line the bottom with a sheet of phyllo dough. Brush the phyllo with melted butter, then repeat this process until you have about 8-10 layers of phyllo dough. 4. Once you've assembled the phyllo dough layers, spread the nut mixture evenly over the top layer of phyllo. 5. Carefully roll the baklava tightly from one end of the tray to the other, applying gentle pressure to ensure it's compact. Use a knife or sharp object to cut the baklava into diamond shapes. 6. Place the baklava in the preheated oven and bake for about 45 minutes, or until the phyllo is golden brown and crisp. 7. Remove the baklava from the oven and let it cool for a few minutes. While it's still warm, drizzle the honey syrup evenly over the top of the baklava. 8. Let the baklava cool completely before serving. This will allow the flavors to meld together and the pastry to absorb the honey syrup. Enjoy your delicious homemade baklava! </CONCLUSION>

{
  "completed_at": "2024-12-02T02:52:29.137832Z",
  "created_at": "2024-12-02T02:51:27.050000Z",
  "data_removed": false,
  "error": null,
  "id": "5mjjshdbs9rma0ckgn3tsz01ym",
  "input": {
    "image": "https://replicate.delivery/pbxt/M4VFa6E18it1vazUahiTB5RjNjDoajLbHcpMgMFBhJvmgGdh/Baklava%281%29.png",
    "top_p": 0.95,
    "prompt": "how to make this pastry",
    "temperature": 0.9,
    "max_new_tokens": 1024
  },
  "logs": null,
  "metrics": {
    "predict_time": 25.45225957,
    "total_time": 62.087832
  },
  "output": [
    "",
    "",
    "",
    "",
    "<SUMMARY> \n",
    "",
    "To ",
    "solve ",
    "the ",
    "",
    "problem, ",
    "I ",
    "will ",
    "analyze ",
    "the ",
    "image ",
    "to ",
    "identify ",
    "the ",
    "key ",
    "characteristics ",
    "of ",
    "the ",
    "",
    "pastry, ",
    "then ",
    "describe ",
    "how ",
    "to ",
    "make ",
    "it ",
    "by ",
    "following ",
    "standard ",
    "steps ",
    "for ",
    "preparation.\n",
    "",
    "",
    "",
    "</SUMMARY>\n\n",
    "",
    "",
    "",
    "",
    "<CAPTION> \n",
    "",
    "The ",
    "image ",
    "shows ",
    "a ",
    "tray ",
    "of ",
    "",
    "",
    "baklava, ",
    "a ",
    "pastry ",
    "made ",
    "with ",
    "layers ",
    "of ",
    "",
    "",
    "phyllo ",
    "",
    "dough, ",
    "filled ",
    "with ",
    "chopped ",
    "",
    "nuts, ",
    "and ",
    "topped ",
    "with ",
    "a ",
    "sweet ",
    "",
    "syrup. ",
    "The ",
    "pastry ",
    "is ",
    "golden ",
    "",
    "brown, ",
    "indicating ",
    "",
    "it's ",
    "been ",
    "baked.\n",
    "",
    "",
    "",
    "</CAPTION>\n\n",
    "",
    "",
    "",
    "",
    "",
    "<REASONING> \n",
    "",
    "To ",
    "make ",
    "",
    "",
    "baklava, ",
    "start ",
    "by ",
    "preparing ",
    "the ",
    "nuts ",
    "by ",
    "",
    "roasting ",
    "",
    "them. ",
    "",
    "Then, ",
    "assemble ",
    "the ",
    "layers ",
    "by ",
    "placing ",
    "",
    "",
    "phyllo ",
    "dough ",
    "sheets ",
    "in ",
    "a ",
    "",
    "tray, ",
    "brushing ",
    "them ",
    "with ",
    "",
    "butter. ",
    "Spread ",
    "the ",
    "nut ",
    "mixture ",
    "evenly ",
    "over ",
    "the ",
    "",
    "",
    "phyllo ",
    "",
    "layers. ",
    "Repeat ",
    "this ",
    "process ",
    "until ",
    "the ",
    "layers ",
    "are ",
    "",
    "complete. ",
    "Cut ",
    "the ",
    "",
    "baklava ",
    "into ",
    "diamond ",
    "shapes ",
    "and ",
    "bake ",
    "until ",
    "",
    "golden. ",
    "",
    "Finally, ",
    "",
    "drizzle ",
    "with ",
    "syrup.\n",
    "",
    "",
    "",
    "",
    "</REASONING>\n\n",
    "",
    "",
    "",
    "",
    "<CONCLUSION> \n",
    "",
    "To ",
    "make ",
    "this ",
    "delicious ",
    "",
    "",
    "baklava, ",
    "follow ",
    "these ",
    "steps:\n\n",
    "",
    "",
    "1. ",
    "Begin ",
    "by ",
    "preparing ",
    "the ",
    "",
    "nuts. ",
    "You ",
    "can ",
    "use ",
    "either ",
    "",
    "",
    "walnuts, ",
    "",
    "",
    "",
    "pistachios, ",
    "or ",
    "a ",
    "combination ",
    "of ",
    "",
    "both. ",
    "",
    "Roast ",
    "the ",
    "nuts ",
    "in ",
    "a ",
    "",
    "",
    "350°F ",
    "oven ",
    "for ",
    "about ",
    "",
    "10 ",
    "minutes ",
    "until ",
    "they ",
    "are ",
    "",
    "fragrant ",
    "and ",
    "slightly ",
    "",
    "toasted. ",
    "Let ",
    "them ",
    "cool ",
    "down.\n\n",
    "",
    "",
    "2. ",
    "In ",
    "a ",
    "large ",
    "",
    "bowl, ",
    "combine ",
    "the ",
    "cooled ",
    "nuts ",
    "with ",
    "the ",
    "sugar ",
    "and ",
    "",
    "cinnamon. ",
    "Mix ",
    "well ",
    "until ",
    "the ",
    "nuts ",
    "are ",
    "evenly ",
    "coated ",
    "with ",
    "the ",
    "sugar ",
    "and ",
    "cinnamon ",
    "mixture.\n\n",
    "",
    "",
    "3. ",
    "To ",
    "assemble ",
    "the ",
    "",
    "",
    "baklava, ",
    "",
    "preheat ",
    "your ",
    "oven ",
    "to ",
    "",
    "",
    "",
    "350°F. ",
    "Take ",
    "a ",
    "rectangular ",
    "tray ",
    "or ",
    "baking ",
    "dish ",
    "and ",
    "line ",
    "the ",
    "bottom ",
    "with ",
    "a ",
    "sheet ",
    "of ",
    "",
    "",
    "phyllo ",
    "",
    "dough. ",
    "Brush ",
    "the ",
    "",
    "",
    "phyllo ",
    "with ",
    "melted ",
    "",
    "butter, ",
    "then ",
    "repeat ",
    "this ",
    "process ",
    "until ",
    "you ",
    "have ",
    "about ",
    "",
    "",
    "",
    "8-10 ",
    "layers ",
    "of ",
    "",
    "",
    "phyllo ",
    "dough.\n\n",
    "",
    "",
    "4. ",
    "Once ",
    "",
    "you've ",
    "assembled ",
    "the ",
    "",
    "",
    "phyllo ",
    "dough ",
    "",
    "layers, ",
    "spread ",
    "the ",
    "nut ",
    "mixture ",
    "evenly ",
    "over ",
    "the ",
    "top ",
    "layer ",
    "of ",
    "",
    "",
    "phyllo.\n\n",
    "",
    "",
    "5. ",
    "",
    "Carefully ",
    "roll ",
    "the ",
    "",
    "baklava ",
    "tightly ",
    "from ",
    "one ",
    "end ",
    "of ",
    "the ",
    "tray ",
    "to ",
    "the ",
    "",
    "other, ",
    "applying ",
    "gentle ",
    "pressure ",
    "to ",
    "ensure ",
    "",
    "it's ",
    "",
    "compact. ",
    "Use ",
    "a ",
    "knife ",
    "or ",
    "sharp ",
    "object ",
    "to ",
    "cut ",
    "the ",
    "",
    "baklava ",
    "into ",
    "diamond ",
    "shapes.\n\n",
    "",
    "",
    "6. ",
    "Place ",
    "the ",
    "",
    "baklava ",
    "in ",
    "the ",
    "",
    "",
    "preheated ",
    "oven ",
    "and ",
    "bake ",
    "for ",
    "about ",
    "",
    "45 ",
    "",
    "minutes, ",
    "or ",
    "until ",
    "the ",
    "",
    "",
    "phyllo ",
    "is ",
    "golden ",
    "brown ",
    "and ",
    "crisp.\n\n",
    "",
    "",
    "7. ",
    "Remove ",
    "the ",
    "",
    "baklava ",
    "from ",
    "the ",
    "oven ",
    "and ",
    "let ",
    "it ",
    "cool ",
    "for ",
    "a ",
    "few ",
    "",
    "minutes. ",
    "While ",
    "",
    "it's ",
    "still ",
    "",
    "warm, ",
    "",
    "drizzle ",
    "the ",
    "honey ",
    "syrup ",
    "evenly ",
    "over ",
    "the ",
    "top ",
    "of ",
    "the ",
    "",
    "baklava.\n\n",
    "",
    "",
    "8. ",
    "Let ",
    "the ",
    "",
    "baklava ",
    "cool ",
    "completely ",
    "before ",
    "",
    "serving. ",
    "This ",
    "will ",
    "allow ",
    "the ",
    "flavors ",
    "to ",
    "meld ",
    "together ",
    "and ",
    "the ",
    "pastry ",
    "to ",
    "absorb ",
    "the ",
    "honey ",
    "syrup.\n\n",
    "",
    "Enjoy ",
    "your ",
    "delicious ",
    "homemade ",
    "",
    "baklava!\n",
    "",
    "",
    "",
    "",
    "",
    "</CONCLUSION>"
  ],
  "started_at": "2024-12-02T02:52:03.685573Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://stream-b.svc.ric1.c.replicate.net/v1/streams/pcl4bncrreukog6mi4bi5tgtgx4mv6tg6bpv4pzp65pvwiuk7pta",
    "get": "https://api.replicate.com/v1/predictions/5mjjshdbs9rma0ckgn3tsz01ym",
    "cancel": "https://api.replicate.com/v1/predictions/5mjjshdbs9rma0ckgn3tsz01ym/cancel"
  },
  "version": "40c17578a6b98679fc08e1a92fcef1b8000f857bf300a381091a13a67c28754d"
}

Generated in

25.5 seconds

Tweak it Share Report

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

🔥 Highlights

LLaVA-CoT is the first visual language model capable of spontaneous, systematic reasoning, similar to GPT-o1!

Our 11B model outperforms Gemini-1.5-pro,GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct on six challenging multimodal benchmarks.

📝 Citation

If you find this paper useful, please consider staring 🌟 this repo and citing 📑 our paper:

@misc{xu2024llavacot,
      title={LLaVA-CoT: Let Vision Language Models Reason Step-by-Step}, 
      author={Guowei Xu and Peng Jin and Hao Li and Yibing Song and Lichao Sun and Li Yuan},
      year={2024},
      eprint={2411.10440},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.10440}, 
}

🙏 Acknowledgement

The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
The service is a research preview intended for non-commercial use only, subject to LLAMA 3.2 COMMUNITY LICENSE AGREEMENT, and Terms of Use of the data generated by OpenAI. Please contact us if you find any potential violations.
The template is modified from Chat-Univi and LLaVA.