chenxwh / lotus

Diffusion-based Visual Foundation Model for High-quality Dense Prediction

Cold

Public
131 runs
L40S
GitHub
Weights
Paper
License

Run with an API

Playground API Examples README Versions

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=chenxwh/lotus

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run chenxwh/lotus using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "chenxwh/lotus:87d8467fb2d23f07f2aa0b9e08585dd60bd3b163bef0c2dc7274be64904f4198",
  {
    input: {
      task: "depth",
      image: "https://replicate.delivery/pbxt/Lka84th0JnWOALVFOCmslNRI7ygy0kUCYNxjpVwZ3nsGTnQo/07.jpg"
    }
  }
);
console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run chenxwh/lotus using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "chenxwh/lotus:87d8467fb2d23f07f2aa0b9e08585dd60bd3b163bef0c2dc7274be64904f4198",
    input={
        "task": "depth",
        "image": "https://replicate.delivery/pbxt/Lka84th0JnWOALVFOCmslNRI7ygy0kUCYNxjpVwZ3nsGTnQo/07.jpg"
    }
)
print(output)

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run chenxwh/lotus using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "87d8467fb2d23f07f2aa0b9e08585dd60bd3b163bef0c2dc7274be64904f4198",
    "input": {
      "task": "depth",
      "image": "https://replicate.delivery/pbxt/Lka84th0JnWOALVFOCmslNRI7ygy0kUCYNxjpVwZ3nsGTnQo/07.jpg"
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can run this model locally using Cog. First, install Cog:

brew install cog

If you don’t have Homebrew, there are other installation options available.

Run this to download the model and run it in your local environment:

cog predict r8.im/chenxwh/lotus@sha256:87d8467fb2d23f07f2aa0b9e08585dd60bd3b163bef0c2dc7274be64904f4198 \
  -i 'task="depth"' \
  -i 'image="https://replicate.delivery/pbxt/Lka84th0JnWOALVFOCmslNRI7ygy0kUCYNxjpVwZ3nsGTnQo/07.jpg"'

To learn more, take a look at the Cog documentation.

Run this to download the model and run it in your local environment:

docker run -d -p 5000:5000 --gpus=all r8.im/chenxwh/lotus@sha256:87d8467fb2d23f07f2aa0b9e08585dd60bd3b163bef0c2dc7274be64904f4198
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d $'{
    "input": {
      "task": "depth",
      "image": "https://replicate.delivery/pbxt/Lka84th0JnWOALVFOCmslNRI7ygy0kUCYNxjpVwZ3nsGTnQo/07.jpg"
    }
  }' \
  http://localhost:5000/predictions

To learn more, take a look at the Cog documentation.

Output

generative

discriminative

Generated in

39.4 seconds

Tweak it ShareReport

Examples

View more examples

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

We present Lotus, a diffusion-based visual foundation model for dense geometry prediction. With minimal training data, Lotus achieves SoTA performance in two key geometry perception tasks, i.e., zero-shot depth and normal estimation. “Avg. Rank” indicates the average ranking across all metrics, where lower values are better. Bar length represents the amount of training data used.

🎓 Citation

If you find our work useful in your research, please consider citing our paper:

@article{he2024lotus,
    title={Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction},
    author={He, Jing and Li, Haodong and Yin, Wei and Liang, Yixun and Li, Leheng and Zhou, Kaiqiang and Liu, Hongbo and Liu, Bingbing and Chen, Ying-Cong},
    journal={arXiv preprint arXiv:2409.18124},
    year={2024}
}