deepseek-ai / deepseek-vl-7b-base

DeepSeek-VL: An open-source Vision-Language Model designed for real-world vision and language understanding applications

Cold

Public
3.5K runs
L40S
GitHub
Paper
License

Run with an API

Playground API Examples README Versions

Input

Run this model in Node.js with one line of code:

npx create-replicate --model=deepseek-ai/deepseek-vl-7b-base

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run deepseek-ai/deepseek-vl-7b-base using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "deepseek-ai/deepseek-vl-7b-base:d1823e6f68cd3d57f2d315b9357dfa85f53817120ae0de8d2b95fbc8e93a1385",
  {
    input: {
      image: "https://replicate.delivery/pbxt/KYKAXfcjSZ7uEkwQoYG4SFXRJpbGcdKhWow4Ul0y4Qkjw6wW/training_pipelines.png",
      prompt: "Describe each stage of this image",
      max_new_tokens: 512
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run deepseek-ai/deepseek-vl-7b-base using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "deepseek-ai/deepseek-vl-7b-base:d1823e6f68cd3d57f2d315b9357dfa85f53817120ae0de8d2b95fbc8e93a1385",
    input={
        "image": "https://replicate.delivery/pbxt/KYKAXfcjSZ7uEkwQoYG4SFXRJpbGcdKhWow4Ul0y4Qkjw6wW/training_pipelines.png",
        "prompt": "Describe each stage of this image",
        "max_new_tokens": 512
    }
)

# The deepseek-ai/deepseek-vl-7b-base model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/deepseek-ai/deepseek-vl-7b-base/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run deepseek-ai/deepseek-vl-7b-base using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "d1823e6f68cd3d57f2d315b9357dfa85f53817120ae0de8d2b95fbc8e93a1385",
    "input": {
      "image": "https://replicate.delivery/pbxt/KYKAXfcjSZ7uEkwQoYG4SFXRJpbGcdKhWow4Ul0y4Qkjw6wW/training_pipelines.png",
      "prompt": "Describe each stage of this image",
      "max_new_tokens": 512
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

The image depicts a three-stage process for training a vision-language model. 1. Stage 1: Training VL Adapter: In this stage, a vision-language adapter is trained using supervised fine-tuning. The adapter is trained on image-text pairs and pure language sequences. 2. Stage 2: Joint VL Pre-training: In this stage, a joint vision-language model is pre-trained using self-supervised learning. The model is trained on image-text pairs and pure language sequences. 3. Stage 3: Supervised Fine-tuning: In this stage, the model is fine-tuned on supervised tasks using image-text pairs and pure language sequences. The model is trained using a hybrid vision-language adapter, which combines a vision-language adapter with a language model. The model is trained on a variety of tasks, including image captioning, visual question answering, and visual reasoning. The model is able to understand the visual content of an image and generate a natural language description or answer.

{
  "completed_at": "2024-03-11T19:55:28.030312Z",
  "created_at": "2024-03-11T19:54:05.161849Z",
  "data_removed": false,
  "error": null,
  "id": "sghdgetbu4kminwy4vmljtugou",
  "input": {
    "image": "https://replicate.delivery/pbxt/KYKAXfcjSZ7uEkwQoYG4SFXRJpbGcdKhWow4Ul0y4Qkjw6wW/training_pipelines.png",
    "prompt": "Describe each stage of this image",
    "max_new_tokens": 512
  },
  "logs": null,
  "metrics": {
    "predict_time": 8.43218,
    "total_time": 82.868463
  },
  "output": [
    " ",
    "The ",
    "image ",
    "depicts ",
    "a ",
    "",
    "",
    "three-stage ",
    "process ",
    "for ",
    "training ",
    "a ",
    "",
    "",
    "vision-language ",
    "",
    "model.\n",
    "\n",
    "",
    "",
    "1. ",
    "Stage ",
    "",
    "",
    "1: ",
    "Training ",
    "VL ",
    "",
    "Adapter: ",
    "In ",
    "this ",
    "",
    "stage, ",
    "a ",
    "",
    "",
    "vision-language ",
    "adapter ",
    "is ",
    "trained ",
    "using ",
    "supervised ",
    "",
    "",
    "",
    "fine-tuning. ",
    "The ",
    "adapter ",
    "is ",
    "trained ",
    "on ",
    "",
    "",
    "image-text ",
    "pairs ",
    "and ",
    "pure ",
    "language ",
    "",
    "sequences.\n",
    "",
    "",
    "2. ",
    "Stage ",
    "",
    "",
    "2: ",
    "Joint ",
    "VL ",
    "",
    "",
    "",
    "Pre-training: ",
    "In ",
    "this ",
    "",
    "stage, ",
    "a ",
    "joint ",
    "",
    "",
    "vision-language ",
    "model ",
    "is ",
    "",
    "",
    "pre-trained ",
    "using ",
    "",
    "",
    "self-supervised ",
    "",
    "learning. ",
    "The ",
    "model ",
    "is ",
    "trained ",
    "on ",
    "",
    "",
    "image-text ",
    "pairs ",
    "and ",
    "pure ",
    "language ",
    "",
    "sequences.\n",
    "",
    "",
    "3. ",
    "Stage ",
    "",
    "",
    "3: ",
    "",
    "Supervised ",
    "",
    "",
    "",
    "Fine-tuning: ",
    "In ",
    "this ",
    "",
    "stage, ",
    "the ",
    "model ",
    "is ",
    "",
    "",
    "fine-tuned ",
    "on ",
    "supervised ",
    "tasks ",
    "using ",
    "",
    "",
    "image-text ",
    "pairs ",
    "and ",
    "pure ",
    "language ",
    "",
    "sequences.\n",
    "\n",
    "",
    "The ",
    "model ",
    "is ",
    "trained ",
    "using ",
    "a ",
    "hybrid ",
    "",
    "",
    "vision-language ",
    "",
    "adapter, ",
    "which ",
    "combines ",
    "a ",
    "",
    "",
    "vision-language ",
    "adapter ",
    "with ",
    "a ",
    "language ",
    "",
    "model. ",
    "The ",
    "model ",
    "is ",
    "trained ",
    "on ",
    "a ",
    "variety ",
    "of ",
    "",
    "tasks, ",
    "including ",
    "image ",
    "",
    "",
    "captioning, ",
    "visual ",
    "question ",
    "",
    "answering, ",
    "and ",
    "visual ",
    "",
    "reasoning. ",
    "The ",
    "model ",
    "is ",
    "able ",
    "to ",
    "understand ",
    "the ",
    "visual ",
    "content ",
    "of ",
    "an ",
    "image ",
    "and ",
    "generate ",
    "a ",
    "natural ",
    "language ",
    "description ",
    "or ",
    "",
    "",
    "answer."
  ],
  "started_at": "2024-03-11T19:55:19.598132Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://streaming-api.svc.us.c.replicate.net/v1/streams/6duul3ruer2sfbowces56rgdkofnjyeaxi36sx2vggdlhdnxlq7a",
    "get": "https://api.replicate.com/v1/predictions/sghdgetbu4kminwy4vmljtugou",
    "cancel": "https://api.replicate.com/v1/predictions/sghdgetbu4kminwy4vmljtugou/cancel"
  },
  "version": "d1823e6f68cd3d57f2d315b9357dfa85f53817120ae0de8d2b95fbc8e93a1385"
}

Generated in

8.4 seconds

Tweak it Report

Run time and cost

This model costs approximately $0.0076 to run on Replicate, or 131 runs per $1, but this varies depending on your inputs. It is also open source and you can run it on your own computer with Docker.

This model runs on Nvidia L40S GPU hardware. Predictions typically complete within 8 seconds. The predict time for this model varies significantly based on the inputs.

Readme

DeepSeek-VL-7b-base

Introduction

Introducing DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. DeepSeek-VL possesses general multimodal understanding capabilities, capable of processing logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelligence in complex scenarios.

DeepSeek-VL: Towards Real-World Vision-Language Understanding

Github Repository

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, Chong Ruan (*Equal Contribution, Project Leader)

Model Summary

DeepSeek-VL-7b-base uses the SigLIP-L and SAM-B as the hybrid vision encoder supporting 1024 x 1024 image input and is constructed based on the DeepSeek-LLM-7b-base which is trained on an approximate corpus of 2T text tokens. The whole DeepSeek-VL-7b-base model is finally trained around 400B vision-language tokens.

License

This code repository is licensed under the MIT License. The use of DeepSeek-VL Base/Chat models is subject to DeepSeek Model License. DeepSeek-VL series (including Base and Chat) supports commercial use.

Citation

@misc{lu2024deepseekvl,
      title={DeepSeek-VL: Towards Real-World Vision-Language Understanding}, 
      author={Haoyu Lu and Wen Liu and Bo Zhang and Bingxuan Wang and Kai Dong and Bo Liu and Jingxiang Sun and Tongzheng Ren and Zhuoshu Li and Yaofeng Sun and Chengqi Deng and Hanwei Xu and Zhenda Xie and Chong Ruan},
      year={2024},
      eprint={2403.05525},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.