myaiteam2/website-crawler

myaiteam2 / website-crawler

Crawl a website and get all the URLs

Public
280 runs

Run myaiteam2/website-crawler with an API

Use one of our client libraries to get started quickly. Clicking on a library will take you to the Playground tab where you can tweak different inputs, see the results, and copy the corresponding code to use in your own project.

Input schema

The fields you can use to run this model with an API. If you don't give a value for a field its default value will be used.

Field	Type	Default value	Description
url	string		URL of the website to scrape
max_pages	integer	50	Maximum number of page URLs to return

{
  "type": "object",
  "title": "Input",
  "required": [
    "url"
  ],
  "properties": {
    "url": {
      "type": "string",
      "title": "Url",
      "x-order": 0,
      "description": "URL of the website to scrape"
    },
    "max_pages": {
      "type": "integer",
      "title": "Max Pages",
      "default": 50,
      "x-order": 1,
      "description": "Maximum number of page URLs to return"
    }
  }
}

Output schema

The shape of the response you’ll get when you run this model with an API.

Schema

{
  "type": "object",
  "title": "Output"
}