Robodex
GitHub stars28.2KDocsSubmit your botStart building
  1. Robodex
  2. AI crawler

AI crawler bots

Crawls and scrapes the web to train LLMs. Does not answer user questions and has no user interface.

AI crawler
Verified

Amazonbot
Amazon
Amazonbot is Amazon's crawler used to improve Alexa shopping search and other Amazon services by accessing public web content.
Verified
AI crawler
CCBot
Common Crawl
CCBot is Common Crawl's crawler used to collect public web crawl data for open datasets used in research and search applications.
Verified
AI crawler
ClaudeBot
Anthropic
ClaudeBot is Anthropic's crawler used to access public web content for Claude and related AI systems according to Anthropic's crawler policy.
Verified
AI crawler
Google-CloudVertexBot
Google
Google-CloudVertexBot is associated with Google Cloud Vertex AI fetching or processing web content for AI workflows.
Verified
AI crawler
GPTBot
OpenAI
GPTBot is OpenAI's web crawler used to access public web content that may improve OpenAI models and AI products.
Verified
AI crawler
PerplexityBot
Perplexity
PerplexityBot is Perplexity's crawler used to retrieve and index web content for Perplexity's AI answer and search products.
Verified
AI crawler
PetalBot
Huawei
PetalBot is Huawei's crawler for Petal Search used to access and index web pages for search results and content recommendations.
Verified
AI crawler
ShapBot
Parallel
ShapBot is Parallel's crawler used to access and process web content for Parallel's AI search and data products.
Verified
AI crawler