- Robodex
- AI crawler
AI crawler bots
Crawls and scrapes the web to train LLMs. Does not answer user questions and has no user interface.
AI crawler
Signed
Verified
Unknown
Amazon
Amazon Q Web Crawler fetches and indexes web content for Amazon Q Business applications.
UnknownAI crawler
Amazon
Amazonbot is Amazon's crawler used to improve Alexa shopping search and other Amazon services by accessing public web content.
VerifiedAI crawler
Common Crawl
CCBot is Common Crawl's crawler used to collect public web crawl data for open datasets used in research and search applications.
VerifiedAI crawler
Anthropic
ClaudeBot is Anthropic's crawler used to access public web content for Claude and related AI systems according to Anthropic's crawler policy.
VerifiedAI crawler
Google
Google-CloudVertexBot is associated with Google Cloud Vertex AI fetching or processing web content for AI workflows.
VerifiedAI crawler
OpenAI
GPTBot is OpenAI's web crawler used to access public web content that may improve OpenAI models and AI products.
VerifiedAI crawler
Klaviyo
KlaviyoAIBot is Klaviyo's web crawler designed to fetch and index publicly available content from domains you have connected to your account. This process enhances AI experiences on your website and within Klaviyo by generating tailored content and recommendations. You can control its crawling behavior through robots.txt and other settings, ensuring it respects your site's rules and only accesses desired content.
SignedAI crawler
Meta
Meta-ExternalAgent is Meta's web crawler for collecting content used to train foundation AI models and improve products through content indexing. Website owners can control its access through robots.txt.
VerifiedAI crawler
Perplexity
PerplexityBot is Perplexity's crawler used to retrieve and index web content for Perplexity's AI answer and search products.
VerifiedAI crawler
Huawei
PetalBot is Huawei's crawler for Petal Search used to access and index web pages for search results and content recommendations.
VerifiedAI crawler
AWS
Web Crawler integration in Amazon Quick, that can create knowledge bases from website content by crawling and indexing web pages
UnknownAI crawler
Parallel.ai
Parallel.ai operates automated web bots that crawl and retrieve web content for its AI-focused search and research platform. Its bots discover, index, and fetch publicly available webpages so AI agents can search the web, extract information, perform deep research, monitor changes, and generate up-to-date answers with source citations. Parallel’s indexing crawler, ShapBot, builds and refreshes its proprietary web index, while its retrieval systems may access pages in response to user or agent requests.
VerifiedAI crawler