- Robodex
- AI crawler
AI crawler bots
Crawls and scrapes the web to train LLMs. Does not answer user questions and has no user interface.
Signed
Verified
Unknown
Recently added
Amazon
Amazon Q Web Crawler fetches and indexes web content for Amazon Q Business applications.
UnknownAI crawler
Amazon
Amazonbot is Amazon's crawler used to improve Alexa shopping search and other Amazon services by accessing public web content.
VerifiedAI crawler
Common Crawl
CCBot is Common Crawl's crawler used to collect public web crawl data for open datasets used in research and search applications.
VerifiedAI crawler
Anthropic
ClaudeBot is Anthropic's crawler used to access public web content for Claude and related AI systems according to Anthropic's crawler policy.
VerifiedAI crawler
Google
Google-CloudVertexBot is associated with Google Cloud Vertex AI fetching or processing web content for AI workflows.
VerifiedAI crawler
OpenAI
GPTBot is OpenAI's web crawler used to access public web content that may improve OpenAI models and AI products.
VerifiedAI crawler
Klaviyo
KlaviyoAIBot is Klaviyo's web crawler designed to fetch and index publicly available content from domains you have connected to your account. This process enhances AI experiences on your website and within Klaviyo by generating tailored content and recommendations. You can control its crawling behavior through robots.txt and other settings, ensuring it respects your site's rules and only accesses desired content.
SignedAI crawler
Perplexity
PerplexityBot is Perplexity's crawler used to retrieve and index web content for Perplexity's AI answer and search products.
VerifiedAI crawler
Huawei
PetalBot is Huawei's crawler for Petal Search used to access and index web pages for search results and content recommendations.
VerifiedAI crawler
AWS
Web Crawler integration in Amazon Quick, that can create knowledge bases from website content by crawling and indexing web pages
UnknownAI crawler
Parallel
ShapBot is Parallel's crawler used to access and process web content for Parallel's AI search and data products.
VerifiedAI crawler