- Robodex
- Search crawler
Search crawler bots
Crawls and scrapes the web to build a search-engine index.
Search crawler
Verified
Unknown
Amazon
Amzn-SearchBot is an Amazon search crawler used to discover and index public web content for Amazon search-related services.
VerifiedSearch crawler
Apple
Applebot is Apple's web crawler used by Siri Spotlight Safari Apple Intelligence and other Apple services to index and retrieve web content.
VerifiedSearch crawler
Baidu
Baiduspider is Baidu's search crawler used to discover and index web pages for Baidu Search and related search services.
VerifiedSearch crawler
Microsoft
Bingbot is Microsoft's search crawler used to discover crawl and index pages for Bing Search and related Microsoft search experiences.
VerifiedSearch crawler
ByteDance
Bytespider is ByteDance's web crawler used to access and index public web content for ByteDance search and content products.
VerifiedSearch crawler
Coccoc
CocCocBot is the web crawler for Cốc Cốc, a Vietnamese search engine and browser. It discovers and indexes web pages for inclusion in Cốc Cốc search results.
VerifiedSearch crawler
DuckDuckGo
DuckDuckBot is DuckDuckGo's search crawler used to improve DuckDuckGo search results and web indexing.
VerifiedSearch crawler
Dassault Systemes
Exabot is the web crawler associated with Exalead, a search technology company owned by Dassault Systèmes. It fetches web pages to build search indexes.
VerifiedSearch crawler
Google
Googlebot is Google's primary search crawler used to discover crawl render and index pages for Google Search.
VerifiedSearch crawler
Google
Googlebot Image crawls and indexes images and image-related resources for Google Images and Google Search features.
VerifiedSearch crawler
Google
Googlebot Video crawls and indexes video pages and video-related resources for Google Search and video search experiences.
VerifiedSearch crawler
Mail.ru
MailRUBot is an automated web crawler operated by Mail.ru. It discovers and indexes web pages to support Mail.ru’s search services, helping users find website content through search results.
VerifiedSearch crawler
Seekport
SeekportBot is Seekport's search crawler used to discover and index public web pages for Seekport's search services.
VerifiedSearch crawler
Seznam
SeznamBot is the web crawler operated by Seznam, a Czech search engine. It discovers and indexes website content for inclusion in Seznam search results.
VerifiedSearch crawler
Sogou
Sogou Spider is the web crawler operated by Sogou, a Chinese search engine. It crawls web pages to collect and index content for Sogou search results.
VerifiedSearch crawler
Yahoo
This is Yahoo's web crawler for collecting and indexing website content. It supports Yahoo Search and gathers content for services such as Yahoo News, Finance, and Sports.
VerifiedSearch crawler
Yandex
YandexBot is Yandex's crawler used to access and index web pages for Yandex Search and related services.
VerifiedSearch crawler
Naver
Yeti is Naver's web crawler used to discover and index pages for Naver Search and related search services.
VerifiedSearch crawler
Alibaba
YisouSpider is Alibaba/UC's search crawler used to discover and index web pages for Shenma/Yisou mobile search services.
UnknownSearch crawler