Robodex
GitHub stars28.2KDocsSubmit your botStart building
  1. Robodex
  2. Search crawler

Search crawler bots

Crawls and scrapes the web to build a search-engine index.

Verified
Spoofed
Unknown

Amzn-SearchBot
Amazon
Amzn-SearchBot is an Amazon search crawler used to discover and index public web content for Amazon search-related services.
Verified
Search crawler
Applebot
Apple
Applebot is Apple's web crawler used by Siri Spotlight Safari Apple Intelligence and other Apple services to index and retrieve web content.
Verified
Search crawler
Baiduspider
Baidu
Baiduspider is Baidu's search crawler used to discover and index web pages for Baidu Search and related search services.
Verified
Search crawler
Bingbot
Microsoft
Bingbot is Microsoft's search crawler used to discover crawl and index pages for Bing Search and related Microsoft search experiences.
Verified
Search crawler
Bytespider
ByteDance
Bytespider is ByteDance's web crawler used to access and index public web content for ByteDance search and content products.
Spoofed
Search crawler
DuckDuckBot
DuckDuckGo
DuckDuckBot is DuckDuckGo's search crawler used to improve DuckDuckGo search results and web indexing.
Verified
Search crawler
Googlebot
Google
Googlebot is Google's primary search crawler used to discover crawl render and index pages for Google Search.
Verified
Search crawler
Googlebot Image
Google
Googlebot Image crawls and indexes images and image-related resources for Google Images and Google Search features.
Verified
Search crawler
Googlebot Video
Google
Googlebot Video crawls and indexes video pages and video-related resources for Google Search and video search experiences.
Verified
Search crawler
SeekportBot
Seekport
SeekportBot is Seekport's search crawler used to discover and index public web pages for Seekport's search services.
Verified
Search crawler
YandexBot
Yandex
YandexBot is Yandex's crawler used to access and index web pages for Yandex Search and related services.
Verified
Search crawler
Yeti
Naver
Yeti is Naver's web crawler used to discover and index pages for Naver Search and related search services.
Verified
Search crawler
YisouSpider
Alibaba
YisouSpider is Alibaba/UC's search crawler used to discover and index web pages for Shenma/Yisou mobile search services.
Unknown
Search crawler