Поисковые боты. User agents.

web-crawler

Список поисковых ботов и соответствующих им HTTP User Agent`s.

1. MSNBot msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
2. Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
3. YandexBot Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
4. Baiduspider Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
5. Mail.Ru bot Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
6. Yahoo! Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
7. StackRambler StackRambler/2.0 (MSIE incompatible)
8. Bingbot Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
9. okhttp/2.5.0
10. Majestic MJ12bot Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php)