Форум сетевых администраторов

У кого нибудь есть база IP-адресов поисковых роботов?
Надо.

Кстати, вот один из способов собирать статистику по IP и User-Agent:

Изменения в .htaccess

code:

RewriteEngine On 
RewriteRule robots.txt robots.php

User-agent: * 


<? 
$tdate = date("dS F Y h:i:s A"); 
$fp = fopen("log.txt", "a"); // сюда пишем логи чмод 777 
$line = "IP Address: $REMOTE_ADDR - Date: $tdate - Browser: $HTTP_USER_AGENT\n"; 
fwrite($fp, $line); 
fclose($fp); 
?>

IP Address: 68.157.245.140 - Date: 25th August 2005 10:42:31 AM - Browser: NetResearchServer/3.5(loopimprovements.com/robot.html) 
IP Address: 213.180.128.151 - Date: 25th August 2005 10:48:40 AM - Browser: Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl)

Я не думаю что это разумно орьентируясь по IP. IP зачастую меняются. Лучше всего использовать user-agent, а список всех поисковиков несложно найти в нете, да и любой парсер логов распологает им, вот например файл который юзает веблог експерт:

code:

arachnophilia = Arachnophilia
architext = ArchitextSpider
aspider = Associative Spider
backrub = BackRub
checkbot = Checkbot
Firefly = Fireball.de Robot
freecrawl = Freecrawl
gazz/2.1 (gazz@nttrd.com) = Nttrd.com Robot
geturl = GetURL
googlebot = Googlebot
gulliver = Northern Light Gulliver
infoseek = InfoSeek Robot
infoseeksidewinder = Infoseek Sidewinder
internetseer = InternetSeer
lycos_ = Lycos Spider
scooter = Scooter
searchhippo = SearchHippo Robot
Yahoo! Slurp = Yahoo! Slurp
slurp = Inktomi Slurp
fast-webcrawler = Fast-Webcrawler
mercator = Mercator
MSIECrawler = MSIE Crawler
ia_archiver = Alexa Robot
ultraseek = Ultraseek
Xenu Link Sleuth = Xenu's Link Sleuth
Xenu's Link Sleuth = Xenu's Link Sleuth
File Flash Spider = File Flash Spider
TrueRobot/1.4 = Echo.com Robot
SlySearch = Slysearch Robot
Openfind = Openfind Robot
LWP::Simple = Perl LWP
lwp-trivial = Search4free Robot
libwww-perl/5.41 = CMP United Media Robot 
libwww-perl/5.48 = Alexa Robot
libwww-perl/5.52 FP = Fast Search Robot
NPBot = NameProtect Robot
Gigabot = Gigablast Robot
Baiduspider = Baiduspider
TurnitinBot = Turnitin Robot
Overture-WebCrawler = Overture Crawler
Gaisbot = Gaislab Robot 
msnbot = MSN Robot 
psbot = Picsearch Robot
appie = Walhello Robot
Ask Jeeves = AskJeeves Robot
linkwalker = LinkWalker
moget = Goo Japan Robot
Netcraft = Netcraft Spider
VoilaBot = Voila Robot
ZyBorg = WiseNut Robot
AlkalineBOT = Vestris Robot
SurveyBot = SurveyBot
yandex = Yandex Robot
StackRambler = Rambler Robot
aport = Aport Robot
grub-client = Grub Crawler
Morning Paper = Morning Paper
search.ch = Search.ch Robot
Vagabondo = WiseGuys Robot
almaden = IBM Almaden Crawler
TutorGig = TutorGig Crawler
QuepasaCreep = Quepasa Crawler
exactseek = ExactSeek Robot
Zealbot = Zealbot
Speedy Spider = Entireweb Robot
NaverBot = Naver Robot
IlTrovatore = IlTrovatore Robot
Girafabot = Girafabot
Powermarks = Powermarks
WebMon 1. = WebMon
Jyxobot = Jyxobot
holmes/ = Holmes Crawler