목차
특정 봇/IP의 로그 제외
일단 나보다도 많이 접속하는 구글 봇과 내 접속 로그가 access.log에서 가장 많은 부분을 차지한다.
따라서 내가 주로 사용하는 IP들도 같이 로그에서 제외할 생각이다.
설정할 사이트의 nginx 서버 설정 파일에 아래의 내용을 추가한다.
제외할 봇과 IP 목록은 서버 블록 밖에 있어야 한다.
# Default HTTPS server configuration
server {
[...]
set $logging 1;
if ( $except_bot = 0 ) {
set $logging 0;
}
if ( $except_ip = 0 ) {
set $logging 0;
}
access_log /var/log/nginx/도메인.access.log combined if=$logging;
error_log /var/log/nginx/도메인.error.log;
[...]
}
[...]
# Log Exception - Bots
map $http_user_agent $except_bot {
~Googlebot 0;
default 1;
}
# Log Exception - IP
map $remote_addr $except_ip {
"1.1.1.1" 0;
"2.2.2.2" 0;
"3.3.3.3" 0;
default 1;
}
단순히 로그 파일만 생성할 때는 access_log /var/log/nginx/도메인.access.log;만 적어도 로그 파일이 생성되지만 이렇게 조건을 추가할 경우 로그 포맷을 같이 적어줘야 한다.
별다른 설정을 하지 않았다면 nginx의 기본 로그 포맷은 combined이니 똑같이 설정해준다.
위 내용을 설명하자면, 기본값이 true인 logging이라는 변수를 설정해서 제외할 봇 리스트나 제외할 IP 리스트 둘 중 하나라도 해당된다면 logging의 값을 false로 만드는 것이다.
설정을 마치고 nginx를 재시작하면 구글봇과 설정한 IP에서의 접속은 로그에 기록되지 않는다.
$ sudo service nginx restart
다만.. 악성 봇이 Googlebot을 달고 들어올 경우에도 로그에 남지 않는다.
가짜 Googlebot으로 공격을 하는 경우가 실제로 있다고 한다.
악성 봇 접속 차단
로그에서 제외할 봇은 많지 않으니 서버 설정 파일에 작성했지만, 차단할 악성 봇은 매우 많다.
아래 명령어로 차단할 악성 봇의 목록을 작성한다.
& sudo nano /etc/nginx/conf.d/fuck_bot.conf
아래 내용을 복사해서 넣어준다.
map $http_user_agent $fuck_bot {
default 0;
~*360Spider 1;
~*360Spider 1;
~*80legs 1;
~*Abonti 1;
~*Aboundex 1;
~*AcoonBot 1;
~*Acunetix 1;
~*adbeat_bot 1;
~*AddThis.com 1;
~*adidxbot 1;
~*ADmantX 1;
~*AhrefsBot 1;
~*AIBOT 1;
~*aiHitBot 1;
~*Alexibot 1;
~*Alligator 1;
~*AllSubmitter 1;
~*AngloINFO 1;
~*Antelope 1;
~*Apexoo 1;
~*asterias 1;
~*attach 1;
~*BackDoorBot 1;
~*BackStreet 1;
~*BackWeb 1;
~*Badass 1;
~*Baid 1;
~*Bandit 1;
~*Baiduspider 1;
~*BaiduSpider 1;
~*BatchFTP 1;
~*BBBike 1;
~*BeetleBot 1;
~*Bigfoot 1;
~*billigerbot 1;
~*binlar 1;
~*bitlybot 1;
~*Black.Hole 1;
~*BlackWidow 1;
~*BLEXBot 1;
~*Blow 1;
~*BlowFish 1;
~*BLP_bbot 1;
~*BoardReader 1;
~*Bolt\ 0 1;
~*BOT\ for\ JCE 1;
~*Bot\ mailto\:craftbot@yahoo\.com 1;
~*BotALot 1;
~*Buddy 1;
~*BuiltBotTough 1;
~*Bullseye 1;
~*BunnySlippers 1;
~*casper 1;
~*CazoodleBot 1;
~*CCBot 1;
~*Cegbfeieh 1;
~*checkprivacy 1;
~*CheeseBot 1;
~*CherryPicker 1;
~*ChinaClaw 1;
~*chromeframe 1;
~*Clerkbot 1;
~*Cliqzbot 1;
~*clshttp 1;
~*Cogentbot 1;
~*cognitiveseo 1;
~*Collector 1;
~*CommonCrawler 1;
~*comodo 1;
~*Copier 1;
~*CopyRightCheck 1;
~*cosmos 1;
~*CPython 1;
~*crawler4j 1;
~*Crawlera 1;
~*CRAZYWEBCRAWLER 1;
~*Crescent 1;
~*CSHttp 1;
~*Curious 1;
~*Curl 1;
~*Custo 1;
~*CWS_proxy 1;
~*Default\ Browser\ 0 1;
~*Demon 1;
~*DeuSu 1;
~*Devil 1;
~*diavol 1;
~*DigExt 1;
~*Digincore 1;
~*DIIbot 1;
~*DISCo 1;
~*discobot 1;
~*DittoSpyder 1;
~*DoCoMo 1;
~*DotBot 1;
~*Download.Demon 1;
~*Download.Devil 1;
~*Download.Wonder 1;
~*Download\ Demo 1;
~*dragonfly 1;
~*Drip 1;
~*DTS.Agent 1;
~*EasouSpider 1;
~*EasyDL 1;
~*ebingbong 1;
~*eCatch 1;
~*ecxi 1;
~*EirGrabber 1;
~*Elmer 1;
~*EmailCollector 1;
~*EmailSiphon 1;
~*EmailWolf 1;
~*EroCrawler 1;
~*Exabot 1;
~*ExaleadCloudView 1;
~*ExpertSearch 1;
~*ExpertSearchSpider 1;
~*Express 1;
~*Express\ WebPictures 1;
~*extract 1;
~*Extractor 1;
~*ExtractorPro 1;
~*EyeNetIE 1;
~*Ezooms 1;
~*F2S 1;
~*FastSeek 1;
~*feedfinder 1;
~*FeedlyBot 1;
~*FHscan 1;
~*finbot 1;
~*Flamingo_SearchEngine 1;
~*FlappyBot 1;
~*FlashGet 1;
~*flicky 1;
~*Flipboard 1;
~*FlipboardProxy 1;
~*flunky 1;
~*Foobot 1;
~*FrontPage 1;
~*g00g1e 1;
~*GalaxyBot 1;
~*genieo 1;
~*Genieo 1;
~*GetRight 1;
~*GetWeb\! 1;
~*GigablastOpenSource 1;
~*Go\-Ahead\-Got\-It 1;
~*Go\!Zilla 1;
~*gotit 1;
~*GozaikBot 1;
~*grab 1;
~*Grabber 1;
~*GrabNet 1;
~*Grafula 1;
~*GrapeshotCrawler 1;
~*GT\:\:WWW 1;
~*GTB5 1;
~*Guzzle 1;
~*harvest 1;
~*Harvest 1;
~*HEADMasterSEO 1;
~*heritrix 1;
~*hloader 1;
~*HMView 1;
~*HomePageBot 1;
~*htmlparser 1;
~*HTTP\:\:Lite 1;
~*httrack 1;
~*HTTrack 1;
~*HubSpot 1;
~*humanlinks 1;
~*ia_archiver 1;
~*icarus6 1;
~*id\-search 1;
~*IDBot 1;
~*IlseBot 1;
~*Image.Stripper 1;
~*Image.Sucker 1;
~*Image\ Stripper 1;
~*Image\ Sucker 1;
~*imagefetch 1;
~*Indigonet 1;
~*Indy\ Library 1;
~*InfoNaviRobot 1;
~*InfoTekies 1;
~*integromedb 1;
~*Intelliseek 1;
~*InterGET 1;
~*Internet\ Ninja 1;
~*InternetSeer\.com 1;
~*Iria 1;
~*IRLbot 1;
~*ISC\ Systems\ iRc\ Search\ 2\.1 1;
~*jakarta 1;
~*Jakarta 1;
~*Java 1;
~*JennyBot 1;
~*JetCar 1;
~*JikeSpider 1;
~*JobdiggerSpider 1;
~*JOC 1;
~*JOC\ Web\ Spider 1;
~*Jooblebot 1;
~*JustView 1;
~*Jyxobot 1;
~*kanagawa 1;
~*Kenjin.Spider 1;
~*Keyword.Density 1;
~*KINGSpider 1;
~*kmccrew 1;
~*larbin 1;
~*LeechFTP 1;
~*LeechGet 1;
~*LexiBot 1;
~*lftp 1;
~*libWeb 1;
~*libwww 1;
~*libwww-perl 1;
~*likse 1;
~*Lingewoud 1;
~*LinkChecker 1;
~*linkdexbot 1;
~*LinkextractorPro 1;
~*LinkScan 1;
~*LinksCrawler 1;
~*LinksManager\.com_bot 1;
~*linkwalker 1;
~*LinkWalker 1;
~*LinqiaRSSBot 1;
~*LivelapBot 1;
~*LNSpiderguy 1;
~*ltx71 1;
~*LubbersBot 1;
~*lwp\-trivial 1;
~*Mag-Net 1;
~*Magnet 1;
~*Mail.RU_Bot 1;
~*majestic12 1;
~*MarkWatch 1;
~*Mass.Downloader 1;
~*Mass\ Downloader 1;
~*masscan 1;
~*Mata.Hari 1;
~*maverick 1;
~*Maxthon$ 1;
~*Mediatoolkitbot 1;
~*megaindex 1;
~*MegaIndex 1;
~*Memo 1;
~*MetaURI 1;
~*MFC_Tear_Sample 1;
~*Microsoft\ URL\ Control 1;
~*microsoft\.url 1;
~*MIDown\ tool 1;
~*MIIxpc 1;
~*miner 1;
~*Missigua\ Locator 1;
~*Mister\ PiX 1;
~*MJ12bot 1;
~*Mozilla.*Indy 1;
~*Mozilla.*NEWT 1;
~*MSFrontPage 1;
~*MSIECrawler 1;
~*msnbot 1;
~*NAMEPROTECT 1;
~*Navroad 1;
~*NearSite 1;
~*Net\ Vampire 1;
~*NetAnts 1;
~*Netcraft 1;
~*netEstate 1;
~*NetMechanic 1;
~*NetSpider 1;
~*NetZIP 1;
~*NextGenSearchBot 1;
~*NICErsPRO 1;
~*niki\-bot 1;
~*NimbleCrawler 1;
~*Nimbostratus\-Bot 1;
~*Ninja 1;
~*nmap 1;
~*Nmap 1;
~*NPbot 1;
~*nutch 1;
~*Octopus 1;
~*Offline\.Explorer 1;
~*Offline\.Navigator 1;
~*Offline\ Explorer 1;
~*Offline\ Navigator 1;
~*Openfind 1;
~*OpenindexSpider 1;
~*OpenLinkProfiler 1;
~*OpenWebSpider 1;
~*OrangeBot 1;
~*OutfoxBot 1;
~*Owlin 1;
~*PageGrabber 1;
~*PagesInventory 1;
~*panopta 1;
~*panscient\.com 1;
~*Papa\ Foto 1;
~*pavuk 1;
~*pcBrowser 1;
~*PECL\:\:HTTP 1;
~*PeoplePal 1;
~*Photon 1;
~*PHPCrawl 1;
~*Pixray 1;
~*planetwork 1;
~*PleaseCrawl 1;
~*PNAMAIN\.EXE 1;
~*Pockey 1;
~*PodcastPartyBot 1;
~*prijsbest 1;
~*probethenet 1;
~*ProPowerBot 1;
~*ProWebWalker 1;
~*proximic 1;
~*psbot 1;
~*Pump 1;
~*purebot 1;
~*pycurl 1;
~*python\-requests 1;
~*QueryN\.Metasearch 1;
~*QuerySeekerSpider 1;
~*R6_CommentReader 1;
~*R6_FeedFetcher 1;
~*RealDownload 1;
~*Reaper 1;
~*Recorder 1;
~*ReGet 1;
~*RepoMonkey 1;
~*Riddler 1;
~*Ripper 1;
~*Rippers\ 0 1;
~*RMA 1;
~*rogerbot 1;
~*RSSingBot 1;
~*rv\:1\.9\.1 1;
~*RyzeCrawler 1;
~*SafeSearch 1;
~*SBIder 1;
~*scanbot 1;
~*Scrapy 1;
~*Screaming 1;
~*SeaMonkey$ 1;
~*search_robot 1;
~*SearchmetricsBot 1;
~*Semrush 1;
~*SemrushBot 1;
~*semrush\.com 1;
~*SemrushBot-BA 1;
~*SentiBot 1;
~*SEOkicks 1;
~*SEOkicks\-Robot 1;
~*seoscanners 1;
~*SeznamBot 1;
~*ShowyouBot 1;
~*SightupBot 1;
~*Siphon 1;
~*SISTRIX 1;
~*sitecheck\.internetseer\.com 1;
~*siteexplorer\.info 1;
~*Siteimprove 1;
~*SiteSnagger 1;
~*SiteSucker 1;
~*skygrid 1;
~*Slackbot 1;
~*Slurp 1;
~*SlySearch 1;
~*SmartDownload 1;
~*Snake 1;
~*Snapbot 1;
~*Snoopy 1;
~*sogou 1;
~*Sogou 1;
~*Sosospider 1;
~*SpaceBison 1;
~*SpankBot 1;
~*spanner 1;
~*spaumbot 1;
~*spbot 1;
~*Spinn4r 1;
~*Sqworm 1;
~*Steeler 1;
~*Stripper 1;
~*sucker 1;
~*Sucker 1;
~*SuperBot 1;
~*Superfeedr 1;
~*SuperHTTP 1;
~*SurdotlyBot 1;
~*Surfbot 1;
~*suzuran 1;
~*Szukacz 1;
~*tAkeOut 1;
~*Teleport 1;
~*Teleport\ Pro 1;
~*Telesoft 1;
~*The\.Intraformant 1;
~*TheNomad 1;
~*TightTwatBot 1;
~*TinEye 1;
~*TinEye\-bot 1;
~*Titan 1;
~*Toata\ dragostea\ mea\ pentru\ diavola 1;
~*Toplistbot 1;
~*trendictionbot 1;
~*trovitBot 1;
~*True_Robot 1;
~*turingos 1;
~*turnit 1;
~*TurnitinBot 1;
~*Twitterbot 1;
~*URI\:\:Fetch 1;
~*urllib 1;
~*URLy\.Warning 1;
~*Vacuum 1;
~*Vagabondo 1;
~*VCI 1;
~*VidibleScraper 1;
~*vikspider 1;
~*VoidEYE 1;
~*VoilaBot 1;
~*WallpapersHD 1;
~*WBSearchBot 1;
~*Web.Image.Collector 1;
~*Web\ Image\ Collector 1;
~*Web\ Sucker 1;
~*webalta 1;
~*WebAuto 1;
~*WebBandit 1;
~*WebCollage 1;
~*WebCopier 1;
~*WebEnhancer 1;
~*WebFetch 1;
~*WebFuck 1;
~*WebGo\ IS 1;
~*WebLeacher 1;
~*WebmasterWorldForumBot 1;
~*WebPix 1;
~*WebReaper 1;
~*WebSauger 1;
~*WebShag 1;
~*Website\.eXtractor 1;
~*Website\ eXtractor 1;
~*Website\ Quester 1;
~*Webster 1;
~*WebStripper 1;
~*WebSucker 1;
~*WebWhacker 1;
~*WebZIP 1;
~*Wells\ Search\ II 1;
~*WEP\ Search 1;
~*WeSEE 1;
~*Wget 1;
~*Whack 1;
~*Whacker 1;
~*Widow 1;
~*WinHTTrack 1;
~*WinInet 1;
~*WISENutbot 1;
~*woobot 1;
~*woopingbot 1;
~*worldwebheritage.org 1;
~*Wotbox 1;
~*WPScan 1;
~*WWW\-Collector\-E 1;
~*WWW\-Mechanize 1;
~*WWWOFFLE 1;
~*Xaldon 1;
~*Xaldon\ WebSpider 1;
~*Xenu 1;
~*XoviBot 1;
~*yacybot 1;
~*Yandex 1;
~*YandexBot 1;
~*YisouSpider 1;
~*Zade 1;
~*zermelo 1;
~*Zeus 1;
~*zh\-CN 1;
~*ZmEu 1;
~*ZumBot 1;
~*Zyborg 1;
~*ZyBorg 1;
}
설정 파일의 서버 블록에 아래의 내용을 추가해준다.
if ($fuck_bot) {
return 403;
}
아래 명령어로 nginx 서버를 다시 시작한다.
$ sudo service nginx restart
아래 사이트에서 robots.txt 파일을 테스트 할 수 있는데, robots.txt에 추가하지 않아도 위의 목록에 있는 봇은 403 페이지로 이동하는 것 같다.
robots.txt Validator and Testing Tool | TechnicalSEO.com
Test and validate your robots.txt. Check if a URL is blocked and how. You can also check if the resources for the page are disallowed.
technicalseo.com
