List of known Spiders / Bots

Overview

Catchpoint allows you to block Bots and Spiders from your real user measurements.

To block Bots and spiders, go to Settings >Apps > and select an app. Then check Ingore: Known Bots/Spiders.

Bots and spiders will be blocked based on their User Agents.

Known Bots and Spiders List

Below is the list of user agents that Catchpoint considers a Bot/Spider.

Scrubby
^Nail
007AC9
192.comAgent
200PleaseBot
360Spider
4seohuntBot
50.nu
A6-Indexer
abby
Aboundexbot
AboutUsBot
Abrave Spider
Accelobot
Ace Explorer
AcoonBot
ActiveBookmark
Activeworlds
AddThis.com
ADmantX
adressendeutschland.de
AdsBot-Google
Advanced Browser
agentslug
AhrefsBot
aiHitBot
Akregator
akula
Alcatel-BG3
AlertInfo
Alexabot
alyze.info
Amagit.COM
amaya
amibot
Amiga-AWeb
AmigaVoyager
AMZNKAssocBot
annotate_google
AntBot
ANTFresco
Apercite
AportWorm
Applebot
AraBot
arachnode.net
Arachnophilia
archive.org_bot
Ask Jeeves/Teoma
AskQuickly
Audiovox
Automattic Analytics Crawler
Avant
avantbrowser
BabalooSpider
backlink-check.de
BacklinkCrawler
Bad-Neighborhood
baiduspider
Barca
BarcaPro
baypup
BDCbot
BDFetch
BegunAdvertising
bingbot
bitlybot
biwec
bixocrawler
Biyubi
bl.uk_lddc_bot
Blekkobot
BLEXBot
BlinkaCrawler
BlockNote
BlogBridge
BlogPulse
bluefish 0.6 HTML editor
bnf.fr_bot
bot.wsowner.com
botmobi
BotOnParade
bot-pge.chlooe.com
BrainbruBot
BrokenLinkCheck.com
Browsershots
BUbiNG
Butterfly
CamontSpider
CareerBot
Castabot
CatchBot
CCBot
CCResearchBot
ChangeDetection
Charlotte
CirrusExplorer
cityreview
classbot
CligooRobot
Cliqzbot
CliqzBot
CloudFlare-AlwaysOnline
CloudServerMarketSpider
CMS Crawler
coccoc
Commerce Browser Center
COMODOSpider
Company News Search engine
CompSpyBot
contentDetection
Cooliris
copyright sheriff
CorpusCrawler
Covario-IDS
crawler for netopian
Crawler4j
CrazyWebCrawler-Spider
Crowsnest
Curious George
datagnionbot
Daumoa
DBLBot
DCPbot
deadlinkchecker
DealGates Bot
Dillo
discoverybot
DKIMRepBot
dlcbot
dlvr.it
Dlvr.it/1.0
DNS-Digger-Explorer
DomainAppender
DomainDB
Dot TK - spider
DotBot
dotSemantic
DreamPassport
DripfeedBot
drupact
DuckDuckBot
EasouSpider
EasyBib AutoCite
eCairn-Grabber
eCommerceBot
EdisterBot
electricmonk
Elefent
ELinks
Embedly
emefgebot
envolk
Esribot
EuripBot
Eurobot
EventGuruBot
EveryoneSocialBot
EvriNid
Exabot
ExB Language Crawler
Experibot
ExpertSearchSpider
Ezooms
FacebookExternalHit
facebookplatform
factbot
FairShare
Falconsbot
fastbot crawler
FauBot
FeedCatBot
FeedDemon
FeedFinder/bloggz.se
Feedly
Feedshow
FeedshowOnline
FeedZcollector
Fetch-Guess
findlinks
firmilybot
Flatland Industries Web Spider
FlightDeckReportsBot
FlipboardProxy
Flocke bot
FollowSite Bot
Fooooo_Web_Video_Crawl
FreeWebMonitoring SiteChecker
FyberSpider
Gaisbot
GarlikCrawler
GeliyooBot
Genieo Web filter
GetFoundBot
GigablastOpenSource
Gigabot
GingerCrawler
Girafabot
gocrawl
GOFORITBOT
gomezagent
gonzo
Googlebot
googlebot
GozaikBot
grabber
Grahambot
GrapeshotCrawler
GurujiBot
Hailoobot
HatenaScreenshot
hawkReader
HeartRails_Capture
heritrix
Holmes
HolmesBot
HomeTags
HostTracker
HostTracker.com
HTMLParse
HuaweiSymantecSpider
HubSpot Crawler
HypeStat
ia_archive
ia_archiver
ibot
ICC-Crawler
ichiro
iCjobs
IdeelaborPlagiaat
idmarch
Iframely
imbot
immediatenet thumbnails
ImplisenseBot
Impressumscrawler
imrbot
Influencebot
Infohelfer
IntegromeDB
internetseer
IstellaBot
it2media-domain-crawler
IXEbot
Jabse.com Crawler
JadynAveBot
JamesBOT
JikeSpider
Job Roboter Spider
JobdiggerSpider
JUST-CRAWLER
Jyxobot
Kalooga
Karneval-Bot
keynote
KimonoLabs
kinshoo
KomodiaBot
Kraken
KrOWLer
kulturarw
L.webis
Leikibot
LemurWebCrawler
LexxeBot
Lijit
LinguaBot
linguatools
Linguee Bot
Link Valet Online
LinkAider
linkdex.com
linkdexbot
LinkedInBot
linkpeek
LinkWalker
Lipperhey Spider
livedoor ScreenShot
LivelapBot/0.2
LoadImpact
LoadTimeBot
ltbot
LuminateBot
magpie-crawler
Mail.Ru bot
meanpathbot
MeMoNewsBot
memoryBot
MergeFlow-PageReader
MetaGeneratorCrawler
MetaHeadersBot
MetaJobBot
MetamojiCrawler
METASpider
Metaspinner/0.01
MetaURI
MetaURI API
MIA Bot
MiaDev
MixBot
MJ12bot
MLBot
MnoGoSearch
moba-crawler
MojeekBot
Motoricerca-Robots.txt-Checker
Mp3Bot
MSNBot
MSRBOT
musobot
Najdi.si
NalezenCzBot
NaverBot
nekstbot
NerdByNature.Bot
NerdyBot
NetcraftSurveyAgent
netEstate Crawler
NetResearchServer
Netseer
NextGenSearchBot
NF-Browser
Nigma.ru
NLNZ_IAHarvester
nodestackbot
Nuhk
Nutch
nworm
Nymesis
oBot
Ocelli
omgilibot
Online Domain Tools
Open Web Analytics Bot
OpenCalaisSemanticProxy
OpenindexSpider
OpenWebSpider
OptimizationCrawler
OrangeBot
OrgbyBot
OsObot
ownCloud Server Crawler
page_verifier
Page2RSS
PagePeeker
Pagespeed
Pandeo Bot
Panscient web crawler
PaperLiBot
ParchBot
parsijoo
PayPal IPN
Peeplo Screenshot Bot
Peepowbot
peerindex
Peew
PercolateCrawler
Pingdom
pingdom.com_bot
Pingoscope
Pinterest
PiplBot
Pixray-Seeker
Plukkie
pmoz.info ODP link checker
Pompos
Port Monitor check service
PostPost
pr-cy.ru Screenshot Bot
ProCogBot
ProCogSEOBot
ProductoDownloadUrlBot/
proximic
psbot
Qirina Hurdler
Qseero
Qualidator.com Bot
Qualidator.com SiteAnalyzer 1.0
QuerySeekerSpider
quickobot
Qwantify
R6 bot
RADaR-Bot
RankurBot
Readability
Riddlerbot
Robots_Tester
Robozilla
rogerbot
Ronzoobot
RSSMicro.com RSS/Atom Feed Robot
Ruky-Roboter
RyzeCrawler
SAI Crawler
SanszBot
SBIder
SBSearch
Scarlett
SCFCrawler
Scooter
ScoutJet
Scrapy
ScreenerBot Crawler
Screenshot Bot
search.KumKie.com
Search17Bot
SearchmetricsBot
SecurityResearchBot
seegnifybot
Semager
Semantifire
SemrushBot
SentiBot
Seobility
SEOCentro bot
SeoCheckBot
SEODat
SEOdiver
SEOENGBot
SEOkicks-Robot
Setoozbot
Setoozbot
Seznam screenshot-generator
SeznamBot
Shareaholicbot
Shelob
ShopWiki
ShowyouBot
sistrix
Site24x7
SiteCondor
Sitedomain-Bot
Siteliner
Slackbot
smart.apnoti.com Robot
SMTBot
Snapbot
SniffRSS
socialbm_bot
sogou spider
Sogou web spider
SolomonoBot
Sosospider
spbot
Speedy Spider
SpiderLing
Spinn3r
SputnikBot
SSLBot
SSL-Crawler
StackRambler
StatoolsBot
Steeler
STINGbot
stq_bot
Strokebot
suggybot
sukibot_heritrix
SurcentroBot
Surphace Scout
SurveyBot
SWEBot
SygolBot
Symfony Spider
Szukacz
Tagoobot
taptubot
Technoratibot
Thumbnail.CZ robot
Thumbshots.ru
ThumbShots-Bot
thumbshots-de-Bot
ThumbSniper
TinEye
TomTom places company search
Topicbot
Toread-Crawler
Touche
trendictionbot
trendictionbot
TurnitinBot
TwengaBot
Twiceler
Twikle
Twingly Recon
UASlinkChecker
uMBot
UnisterBot
UnwindFetchor
Updownerbot
UptimeDog
UptimeRobot
URLAppendBot
urlfan-bot
Urlfilebot (Urlbot)
Vagabondo
Vedma
VideoSurf_bot
Visbot
VoilaBot
voltron
voyager
VR FrontPage Fetcher
WASALive-Bot
WatchMouse
WBSearchBot
web downloader
web+downloader
webauto
webcapture
webcheck
WebCookies
WebCopier
WebCorp
WebImages
webinatorbot
webmastercoffee
Webmetrics
Web-Monitoring
WebNL
WebRankSpider
websitepulse
Websnapr
Web-sniffer
WebTarantula.com Crawler
WebThumb
WebThumbnail
WebWatch/Robot_txtChecker
WeSEE
WeViKaBot
Whoismindbot
WikioFeedBot
wikiwix-bot
Willow Internet Crawler
WillyBot
WinWebBot
WMCAI_robot
Woko
WordPress.com mShots
woriobot
WorldBrewBot
Wotbox
wsAnalyzer
wscheck.com
x28-job-bot
XmarksFetch
XML Sitemaps Generator
XoviBot
XRL
Yaanb
yacybot
Yahoo!
YamanaLab-bot
YandexBot
YandexNews
Yanga
YioopBot
YodaoBot
YoudaoBot
YowedoBot
YRSpider
YYSpider
zealbot
ZeerchBot
ZipCommander
Zookabot
ZookaBot
ZumBot