Block Spy Bots

Protect Private Blog Network (PBN) From Rev Engineering

Block All The above spy Bots. Spyder Spanker is a good Option. But Excess use of Spyder Spanker can be a foot print.

  • Moz (rogerbot)
  • Majestic SEO (MJ12bot)
  • SEO Spyglass
  • Ahrefs - AhrefsBot
  • Ahrefs - AhrefsBot
  • Open Site Explorer (Moz)
  • Majestic SEO
  • GgigaBlast (gigablast.com)

.HTACCESS CODE

Example

RewriteEngine On
RewriteBase /

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

ROBOTS.TXT

Example

robots.txt file code ( in robots.txt which should block everything except Google, Yahoo and Bing.)

Like:

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: rogerbot
Disallow: /

User-agent: exabot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: *
Disallow: /