Unviewed/Bot Traffic Is Killing Me

Please see the attached screenshot, it only happens on Dolphin sites. 

How can I stop BOTs from accessing my Dolphin 7.1.4 site?

You will need to set up some controls to block the bots; forget trying to use robots.txt because that does not stop the bad bots.  If you are running Apache webserver you can use .htaccess to set up a whitelist, not a blacklist as some keep yelling about because your blacklist will just grow and grow and grow and grow.  A whitelist says, let these agents come in, everyone else block.  A whitelist is much easier to maintain.  The bad bots will continuously change their agent id so you are continuously adding to your blacklist.

Read IncrediBILL's Rant:

http://incredibill.blogspot.com/2006/09/whitelist-opt-in-htaccess-file.html

It is like setting up iptables at the server root, you tell iptables to block everything and then you open up the ports you need for people to access your site.

If you can use .htaccess you can use that to block them in other ways too. (Bots are a P.I.T.A.)

 http://www.drupaldriven.com/content/how-ban-ip-addresses-and-block-bots-htaccess

One other method is to run your site through Cloudflare. You can manually block countries from their admin panel. It doesn't work 100% of the time, but it does help. It's also helpful as far as security and speed go. There are some haters out there, but frankly.... I don't think they know what the hell they are talking about. I've gotten amazing results. The proof is in the pudding.

Enable the China DNS blocklist in Administration => Tools => Antispam Tools. Under Antispam Tools => Settings, make sure the DNS blocklists option is ticked and the join behavior option is set to "block".

I've found this blocks most spam activity. If you need something with more kick, Deano has a module in the market.

Edit: I should clarify this is for Dolphin. To block activity at the Web server level, look into a .htaccess-based block as mentioned above.

Read IncrediBILL's article on bad bots coming from datacentres:

http://incredibill.com/blog/why-data-centers-must-be-blocked/

so I follow Dr. Nathan Paton's advice and ENABLED the anti-spam, I think that's the reason but let's wait and see. 

Why in the world bots like Dolphin so much?

It's not just Dolphin. I host over 30 Wordpress sites and I promise you....the bots hammer the hell out of them if I allow it. Particularly the ones from China. They don't seem to pay any attention to robots.txt either.

You're likely to drive yourself crazy if you try to block them all.  Just get rid of the worst offenders and you will have much joy unless your resources are extremely limited. If that's the case you should consider getting a better hosting account.

Yea. Not just dolphin. I have seen many sites get hammered. Wordpress seems to be a good target as well as phpbb forums.

From the note, it says that the "not viewed" traffic 'includes' bots, worms, etc., - which leads me to believe that there is more that is included in "not viewed" traffic, but not listed. That is a high usage of bandwidth, and I am wondering what else could be using so much. Could it possibly be showing bandwidth from uploading/downloading videos/files or viewing video files?

Which is called Blacklisting.  You don't blacklist, you whitelist.  Listen to IncrediBILL, you lock your site out and only allow the agents you want in; you set up a list of allowed agents, you block everything else.

 Nope, no video or audio (mps). Only 1 Dolphin and 1 phpBB "integrated" together. And the stats in Cpanel clearly show it's bot traffic also. 

Bad bots ignore the robots.txt file. The best you can do is try to identify abusive access patterns, and block the corresponding IP.

User-agent: *

Disallow: /

only helps with well-behaved robots.

I use Incapsula.com which has a free subscription.

I like guys named Bill...

Here is the script he posted to place in my .htaccess.... I am aware of how to do it but want to make sure I'm not blocking my users. Can someone explain exactly how this works in laymans terms?

If someone tells me I'm a dummy and tells me to read the instructions, I'm gonna run over em with my tow truck.....

#allow just search engines we like, we're OPT-IN only

#a catch-all for Google
BrowserMatchNoCase Googlebot good_pass
BrowserMatchNoCase Mediapartners-Google good_pass

#a couple for Yahoo
BrowserMatchNoCase Slurp good_pass
BrowserMatchNoCase Yahoo-MMCrawler good_pass

#looks like all MSN starts with MSN or Sand
BrowserMatchNoCase ^msnbot good_pass
BrowserMatchNoCase SandCrawler good_pass

#don't forget ASK/Teoma
BrowserMatchNoCase Teoma good_pass
BrowserMatchNoCase Jeeves good_pass

#allow Firefox, MSIE, Opera etc., will punt Lynx, cell phones and PDAs, don't care
BrowserMatchNoCase ^Mozilla good_pass
BrowserMatchNoCase ^Opera good_pass

#Let just the good guys in, punt everyone else to the curb
#which includes blank user agents as well

<Limit GET POST PUT HEAD>
order deny,allow
deny from all
allow from env=good_pass
</Limit>

Laymens terms. I will try.

BrowserMatchNoCase tells Apache to do a case insensitive search for matches within the User Agent String header.

Most standard web browser User agent strings look something like this. This exact one is Mine for my os and browser i am currently using.

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0

Per the last rule in that .htaccess file.

#allow Firefox, MSIE, Opera etc., will punt Lynx, cell phones and PDAs, don't care
BrowserMatchNoCase ^Mozilla good_pass
BrowserMatchNoCase ^Opera good_pass

I would be allowed in because Mozilla will be found in my user agent string. And despite the fact that it says it will punt cell phones, that's not exactly 100% accurate. It depends on the user agent string the phone uses. For example my iphone 5 also has the word Mozilla in it, so my phone would work.

Bill has been through some horrible illness and that was posted a few years back but it gives you the ideal of how whitelisting works.  I agree with Bill that whitelisting is the way to go, blacklisting just builds these horribly long lists that if they grow long enough can put a hit on the response time since it has to scan the list from the top to the bottom.

it seems to be the biggest problem at Dolphin (also some website as Deano said).  You can use Deano module to handle this and set new members by invitation only, they are not a bot, but fake users (spamers) are common too. 
 

What about a join form that is dynamic and build on the fly.
I mean give the fields random numbers as ID or name and the bots can't track it anymore.
Only thing is that the join core code needs to written from scratch again.

exaple: username => 1987654 and is set in Admin => profile fields
So Admins can change this every week/day/month/never

Hmmm... maybe a new mod is born!!!