Facebook BOT?

I got site A and site B, both are using beckmedia POS Facebook module, (Only reasons for using? One button importing of all my Facebook Events and both sites are for looking up local events..)

I have no problems with it, the reason for the post...

Site A, facebook bot spends all day on my site for a week or two, 80-90 unique hits a day all from different IP's but all say FB Bot and not a single page that it is actually looking at, from Oct 7th running till today..

Site B never gets visited by this bot...

The bot activity only starts after importing about 100 events to my site.

I'm sure nothing to worry about, I just am curious if other members are getting slammed by this Facebook Bot?

Let's get one thing straight.  You have a Beck Media FB module that actually runs under 7.0.9?? 

 it's running on mynewbeetle.net 7.1... and all those events on the DEMO site were pulled from FB with his module. [edit] I changed tons of settings, but got it working

HL, I have almost fixed all the problems with that shit piece module.

Like you stated, all things happen in the class files, that is where I started..

Oh and yes, I have it working perfect on three other popular sites, imports, photos, and the newfeeds wall all work perfect..

I know that fu*^3&@ module like the back of my hand.. I also doctored his "Locations with Mobile" to work as well for 7.0.9 and 7.1

HL, use mobile device.. for testing Dolphin 7.1 mobile side with his module

mynewbeetle.net/7.1/

user: demo

pass :demo1

The locations module is there and somewhat working.. let me know what you find

So no one is being hammered via FB bots..

so far today, started at 9AM and they are still doing massive page crawlings..

I don't get it!

 Also fixed the games module to work on 7.1 so that is three so far... some errors but nothing I can't handle now..

Oh, and the "Faceslook" template we have working with 7.0.9, the one so many hated, not the Biggi, but the one that exactly matches FB.

I do have to say those modules are written like they were HI on something..

maybe fb bot loves your site :P

well i don't know about fb bot algorithm but probably your site A got lots of links in loop or something.

I guess, the thing is in a few more days, it will stop. Completely..until the events are imported.

This is also usually when someone shares or likes something from this one site.

No effects on performance..

It's just that the site has been operating this way for two years, just this past month is when I noticed ALL day long Facebook bots, doing 15 views from many many different IP so far today now over 200!

On a slightly similar note

my site was getting hit every minute of the day from IP 180.76.5.54  (Baiduspider)

In September I bloocked the IP Range 180.76.0.0 - 180.76.255.255

Prior to this I was geeting 100-200 join atempts daily and many hits from various IP in China

Once I blocked the above IP range I get zero (0) hits and 0 jion atempts from China, also no spam on the site since then.

I don't get the facebook bot... but I'm getting really pissed at the google bot:

Oct 19 22:32:11 myserver.com proftpd[27647] myserver.com (crawl-66-249-71-21.googlebot.com[66.249.71.21]): FTP session opened.

Oct 19 22:32:11 myserver.com proftpd[27647] myserver.com (crawl-66-249-71-21.googlebot.com[66.249.71.21]): USER anonymous: no such user found from crawl-66-249-71-21.googlebot.com [66.249.71.21] to 99.11.11.11

Oct 19 22:32:11 myserver.com proftpd[27647] myserver.com (crawl-66-249-71-21.googlebot.com[66.249.71.21]): FTP session closed.

This is becoming an issue now.. all 10 of my sites are being attacked.

No one else is having this issue?

https://developers.facebook.com/bugs/482090818482316?browse=search_505341c73679b1b27524924

start killing them by filtering out the useragent in your .htaccess

I haven't noticed anything like this yet...

 Thanks, I'll look at that.

Can I use something like this? But I'm not sure on how to block the Facebook BOT.. 

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>

We are interested in the part of the file with the  <Limit GET POST>.This is the area that we are going to modify. First of all, the line:order deny,allowneeds to be changed to: order allow,deny. Next, we need to change the linedeny from all to Deny from env=bad_bot. 

Then above this section add the list of bad bots. So when its all done, this modified section should look like this:

SetEnvIfNoCase User-Agent "^Black Hole" bad_bot
SetEnvIfNoCase User-Agent "^Titan" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot 
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot 
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^TeleportPro" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^Website Quester" bad_bot
SetEnvIfNoCase User-Agent "^WebZip" bad_bot
SetEnvIfNoCase User-Agent "^moget/2.1" bad_bot
SetEnvIfNoCase User-Agent "^WebZip/4.0" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^httplib" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^Harvest/1.5" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye/1.0" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4.0 (compatible; BullsEye; Windows 95)" bad_bot
SetEnvIfNoCase User-Agent "^Crescent Internet ToolPak HTTP OLE Control v.1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerSE/1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker /1.0" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit/3.50" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control - 5.01.4511" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial/1.34" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control - 6.00.8169" bad_bot
SetEnvIfNoCase User-Agent "^URLy Warning" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Mata Hari" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^Web Image Collector" bad_bot
SetEnvIfNoCase User-Agent "^The Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish/1.0" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc/4.2" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^VCI WebViewer VCI WebViewer Win32" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^QueryN Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^Openfind data gathere" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^Xenu's Link Sleuth 1.1c" bad_bot
SetEnvIfNoCase User-Agent "^Xenu's" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey Bait & Tackle/v1.01" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 32297 Webster Pro V2.9 Win32" bad_bot
SetEnvIfNoCase User-Agent "^Webster Pro" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a Unix" bad_bot
SetEnvIfNoCase User-Agent "^Keyword Density/0.9" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin Spider" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot

<Limit GET POST>
order allow,deny
allow from all
Deny from env=bad_bot
</Limit>

I guess not..

so far today, 228 bot hits averaging about 25 hits per visit..

at least it's not taking the site down like others have complained about.

No one can help? 

This is still going on every day.. 

Try adding this immediately after the opening php tag in inc/header.inc.php

$ua = $_SERVER['HTTP_USER_AGENT'];

if (preg_match('/facebookexternalhit/si',$ua)) {
header('Location: GoAwayFB.php');
die() ;
}

Create the file GoAwayFB.php in your sites root.

Then keep an eye on the accesses to this file.

If you find you have a ton of FB bot accesses to this file, then you can do something like this:

$ua = $_SERVER['HTTP_USER_AGENT'];

if (preg_match('/facebookexternalhit/si',$ua)) {
header('Location: http://facebook.com/index.php');
die() ;
}

That ought to drive the FB bot nuts.

Thanks HL for that.. 

I have learned why now, this is only on my sites using the _beckmedia Facebook module, if I have imported events, which I do every other day, I noticed the bot is active for couple days.

I'm thinking now I shouldn't block them, as the server is not affected by this.

If it becomes a resource issue I will implement your solution above and see how it works.

 OK,

I implemented your second suggestion without even trying the first one.

This morning at around 5:30AM they hit the site 213 hits, 20 visits on each "hit" and my server SQL database for the one site was knocked out for an hour after that.

How do you know this works? Is that written by you?

I don't fully understand php so I hope is does something to the damn thing, there has not been a single day FB has not crawled my site ALL day, this is getting out of hand..

I guess I will know it's working when my stats are no longer filled with FB bot hits..

:Fingers Crossed: 

I am guessing HL got the code from here. http://stackoverflow.com/questions/8195663/block-facebook-from-my-website or a similar page.

Take a look at this page as well. I am guessing there is a reason facebook is hitting your site. http://serverfault.com/questions/383120/why-are-facebooks-servers-hitting-my-site-nearly-once-per-minute-and-the-time

Chances are tho if you want to block it so it does not even show up in your logs, then the IP addresses will need to be blocked at a firewall before it can get to your site.

Thanks Deano,

OMG there are hundreds of IP's

[edit] I guess I could start with 69.171.*.*

This is taken from one of those postings..

Perhaps something on your site is being shared around on Facebook.

Facebook makes requests to get little image and text previews before users post a link to their wall.

This is true, I have a lot of FB events on this one site and each event is tweeted and "liked and shared" back to facebook.

So maybe that is just it, but it is taking the site down sometimes.

Now even more confused on blocking them or not.

Newt, that script detects specific user agents and when those detected user agents land on any page of your site, they are redirected to that one file.  Redirecting them to an external file just gets them off your site immediately.  Not sure exactly which site I saw the code on.... it's all over the place. 

If you have a lot of stuff being shared on FB, this is probably all perfectly normal.  Maybe you need a mo bigger server.

Not a single hit since yesterday morning.

Thanks Houston..

Blacklists just become albatrosses, whitelist is the way to go.

It might be. 

But this stopped FB dead.. not a single out of 60 days of constant crawling..

Now I guess this would work for ANY website I don't want on my server..

 RE:

 If you know the user agent it will.  You can also add this to a robots.txt file in your sites root:

User-Agent: *
# Directories
Disallow: /administration/
Disallow: /backup/
Disallow: /cache/
Disallow: /cache_public/
Disallow: /flash/
Disallow: /inc/
Disallow: /langs/
Disallow: /periodic/
Disallow: /plugins/
Disallow: /simg/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /m/google_search/
# Files
Disallow: /forgot.php
Disallow: /join.php
Disallow: /logout.php
Disallow: /get_rss_feed.php
Disallow: /searchKeyword.php
Disallow: /searchKeywordContent.php
Disallow: /search_home.php
Disallow: /search.php
Disallow: /crossdomain.xml


# Block these Robots:

User-agent: ia_archiver
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: grub
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: larbin
Disallow: /

User-agent: b2w/0.1
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: psbot
Disallow: /

User-agent: Python-urllib
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: Mozilla
Disallow: /

User-agent: mozilla
Disallow: /

User-agent: mozilla/3
Disallow: /

User-agent: mozilla/4
Disallow: /

User-agent: mozilla/5
Disallow: /

User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow: /

User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow: /

User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow: /

User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow: /

User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: moget/2.1
Disallow: /

User-agent: WebZip/4.0
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: RMA
Disallow: /

User-agent: libWeb/clsHTTP
Disallow: /

User-agent: asterias
Disallow: /

User-agent: httplib
Disallow: /

User-agent: turingos
Disallow: /

User-agent: spanner
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: Wget/1.6
Disallow: /

User-agent: Wget/1.5.3
Disallow: /

User-agent: Wget
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: moget
Disallow: /

User-agent: hloader
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Mata Hari
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: toCrawl/UrlDispatcher
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: suzuran
Disallow: /

User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /

User-agent: VCI
Disallow: /

User-agent: Szukacz/1.4
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: Openfind data gathere
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: Microsoft URL Control
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: URL Control
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: Keyword Density/0.9
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: GetRight/4.2
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: Aqua_Products
Disallow: /

User-agent: Radiation Retriever 1.1
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: PerMan
Disallow: /

User-Agent: OmniExplorer_Bot
Disallow: /

More reading: http://perishablepress.com/5g-blacklist-2012/

follow up here..

Thanks HL, this has completely stopped all facebook bots!!! to this day..

 the robots.txt solution or the php code?

 RE:

 The php code snippet blocks FB bots.  The robots.txt file is something EVERY dolphin site should use.  It prevents undesirable bots from mining your site for various content, and using up bandwidth.  It also stops search engines like Google from indexing entire directories of your site, that have no business being in the Google index.