BoonEx.com under DDoS attack. UPDATED.

Andrew Boon posted 5th of June 2010 in . 54 comments.

As of now, we are at the second day of fighting with an ongoing DDoS attack. Alex and HFW team managed to relief the situation, but there're still some issues (forum feed, occasional slowdowns), and if the patterns happen to change BoonEx.com may be unavailable for some time.

Fret not though, whatever doesn't kill us, makes us stronger. And this is certainly not a kind of thing/people that may kill BoonEx.

UPDATE:

We have finally managed to take the attack under control. Due to some changes we had to make to traffic management some subdomains didn't work, including those that serve licences verification, download, etc. We're fixing it all now. Some of you may have been unable to login to your admin panel, since BoonEx data couldn't load. This problem occurred before due to BoonEx News RSS and license verification. We removed this dependancy some time ago, so Admin panels do not depend on RSS feed or license check. Unfortunately we have overlooked version checking mechanism, which still depends on BoonEx.com. We are really sorry about that. Long time ago our intent was to free your admin panel completely, and this is a misfortunate mistake. We are fixing it in 7.0.2 and in your current installation you may use this workaround: ------- set the following setting option to 0: [8/06/10 9:27:16 AM] Alex Trofimov: Admin -> Settings -> Advances Settings -> Other -> Show boonex news in admin panel ------- NOTE: version check and RSS feed in your admin panel are there for good reason. It is the way for us to notify you if there's something urgent, like a security update or a new version. This is not to spam you, but to inform you, and it can be turned off.

--------------

--------------

Yeah, weekend sucked, but we've learned a great deal. At the very peak they were firing 50Mbps of traffic at us for a few hours. Somebody must have ordered an expensive botnet attack. We almost feel flattered. 

Takeaways from this weekend:

1. We found a few spots that needed improvement. That will help us writing software that performs better.
2. We learned a few performance-tuning tricks for our dedicated server.
3. Found a good 3rd party protection service.
4. Found that pesky version-checking in admin panel.
5. Made a few notes for upcoming Dolphin releases.
6. Figured that we need to get rid of clutter in some (many) places.

All in all, we're good. On to 7.0.2 preparation.

 
Comments
·Oldest
·Top
Please login to post a comment.
Nathan Paton
I had a feeling this was the culprit of the continued downtime over the last few days. I wonder who'd want to take down the BoonEx web site?
LightWolf
Hmmm..I smell a rat has gotten loose. Wonder who it could be?..Well I have my ideas,but am sorry to see this happening to Boonex.
patrick81
grrrrrr go kill them, boonex. :)
CALTRADE
Who is doing it - and why - does anyone know? I couldn't get on yesterday.
theguypc
It's amazing the trouble some people will go through just to mess up a site.

Good luck guys. Don't let it get you down.
houstonlively
It could be the person that actually said in a blog post about a week ago that they were going to do it
houstonlively
That would be my first guess.
CodeSatori
As for possible motivations --- aside the general DOS in progress --- it's quite possible that this is an attempt to exploit the "database failure leads to compromised DB credentials in the debug output" scenario that has been around for the last few months.

@houston: Someone did mention an umphy botnet at their disposal a while back, yes. Might be a good idea to set up a separate domain/server(s) for redundancy to take over when one server is down. Just set up a script for checking see more current DDOS targeting and rotate servers automatically as necessary.
buckmcgoo
Well no one else has said it so it might as well be me... if you weren't using "Hostforweb" your site's would have been back up MUCH quicker and any other host would have been able to tell you who/what/how this happened... well actually any other host would have been able to stop this as it was happening.
Nathan Paton
@buckmcgoo: What makes you so sure that any other web hosting provider could have done any better? As far as I can tell, they have been stopping it as it is happening (otherwise this web site would still be down). These types of attacks can be difficult to stop, as any web hosting provider, large and small will tell you.
buckmcgoo
This site was down all day yesterday, or at least every time I tried to access it it was down... that is NOT what I call stopping it as it happens. Most providers stop DOS attacks at the hardware level and the users will never know anyone even tried. HFW doesn't stop it at any level. The reason I'm so sure is I tried using them in the past an their tech support didn't know their @ss from their elbow, they couldn't even answer the simplest questions. Kids try this crap every day with "bot nets" see more but it doesn't work if you are using a real host.
houstonlively
CS... I don't think there's any technical motivation. I think it's just that the little voices in their head told them to do it.
annabel
Is this the reason why I can't get into my admin panel ?
gameutopia
Usually it is someone extremely pissed off. You never know though. Plenty of crazy's out there.

I am surprised we haven't seen something like this here before now being dolphin is fairly popular. One of the things any server administrator should look into preventing and might deal with at some point.

Hope you got it tuned up and squared away. You know what they say shit happens!
toasty525
Is this the reason downloads have been disabled ?
tomakali
hmm, real crappy people snooping on boonex beware...
tomakali
i had a chat with HFB and asked them to report this issue to andrew, they asked me to do it myself. as a hosting partner shouldnt they be more supportive on behalf of boonex? something fishy........
marcoart
would this take down access to our own sites admin?
deano92964
Yes. It affects admin access to your site until the licence check times out. After you log into admin, just wait. The timeout can take a while. Give it a couple of minutes it will eventually timeout and finish loading.
brenaris
I have not been able to access my admin panels on my sites all damn day and it does not eventually timeout and finish loading a get a page that reads database error. Geezuz I'm so fed up with all the damn issues with this software!!!
houstonlively
@ Deano.... I don't believe the license is checked on every admin login anymore and there are no callbacks. They changed it so it's only checked when registering the license the first time.

I've had no trouble logging into my admin. You have to set Admin --> Advanced Settings --> Other --> Show boonex news in admin panel: To 0

This stops the admin pages from waiting forever on the boonex news to load.
ZopfWare
Houston, that would make sense because we were seeing some sort of errors that has to do with rss feed (probably from boonex) Thanxs for the fix.
LightWolf
Good to see things are getting better (hope).We need a no-pest-strip..lmao I still stand by my idea that an ex boonex member did this,(dead man walking). And he is probably reading all this as we speak .
brenaris
If I can't log into my admin then how do I change the advanced settings?

@LightWolf my intuition says you are correct on the "dead man walking" response...
houstonlively
brenaris, log into your site as admin on the user side, then go directly to yoursite,com/administration/advanced_settings.php

It's only the admin home page that's waiting on the boonex news feed.
Nathan Paton
@tomakali: HostForWeb has been aware and actively aiding in stopping the DDoS attack.
modmysite
I just sent out a mass mail to the 10k or so ModMySite.Com users with the advanced_settings.php and turn off "BoonEx News In Admin" fix stuff.... the same problem & solution as a while back when BoonEx had a problem and people could not get in to their admins. We sent out a mass mailing then too.
Anyways, should cut down on the panic some (at least for the people in our database).
-Smoge
124c41plus
Thanks HL for the tip on how to get to the advanced settings from the user side.

Knew something was up the other day when I got an insane error message page trying to load Unity. (It was so screwy I decided to save it as a webpage in case it was important.) If this can happen to Boonex, what chance do WE have?
deano92964
Hmmm. I could not get into my admin section yesterday from either the front, or the back door. But it did eventually load after a couple of minutes. So i am not sure what it is. Perhaps my affiliate ID. I will have to do some more checking.

There is also a featured modules section shown on the admin dashboard. Anyway to shut that off?
i am not able to get on the boonex site with my normal pc from germany,
firefox,google chrome and ie8 break the connection off.
only with my mobile phone i can write this.
houstonlively
Deano, I deleted all that stuff from the code. I want an admin section.... not an advertising billboard. Why should I have any external content at all in MY admin section? If I need Boonex news or content, I know where to find it.
theguypc
HL, you're my new hero!

Damn, this had been driving me nuts. I had no idea that there was another way into the admin section that would work.

Thanks man, you're a life saver.
mjsunifc
thank you so much houstonlively for the tip! your a lifesaver :)
bennyan
I can not active my dilphin 7 license through admin panel, anybody know when the site is fully functional?
theguypc
@HL - Is there a way to get rid of the "Featured Modules" block too?

@bennyan - Boonex has been under attack for several days. It may be a while yet seeing as how things aren't functioning correctly here yet : (
Keep trying and maybe you'll get lucky.
how for such business your company never considered to have a the server behind a firewall where it protects it from such a attacks ...
Nathan Paton
I don't think people actually know what a D/DoS attack is, or how it's resolved.
Andrew Boon
UPDATE:

We have finally managed to take the attack under control. Due to some changes we had to make to traffic management some subdomains didn't work, including those that serve licences verification, download, etc. We're fixing it all now.

Some of you may have been unable to login to your admin panel, since BoonEx data couldn't load. This problem occurred before due to BoonEx News RSS and license verification. We removed this dependancy some time ago, so Admin panels do not depend on RSS see more feed or license check. Unfortunately we have overlooked version checking mechanism, which still depends on BoonEx.com. We are really sorry about that. Long time ago our intent was to free your admin panel completely, and this is a misfortunate mistake. We are fixing it in 7.0.2 and in your current installation you may use this workaround:

-------
set the following setting option to 0:
[8/06/10 9:27:16 AM] Alex Trofimov: Admin -> Settings -> Advances Settings -> Other -> Show boonex news in admin panel
-------

NOTE: version check and RSS feed in your admin panel are there for good reason. It is the way for us to notify you if there's something urgent, like a security update or a new version. This is not to spam you, but to inform you, and it can be turned off.
theguypc
Thanks Andrew.

I bet this isn't the way you all planned on spending your weekend huh?

Good job keeping things fairly functional as much as you have.
Andrew Boon
Yeah, weekend sucked, but we learned a great deal. At the very peak they've been firing 50Mbps of traffic at us for a few hours. Somebody must have ordered an expensive botnet attack. We almost feel flattered.

Takeaways from this weekend:

1. We found a few spots that needed improvement. That will help us writing software that performs better.
2. We learned a few performance-tuning tricks for our dedicated server.
3. Found a good 3rd party protection service.
4. Found that pesky version-checking see more in admin panel.
5. Made a few notes for upcoming Dolphin releases.
6. Figured that we need to get rid of clutter in some (many) places.

All in all, we're good. On to 7.0.2 preparation.
Nathan Paton
I see you've changed your avatar, Andrew.
Andrew Boon
Yes, read an article - they say you should use your face, not logo :)
bennyan
as you say "3.Found a good 3rd party protection service." could you let me know what kind of service is that, we would like to consider that as well.
naluv4u2
I am trying to bear with all the problems, as I know there is NO SUCH THING as a perfect software, but this is rather disheartening. Modmysite provided the best information on the admin panel situation and I thank u 4 that. I was able to get in with this fix after being locked out of it all day. But now, I went from not getting into admin to NOT BEING ABLE TO ACCESS MY SITE AT ALL and am getting this error "Forbidden You don't have permission to access / on this server. Additionally, a see more 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request." THIS IS JUST 2 MUCH 4 ME! I have tried Dolphin on 4 different hosts (HostGator, BounceWeb, PacificHost) and now HostForWeb. In my opinion THEY ALL fell short (especially with my videos). Please someone tell me what's going on here really. . . Aside from that serious issue IS THERE ANYONE who can please tell me how to make the video quality better on my site. I have tried both the fixes provided by Sydeburnz and Flurmingo and my video quality looks exactly the SAME. No improvement whatssoever. I was only told the problem was fixed in version 7.0.2 which has an updated ffmpeg and codecs but how in the hell is this gonna help me if you can't upgrade live sites with it? PLEASE SOMEBODY explain what else can I do!
modmysite
As for their comment on version checking....
"4. Found that pesky version-checking in admin panel."

It has been an issue since 2005 with aeDating / BoonEx products.
http://www.modmysite.com/free-mod-exchange/325-quickmod-3-disable-version-checking-admin-faster-login.html#post612
and other posts on ModMySite.Com

Andrew, while you are at it, can we also beg for the Prune(Clean Database) settings not defaulting to 180 days! Maybe NEVER (or 9999) is a better default setting. Do see more you have any idea how many emails we have had to respond to over the years from people wondering why profiles are disappearing from their sites!

Post from 2007 (but it has been a problem since... what... 2003?)
http://www.modmysite.com/general-issues-comments-questions/5126-boonex-do-one-thing.html#post39875

Smoge
Andrew Boon
Sure thing. Will try to squeeze into 7.0.2
modmysite
>> 7.0.2 which has an updated ffmpeg and codecs
Off topic but... worth a reply perhaps.
Why use ffmpeg.exe ? Even on a shared account, the server administrator can add this to the server as a system call.
Use a system based ffmpeg... compiled from source or a yum (or apt-get) load.
With this, you have the latest and greatest ffmpeg, and you (or the system admin) can add codecs as needed!
Smoge
Denre
Although I condemn the attack and believe not even losers gain self esteem by causing harm to others, there seem to be some positive effects.

1.We learned a few performance-tuning tricks for our dedicated server.
2.Found a good 3rd party protection service.
3.Made a few notes for upcoming Dolphin releases.

Maybe Andrew would like to share some of these things with us, like the performance-tuning and protection service. It seems this might be useful information to anyone running a Dolphin see more site.

Besides these positive effects there are also some weaknesses shown. Three of the “found” issues, could have been found in the forum and it's a shame that it needed a DDOS attack to get the issues addressed.

1.We found a few spots that needed improvement. That will help us writing software that performs better.
2.Found that pesky version-checking in admin panel.
3.Figured that we need to get rid of clutter in some (many) places.

Never the less, lessons can be learned and we can transform weaknesses into strengths. The community is there for a reason and most of it only wants the best for Dolphin, so value its input and act on it.
Andrew Boon
The service that helped most was ServerOrigin/ethProxy. The cost is about $400/month for "cleaning pipes". You'd need such thing if you have a medium-to-large site. Hostforweb was also quite helpful at the first wave, but since they didn't have a hardware firewall or proxy with anti-DDoS software we had to seek for help elsewhere.

Alex also made a few changes that lowered the load to our server while attack was on it in full. He'd be able to share his observations later.

As for see more Dolphin - some of our views have changed, but I would like to tell about it a little bit later, when we have something to show. For now the plan is to clean out remaining callbacks, make a few more fixes and release 7.0.2. We're also in negotiation about installing 7.0.2 to a large site, which should help us tune Dolphin to work well with a large userbase. That should be the best part of 7.0.3.
mickscool
Thanks Andrew for keeping us updated with the situation via regular updates here and on Twitter. It really makes us feel that we are part of one big family and that's one main reason why there so many hard-core Dolphin fans around.

It was really good to know that you guys are negotiating for a big userbase site, it will really help grow Dolphin fast and I am pretty sure this is just the start for Dolphin ....

Keep Rocking and keep Smiling !!!
Mick - Toronto
mastermindsro
Good it's all gone and most importantly Boonex got many advantages and learned a lot out of this situation.. I just want to say to whoever done this attack: you guys should get a life ;)
LightWolf
Seems things are still unsettled. Been trying to get onto site today and get that temporary out of service page.
modmysite
Most larger data centers can install a firewall and other Denial Of Service mitigating service that can be provisioned on demand without service interruption. So paying a monthly fee for this is a waste for most unless you have continuing Denial Of Service issues. For less powerful DOS attacks, software can often handle it... csf, litespeed, failtoban, and so on, with no monthly cost. In addition, most larger data centers also offer, usually for free, Network IDS/IPS Protection.

Smoge
 
 
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.
PET:0.24234104156494