Online since 2002. Over 3300 puzzles, 2600 worldwide members, and 270,000 messages.

TwistyPuzzles.com Forum

It is currently Sun Apr 20, 2014 4:02 am

All times are UTC - 5 hours



Post new topic Reply to topic  [ 20 posts ] 
Author Message
 Post subject: We're back.
PostPosted: Tue Sep 09, 2008 8:02 am 
Offline
User avatar

Joined: Wed Nov 24, 1999 12:18 pm
Location: Palerang Shire, NSW, Australia
Massive thank you to Sandy for organising the fix. I better get that back up done! :)

_________________
Wayne Johnson (Developer)
http://waynejohnson.net


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 8:56 am 
Offline
User avatar

Joined: Thu Jan 24, 2002 1:10 am
Location: Toronto, Canada
Yeah, so one of my scripts (the same one that caused us downtime last time this happened) was causing our host's server to die somehow. You guys are a pretty smart bunch, and rather than try to figure this all out myself, I'm going to ask for your help.

Attached to this message is a copy of the Apache server log. These are supposedly the ongoing requests that were "saturating the web server concurrent connection limit". He also said: "It looked to me like a DDoS attach based on the rapid rate that the connections were coming into the server and this is why it was disabled." I have no idea what that means. Maybe that my script is not closing database connections properly? What happens to a perl process and database connection running on the server if the user clicks the stop browser button or another link while it is running?

Examining the logs, it looks like most of them were clicks on the "Show me Everything" Quick Link on the search page. I removed that link, and also added some failsafes that limit puzzle database search strings to 3 more characters. Also, the Collection Manager search seems to still be broken in spite of my changes. I have disabled it completely in fear that I may have already PO'ed the host by testing it a few times. I really need to set up a stand-alone environment for testing changes like this.

Sandy


Attachments:
File comment: A tab-delimited text file containing the Apache server logs of the processes that were running when our host decided to pull the plug on our site.
TwistyPuzzlesErrorLog-2008-09-08.txt [20.36 KiB]
Downloaded 258 times
Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 9:13 am 
Offline
User avatar

Joined: Fri Feb 08, 2008 1:47 am
Location: near Utrecht, Netherlands
A DDoS attack is when malicious people send in many requests to a site (often using a botnet), flooding the server and ultimately, killing it. The search was a likely target, as it uses alot of CPU. (It searches a rather large database). I cannot tell from the logs, but I do not think your script doesn't close the connections to the database properly.

I think it would be wise if you built in some kind of delay between searches, to prevent people from flooding this CPU-intensive script.

It is striking alot of requests are made for an empty search (and often from the same IP), which will return all results. This causes alot of strain on the server. I think stopping the script once a certain number of matches is found (500+), and returning an error such as: "Too common search tags, please narrow down your search" will stop these attacks from abusing this script. A three character limit is usefull, but it will stop noone from searching for this: 'a e o', which will still return alot of results.

If the user terminates the conncection before the download is finished the script will close like normal. Nothing will happen, as the server won't know the user stopped and will just send the data, which will then go whereever underlivered data goes.

_________________
Tom's Shapeways Puzzle Shop - your order from my shop includes free stickers!
Tom's Puzzle Website


Buy my mass produced puzzles at Mefferts:
- 4x4x6 Cuboid for just $38
- Curvy Copter for just $18
- 3x4x5 Cuboid for just $34


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 9:24 am 
Offline
User avatar

Joined: Mon Jul 21, 2008 4:52 am
Location: Brighton, UK
Sandy wrote:
He also said: "It looked to me like a DDoS attach based on the rapid rate that the connections were coming into the server and this is why it was disabled." I have no idea what that means.

DoS attack stands for Denial of Service attack, where a website/webserver is hit with a rapid stream of time-consuming requests in an attempt to bring it down. I don't know anything about Apache servers but I do have a general suggestion: could you add to your code a condition such as, "If this is the 10th or more use of this search in the last 2 minutes, don't search but instead display a message, 'Sorry, the database is busy at this time, please try later?'" That should shield both site and host from further impact, also, when a user encounters that warning message they can post here with the date and time they received the message for you to try to spot a pattern, if it keeps happening. Hope this helps.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 9:45 am 
Offline
User avatar

Joined: Thu Jan 24, 2002 1:10 am
Location: Toronto, Canada
Well, I suppose it's possible a DoS attack could have hit us, but that seems unlikely. Probably a group of search engine spiders or something. The database could be improved, indexed better, making searches faster, and my SQL statements could likely be improved to. I was a relatively beginner in this stuff when I initially wrote those scripts. Searching for keywords through various text fields will likely always be slow as I have no intention of going the route of indexing all the words in the entire puzzle database. There are less than 2000 rows in this database, I'm sure the problem can be "solved" through optimization.

I'm still curious about which numbers in the server logs are the "bad" ones.

Sandy


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 10:17 am 
Offline
User avatar

Joined: Sun Sep 09, 2007 7:53 pm
Location: Melbourne, Australia
Hmm, maybe you should scrap the search feature in the mean time to avoid excessive CPU load/bandwidth usage.

I generally use Google to search the content of a site... I should look at the Zettair project at my uni and figure out something to contribute to the search engine here <.<

Tim.

_________________
3x3x3 Single: 16.02 | 3x3x3 Average: 21.90 | Magic Single: 1.11 | Magic Average: 1.23


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 10:40 am 
Offline
User avatar

Joined: Thu Jan 06, 2005 8:53 pm
Location: Los Angeles
I don't know much (anything) about webhosting, but I did read something a couple days back.
Perhaps it was Cuil's twiceler spider?
http://www.techcrunch.com/2008/09/01/is ... -websites/


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 11:15 am 
Offline
User avatar

Joined: Fri Oct 05, 2007 1:45 pm
Phew. Glad we're back. I do hope it wasn't because of a DDoS attack that the site went down.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 2:50 pm 
Offline
User avatar

Joined: Sun Jun 04, 2006 10:05 am
Location: Minneapolis, Minnesota, USA
Bounb wrote:
Phew. Glad we're back. I do hope it wasn't because of a DDoS attack that the site went down.

I don't see who would have any problems with us though. :\

_________________
Fridrich
3x3 PB 22.63
3x3 Av 30.57

25, Male
Started cubing Oct 15 '05

Out of the game, but not completely.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 4:19 pm 
Offline
User avatar

Joined: Mon Aug 04, 2008 10:32 pm
Location: Near Cincinnati, OH
Noah wrote:
Bounb wrote:
Phew. Glad we're back. I do hope it wasn't because of a DDoS attack that the site went down.

I don't see who would have any problems with us though. :\


It's the internet...there are people who think it's fun to sabotage other people's hard work.

Great to hear we're back. I'd love to see the puzzles that i've never seen before be uploaded into the database. Also it'd be nice if there were more mechanism dissection pages, but that'll come soon i hope. Good to see you as well, Sandy.

-CC10

_________________
Image

Best solve: 28.84s

SSCoasters.net - the biggest K'Nex roller coaster website on the planet
KnexForum.com - the best forum for everything else K'Nex


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 09, 2008 7:38 pm 
Offline
User avatar

Joined: Sun Apr 06, 2008 6:43 pm
Location: right here
Just a few thoughts...

- looking at the log file, there are no repeating source IP addresses. Even a DDoS will have many hits from several hosts, so it doesn't look like a DDoS attack, at least from what I'm seeing.

- if it was a search spider, those addresses would look similar- like the first two octets at least would be the same. I get that on my site a lot with google and altavista. You can "tell" the search spiders to ignore certain files and directories using a robots.txt file. If you want some help setting one up, I'd be happy to help, but a simple search on google for robots.txt will point you in the right direction.

- I agree with the idea of restricting the number of searches per minute. Even one every 10 seconds should reduce the CPU load, if the search script is what's doing it.

- Has anything been added recently? I added an off-the-shelf intrusion detection system script on my site, and that ended up sucking down a ton of CPU cycles.

- I don't know what type of hosting you have, but you may want to talk with your provider and see if they are able to restrict the number of simultaneous connections. I had a similar problem about a year ago. A link to something on my site appeared on one of the major sites, and I got over 40,000 hits that day. My provider cut me off. Even after that, the processing from my site has grown, even under a normal user load. Because I'm not paying for a dedicated server, I had to either pay more for a semi-dedicated hosting, or restrict the number of users. So somehow my provider was able to set a parameter in apache to only allow a certain number of simultaneous connections. It's still in the thousands, but when you're talking about a lot of elements (images, etc), it's less than you think. Anyway, this is still better than getting shut down.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Wed Sep 10, 2008 2:52 am 
Offline
User avatar

Joined: Fri May 06, 2005 10:13 am
Location: Norway
Hi :-)

Since u have the log of all the requests done. Is it not possible to block some or all of those IP's entirely, at least for a while. If some user from those ip's complain about lost access you can make further investigation of their recent actions. Through asking and logs or whatever.

Rapid complex or time consuming queries is a frequent way of DDos attacks. Other ways i have heard is repeated request of ip renewal etc. All these requests will ultimately hog the server and bring it to its knees :|

IP-filtering is a useful way to go. If possible also set up a proper firewall around the hosting. This would require hosting on a dedicated server i guess (expensive).

- Per

_________________
"Life is what happens to you while you are busy making other plans" -John Lennon, Beautiful Boy


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Wed Sep 10, 2008 6:24 am 
Offline
User avatar

Joined: Sun Apr 06, 2008 6:43 pm
Location: right here
The problem with IP filtering is your system gets bogged down looking up and comparing all those IP addresses, even before it gets a chance to serve the page. The best way to deal with an issue like this is to find the vulnerability and fix it.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Wed Sep 10, 2008 6:50 am 
Offline
User avatar

Joined: Fri May 06, 2005 10:13 am
Location: Norway
flambore wrote:
The problem with IP filtering is your system gets bogged down looking up and comparing all those IP addresses, even before it gets a chance to serve the page. The best way to deal with an issue like this is to find the vulnerability and fix it.


Yes it was meant to be a temporary solution only. A relatively short IP-ban list wont make things slow. A long long list will. Especially a list of single addresses and no ranges.

-Per

_________________
"Life is what happens to you while you are busy making other plans" -John Lennon, Beautiful Boy


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Wed Sep 10, 2008 12:23 pm 
Offline
User avatar

Joined: Sun Jun 04, 2006 10:05 am
Location: Minneapolis, Minnesota, USA
coastercrazy10 wrote:
Noah wrote:
Bounb wrote:
Phew. Glad we're back. I do hope it wasn't because of a DDoS attack that the site went down.

I don't see who would have any problems with us though. :\


It's the internet...there are people who think it's fun to sabotage other people's hard work.

But I could see a group who would partake in a DDoS would much rather do it to other websites, such as Neopets or Gaiaonline.

_________________
Fridrich
3x3 PB 22.63
3x3 Av 30.57

25, Male
Started cubing Oct 15 '05

Out of the game, but not completely.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Wed Sep 10, 2008 1:01 pm 
Offline
User avatar

Joined: Fri Oct 05, 2007 1:45 pm
Well... there are some people that might hold a grudge... I'm not suggesting that they had ANY involvement. I'm just trying to rationalise.


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Mon Sep 15, 2008 8:24 am 
Offline
User avatar

Joined: Thu Jan 24, 2002 1:10 am
Location: Toronto, Canada
Well, there's a nice thought. No doubt we're just looking at search engine spider bots, which seem to chew through the site on an almost continuous basis.

I've definitely got some work to do in cleaning up my scripts, and am already started on it. I hope to have the Collection Manager search back on line this evening.

Sandy


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 16, 2008 12:01 am 
Offline
User avatar

Joined: Thu Jan 24, 2002 1:10 am
Location: Toronto, Canada
Sandy wrote:
I hope to have the Collection Manager search back on line this evening.


"Keyword / Keyphrase Search" and "Advanced Search" are now working, but the "Search For New Additions" and "Browse By Section" searches are still busted.

Way past my bedtime.

Sandy


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 16, 2008 12:08 am 
Offline
User avatar

Joined: Wed Nov 24, 1999 12:18 pm
Location: Palerang Shire, NSW, Australia
Thanks, Sandy. Be looking forward to testing that out!

_________________
Wayne Johnson (Developer)
http://waynejohnson.net


Top
 Profile  
 
 Post subject: Re: We're back.
PostPosted: Tue Sep 16, 2008 3:09 am 
Offline
User avatar

Joined: Fri Nov 04, 2005 12:31 am
Location: Greece, Australia, Thailand, India, Singapore.
I did some testing with the search function and it works as good as it used to.
In the recent past it only covered one year's posts, but now it includes everything again.
Thanks Sandy, this is brilliant news!

:)


Pantazis

_________________

Educational R&D, Gravity, 4D Symmetry, Puzzle Ninja, Matrix Mech, Alien Technology.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 20 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Forum powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group