Our forum was shut down for a few hours due to an unusually heavy server load. My host thinks it may be an attack, but I suspect it's search engine spiders going nuts. Frankly, I have no good ideas of what to do about it yet. I could disable all bots from reading the forum, but that's a drastic move that I don't think I want to take. A spider bot forum style would probably help. I'll look into installing the latest forum software upgrades this evening just in case there is a security hole.

If anyone has any ideas, post them quick, before it goes down again! Heh!

Sandy

What about keeping track of which IP's make unusually large amounts of requests to the forum and then ban those IP's? Or the IP range if that is required. You could have a simple script make log files for each seperate day. The log file wouldn't need to be extensive, just keep track of the amount of hits for each IP.

Banning the bots from the forum is rather easy since the forum puts all the bots into a single group. You've probably seen the bots listed in the "who's online" section on the "Board index" page. The forum settings for the bots group is basically a black and white "allow" or "ban", though, which makes it undesirable. The consequences to banning spider bots is that search engine users will no longer be able to discover the site. According to the site stats, more people come here from google.com/search than anywhere else (although a large Chinese forum site is close behind). I'd like those users to continue finding the site's content!

It's a pickle. If the host shuts down the site again, I may have no choice.

Sandy

I wouldn't want all bots being banned from the site. I search on google regularly to find back topics on this site.

Isn't it possible like I suggested to track down the 'problem bots' and just ban those? You don't need to ban using the PHPBB settings, you could also ban them using htaccess files, which would allow you to specify individual IP (ranges).

Just in case you haven't noticed... I'm updating the forum this evening!

Sandy

I've been noticing little changes as I navigate between topics. It's kind of funny, actually.

it is a lot better
but how do you get to articles from the forum?

The top of the screen is looking much better to me already!
& so far everything is looking real clean!

Awesome. I like how now there's a GIANT "board index" button at the top of the page. (the phpbb thingy.)

I love this new layout, its cleaner in my opinion

type it out and add it to your hot bar.


Btw I love that board presence is back up. How about avitars?

I like the new layout.

Same here I love it very nice

Hehe, it's kinda interesting to watch as the changes are made...

I think it will look good once the changes are done.

Okay, forum updates complete for the time being. Let me know what I broke.

Sandy

I am not sure yet, but I might be missing the "mark all posts read" link.

Well the site was down again

I'm sure already that I miss it.

The "mark topics read" link in each sub-forum is up and running for me. Is this what you're talking about or was there some other button I never saw?

I see the "Mark all forums read" command on the board index page at the top right.

I see both now at the right top, whereas it used to be at the bottom left.

Think of it as a puzzle!

Sandy

Yep! The puzzle solved!

When I try to post on the general topics, the top url address bar is yellow. I have to close a window that opens up. it says this website is phishing? whatever that means. I've never had this before.

Hi. Include a robots.txt file to your website. This can specify what bots are allowed to crawl/index what parts of the website. Most legitimate crawlers would follow the "guidelines" politely. You could for instance allow google and yahoo spiders only ...

Per

PS!

Most likely you can analyse bot traffic through your hosting control panel stats section !

Last night while writing and e-mail to Sausage, I had what I believe to be a great insight into a major contributor to the original downtime. The forum style templates were all being stored in the database. I'm not sure exactly how this happened, but it was like that for a while. Each page that loads in this forum requires at least 3 templates, and often more, which means that many more hits on the database in addition to whatever else is requested from the database to display the page itself. When I updated the forum style last night, I switched this back to using the file system. I expect the load on the database server will be cut in half or more through that one action alone. So for now, Google can continue having it's way with the forum, since I'm paying for the bandwidth it eats up. I will definitely be adding in a stripped down bot style, though, to reduce the load even further. But for now, the pressure is off.

Roxanne, I haven't seen the fishing message in either Explorer 7 or Firefox 2. Can you give me a step-by-step process to reproduce the error? Also, let me know your browser, browser version, and operating system.

Sandy