Closed Thread
 
Thread Tools
Old 06-01-2004, 11:46 AM   #1   [permalink]
eva2000
Administrator
 
eva2000's Avatar
 
Join Date: 23 Jun 2000
Location: Brisbane, Australia
Posts: 12,407
Animeboards.com server downtime [Dec 3, 2003 - Jan 9, 2004]

What a mess it has been for the past 5 weeks due to various server hardware failures. But now Animeboards.com is back and on new beefy hardware as a result of replacing the dead hardware!

New Server specs

2U Rackmount
Dual AthlonMP 2400+ (replaced)
2GB REG ECC RAM
1x 18GB IBM 10k SCSI
1x 18GB Seagate Cheetah 15K.3 15k SCSI (replaced)
1x 120GB IDE backup drive (awaiting installation)

Timeline of events

I'll give you the timeline (AEST time) of events that happened and the previous server's specs.

2U Rackmount
Dual AthlonMP 2000+
2GB ECC REG DDRAM
2x 18GB IBM 10k SCSI
1x 80GB IDE backup drive

Dec 3, 2003:
AB's server (dual AthlonMP 2000+) exhibited problems rebooting. When it was manually rebooted at the datacenter NAC.net, it worked fine.

Dec 4-5, 2003:
AB server crashed and could not be booted up manually at the datacenter, NAC.net. After talking to Daniel at Effectonline.com who colocates and sponsors AB's server at NAC.net, it turns out one of the AthlonMP 2000+ cpus' fans failed and killed the cpu Lucky as a replacement cpu i think shouldn't cost more than US$120-150. If the motherboard had died it would of cost US$400-1000 to replace!

Dec 5 - 15, 2003:
The AB server has been running with 1 cpu instead of 2 cpus and even handling email etc has been alot slower. I had planned to upgrade AB to vBulletin 3 Gamma prior to this incident but the upgrade definitely needs the power of 2 cpus.

Dec 15 - Dec 31, 2003:
AB server went down again this time, the Tyan Dual AthlonMP motherboard's onboard scsi controller died. Also the 80GB EIDE backup drive died along with one scsi hdd detected as failing. So needed to order in new parts including a 64bit riser card, scsi controller card, a new IDE backup drive and a new scsi hdd to replace the failing one. All parts were ordered in and arrived except for the scsi hdd which is delayed due to Xmas period That's okay US$800-1000 in replacement parts not including the datacenter on hands help fees which normally for 12+ hrs of diagnosing would cost > US$900 by itself

Jan 1, 2004 6:30am:
AB server comes online again with new Dual AthlonMP 2400+ cpus, new 64bit riser card, scsi card, and a new 18GB Seagate Cheetah 15K.3 15K scsi hdd to replace the only 18GB IBM SCSI 10k hdd that holds the mysql database data to the forums. Right now just missing the installation of the IDE backup drive.

Jan 1 - 7, 2004:
Animeboards.com forum software is upgraded from vBulletin 2.3.3 to vBulletin v3 Gamma, RC1 and then RC2. However, still need to go down one more time to install the replacement IDE backup hard drive. Once the back drive is installed, new member registrations will be enabled again!

Jan 7 - 9, 2004:
Upgraded Apache from 1.3.28 to 1.3.29, and upgraded PHP from 4.3.2 to 4.3.4. However, the upgrades broke Apache and Turck Mmcache app. So had to recompile Apache and PHP a few times, to get everything working again. Unfortunately, I had to remove Turck Mmache app, so PHP scripts are no longer cached/optimised which you may notice will slow down PHP pages on the forums


As you can see, the server chucked a mighty big fit right before Xmas which was out of my control. I'm sorry for the inconvenience this has caused. We missed Xmas and New Year celebrations on the forums

I'd like to thank Animeboards.com's host and sponsor, Daniel Pearson @ http://effectonline.com for his help through this. Daniel specifically purchased this dedicated server to host Animeboards.com and sites which cost an equivalent AUD$5,000 18 months ago! Without Daniel's sponsorship and hosting AB for the past 18 months, I would have struggled to stay online!

best regards your admin

George Liu
aka eva2000
Animeboards.com
__________________
be afraid... Admin cap is back... !

Last edited by eva2000; 08-01-2004 at 11:41 PM.
eva2000 is offline  
Old 08-01-2004, 05:40 PM   #2   [permalink]
eva2000
Administrator
 
eva2000's Avatar
 
Join Date: 23 Jun 2000
Location: Brisbane, Australia
Posts: 12,407
Note:
Animeboards.com forum's server will need to go down one more time later on in order to install the replacement IDE back up hard drive.
__________________
be afraid... Admin cap is back... !
eva2000 is offline  
Old 20-01-2004, 12:14 AM   #3   [permalink]
GouryG
Member
 
GouryG's Avatar
 
Join Date: 11 Jan 2004
Location: Central, NJ
Posts: 83
Send a message via AIM to GouryG Send a message via MSN to GouryG Send a message via Yahoo to GouryG
Oh man, it sounds like you had a similar problem like the AIC board had. Their problems started when a HD crapped out and they lost about a month's worth of posts because it hadn't been backed up in a while. They replaced the HD and copied the saved files to the new HD and then that's when the real problems started. They couldn't get the system to boot into a stable enviorment, They tried to do a work around untill they could find the culprit causing the problem then they had some sort of a cascade failure of the entire system. That failure corrupted everything. They decided to build a bulletproof web server for the board with redundency built in. Their biggest problem was recovering the BBS itself. They managed to recover 80% of the posts from the corrupted HDs. They finally got the board online again the last week of december. I guess it just goes to show that The computer gremlins can hit anyone at any time.
__________________
No Matter Where You Go, There You Are.
GouryG is offline  
Closed Thread

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New Fansubs for the Upcoming Season? jennwenn General and Other Anime 44 17-02-2003 12:55 AM
new anime in january? Fatal Outlaw General and Other Anime 10 26-12-2002 02:44 AM
Nov 20th server downtime eva2000 News 2000-2003 0 20-11-2001 10:52 AM
[News] Animeboards downtime on Jan 13, 2001 eva2000 News 2000-2003 2 13-01-2001 12:25 AM
[Log] Forum closure for Jan 1 - 5, 2001 eva2000 News 2000-2003 8 06-01-2001 08:00 PM


New To Site? Need Help?

All times are GMT -4. The time now is 05:58 PM.


Powered by vBulletin® Version 3.8.10
Copyright ©2000 - 2017, vBulletin Solutions, Inc.