View Single Post
Old 06-01-2004, 11:46 AM   #1   [permalink]
eva2000
Administrator
 
eva2000's Avatar
 
Join Date: 23 Jun 2000
Location: Brisbane, Australia
Posts: 12,407
Animeboards.com server downtime [Dec 3, 2003 - Jan 9, 2004]

What a mess it has been for the past 5 weeks due to various server hardware failures. But now Animeboards.com is back and on new beefy hardware as a result of replacing the dead hardware!

New Server specs

2U Rackmount
Dual AthlonMP 2400+ (replaced)
2GB REG ECC RAM
1x 18GB IBM 10k SCSI
1x 18GB Seagate Cheetah 15K.3 15k SCSI (replaced)
1x 120GB IDE backup drive (awaiting installation)

Timeline of events

I'll give you the timeline (AEST time) of events that happened and the previous server's specs.

2U Rackmount
Dual AthlonMP 2000+
2GB ECC REG DDRAM
2x 18GB IBM 10k SCSI
1x 80GB IDE backup drive

Dec 3, 2003:
AB's server (dual AthlonMP 2000+) exhibited problems rebooting. When it was manually rebooted at the datacenter NAC.net, it worked fine.

Dec 4-5, 2003:
AB server crashed and could not be booted up manually at the datacenter, NAC.net. After talking to Daniel at Effectonline.com who colocates and sponsors AB's server at NAC.net, it turns out one of the AthlonMP 2000+ cpus' fans failed and killed the cpu Lucky as a replacement cpu i think shouldn't cost more than US$120-150. If the motherboard had died it would of cost US$400-1000 to replace!

Dec 5 - 15, 2003:
The AB server has been running with 1 cpu instead of 2 cpus and even handling email etc has been alot slower. I had planned to upgrade AB to vBulletin 3 Gamma prior to this incident but the upgrade definitely needs the power of 2 cpus.

Dec 15 - Dec 31, 2003:
AB server went down again this time, the Tyan Dual AthlonMP motherboard's onboard scsi controller died. Also the 80GB EIDE backup drive died along with one scsi hdd detected as failing. So needed to order in new parts including a 64bit riser card, scsi controller card, a new IDE backup drive and a new scsi hdd to replace the failing one. All parts were ordered in and arrived except for the scsi hdd which is delayed due to Xmas period That's okay US$800-1000 in replacement parts not including the datacenter on hands help fees which normally for 12+ hrs of diagnosing would cost > US$900 by itself

Jan 1, 2004 6:30am:
AB server comes online again with new Dual AthlonMP 2400+ cpus, new 64bit riser card, scsi card, and a new 18GB Seagate Cheetah 15K.3 15K scsi hdd to replace the only 18GB IBM SCSI 10k hdd that holds the mysql database data to the forums. Right now just missing the installation of the IDE backup drive.

Jan 1 - 7, 2004:
Animeboards.com forum software is upgraded from vBulletin 2.3.3 to vBulletin v3 Gamma, RC1 and then RC2. However, still need to go down one more time to install the replacement IDE backup hard drive. Once the back drive is installed, new member registrations will be enabled again!

Jan 7 - 9, 2004:
Upgraded Apache from 1.3.28 to 1.3.29, and upgraded PHP from 4.3.2 to 4.3.4. However, the upgrades broke Apache and Turck Mmcache app. So had to recompile Apache and PHP a few times, to get everything working again. Unfortunately, I had to remove Turck Mmache app, so PHP scripts are no longer cached/optimised which you may notice will slow down PHP pages on the forums


As you can see, the server chucked a mighty big fit right before Xmas which was out of my control. I'm sorry for the inconvenience this has caused. We missed Xmas and New Year celebrations on the forums

I'd like to thank Animeboards.com's host and sponsor, Daniel Pearson @ http://effectonline.com for his help through this. Daniel specifically purchased this dedicated server to host Animeboards.com and sites which cost an equivalent AUD$5,000 18 months ago! Without Daniel's sponsorship and hosting AB for the past 18 months, I would have struggled to stay online!

best regards your admin

George Liu
aka eva2000
Animeboards.com
__________________
be afraid... Admin cap is back... !

Last edited by eva2000; 08-01-2004 at 11:41 PM.
eva2000 is offline