Server Downtime - July 31 - August 12, 2012 - Power Outage
Hi, everyone!
Sorry for the extended downtime...the EGB forums were unavailable for a while thanks to a power outage that occurred on July 31st, 2012.
While I was on vacation in South Carolina, a heavy storm hit Northern NJ, which caused many people to lose power completely for 4-6 hours. While I thankfully live very close to my power station (and therefore almost never lose power), I was getting brownouts and quick blips of outages for a while. As such, my UPS kept switching back and forth between running on the mains and running on battery. Eventually, the battery hit a sufficiently low level of power and initiated a shutdown of Delorean.
Thankfully, Delorean shut down successfully and gracefully, but the server never came back up. While I have my system configured to power on whenever it senses the power come on, the power never was fully lost. The UPS did hold the whole time...it just hit the 10% battery reserve threshold. Normally this wouldn't be an issue, but since I was away between July 26th and August 12th, nobody was there to hit the power button.
Anyway, the server is back up and running, and appears to be glad for the break. The system heatsink was very caked with dust, and I was able to disassemble the hardware a good amount and clear out a good bit of dust. Also, I was able to get the solid-state drive I installed a while ago for the Minecraft server fully installed inside the chassis (not just inside the external disk array).
OS-wise, the system is fine (not that I was really worried that Debian would fail, but after almost 700 days of updates and no reboots, sometimes you worry if you somehow uninstalled the kernel or something stupid...). The system is now running on Debian stable's latest kernel and everything is running just fine. The only issue at boot time was that the partitions all needed filesystem checks, which took about an hour to complete.
The only issue left at this point is that the backup disk array came up degraded, so I had to manually reintroduce the second RAID disk and initiate a resync. That will finish in about 6 hours or so, and I don't really want to bring up most of the servers until that's done. However, the EGB Forums will not be affected, as the web server and mysql array it uses do not cause much of a load on the system.
Sorry for the huge downtime, folks!
Sorry for the extended downtime...the EGB forums were unavailable for a while thanks to a power outage that occurred on July 31st, 2012.
While I was on vacation in South Carolina, a heavy storm hit Northern NJ, which caused many people to lose power completely for 4-6 hours. While I thankfully live very close to my power station (and therefore almost never lose power), I was getting brownouts and quick blips of outages for a while. As such, my UPS kept switching back and forth between running on the mains and running on battery. Eventually, the battery hit a sufficiently low level of power and initiated a shutdown of Delorean.
Thankfully, Delorean shut down successfully and gracefully, but the server never came back up. While I have my system configured to power on whenever it senses the power come on, the power never was fully lost. The UPS did hold the whole time...it just hit the 10% battery reserve threshold. Normally this wouldn't be an issue, but since I was away between July 26th and August 12th, nobody was there to hit the power button.
Anyway, the server is back up and running, and appears to be glad for the break. The system heatsink was very caked with dust, and I was able to disassemble the hardware a good amount and clear out a good bit of dust. Also, I was able to get the solid-state drive I installed a while ago for the Minecraft server fully installed inside the chassis (not just inside the external disk array).
OS-wise, the system is fine (not that I was really worried that Debian would fail, but after almost 700 days of updates and no reboots, sometimes you worry if you somehow uninstalled the kernel or something stupid...). The system is now running on Debian stable's latest kernel and everything is running just fine. The only issue at boot time was that the partitions all needed filesystem checks, which took about an hour to complete.
The only issue left at this point is that the backup disk array came up degraded, so I had to manually reintroduce the second RAID disk and initiate a resync. That will finish in about 6 hours or so, and I don't really want to bring up most of the servers until that's done. However, the EGB Forums will not be affected, as the web server and mysql array it uses do not cause much of a load on the system.
Sorry for the huge downtime, folks!
![Image](http://www.eagleworld.net/pics/eaglerockfahrenheit.jpg)