Current mood:

amused

Interesting weekend
Well, this weekend had some interesting issues.

I decided to upgrade our mail server to a different mail server product. The conversion application tells me it may take a few minutes to convert the data -- Well, it took over a day to do so.

Suddenly the network discovers we have unwanted visitors, a Trojan botnet army moved in, so I quickly added a line of code that automatically dealt with the evils.

During this time, I do various server software upgrades on Hal-9000, which leads to some, catastrophic issues with certain libraries behaving badly. Thus, I had to roll back the system. At that point one of the secondary services start acting funny (locking up, crashing) and I fear that some library is screwed up from the upgrade which wasn't removed with the roll back.

As I start investigating this issue, Trajan determines that it's primary file system has gone bad, so it sets everything to read only. Preventing services from saving it's databases (apparently caused by a power outage we had a while back) -- This issue was quickly resolved.

After extensive debugging on Hal-9000, I determined the issue was to do with previous problems we've had with a 'mysql' library, which is usually caused by connections issues. Sure enough, after investigating a little, I find that the network cable that connects Trajan to Hal-9000 is damaged.

Needless to say, all has been resolved now.

Update: I needed to replace three network cables and a network card - Did we get cursed or something?