Brad Fitzpatrick ([info]bradfitz) wrote in [info]lj_maintenance,
@ 2004-05-17 19:54:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
20 servers down....
1 of our 4 racks of servers just lost power, so 20 servers are offline.

Unfortunately, users on the Madcow database cluster are temporarily off the net, since both Mad and Cow (which mirror each other) were in the same rack. (We should fix that!)

In the meantime, the site's just kinda slow because we lost most our new webservers.

Will post more when we know more.... Lisa and the Internap engineers are looking into what happened.

Page 1 of 6
<<[1] [2] [3] [4] [5] [6] >>

(Post a new comment)


[info]pat_rice167
2004-05-17 08:01 pm UTC (link)
sorry about that....was wondering why it was slow tho...

(Reply to this)


[info]patriot
2004-05-17 08:03 pm UTC (link)
That explains the problems with the site being slow and coming up unable to find web page.

Twenty servers down sounds like a pretty tall job for you guys.

(Reply to this)


[info]nrs135
2004-05-17 08:04 pm UTC (link)
Yikes, a whole rack. Thanks for keeping us posted on this, hope Internap figures it out soon.

(Reply to this)


[info]altena
2004-05-17 08:04 pm UTC (link)
Ah, I was wondering why I was having difficulties.

No worries, though! I'm about to go to bed, so you have approx. 8 hours to get things back up to speed. ^_^

(Reply to this) (Thread)
hi
[info]warloc
2004-05-31 11:52 am UTC (link)
i was just wondering to see if you have actually seen chobits i think its a cool series so that's all


laters

(Reply to this) (Parent)(Thread)(Expand)
Re: hi - [info]altena, 2004-05-31 01:15 pm UTC (Expand)

[info]eddiewoods
2004-05-17 08:04 pm UTC (link)
Yea I'm on Porkchop and I get "Page cant be displayed" after i check my friends page and I have to refresh to get it to load...

(Reply to this) (Thread)(Expand)

[info]ex_shattered767
2004-05-17 08:09 pm UTC (link)
Double check where you are.

I thought I was on Porkchop too, but I guess I was moved to Madcow within the last month or something.

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]bleything, 2004-05-17 08:54 pm UTC (Expand)
(no subject) - [info]ex_shattered767, 2004-05-17 08:58 pm UTC (Expand)
(no subject) - [info]bevived, 2004-05-18 11:27 am UTC (Expand)
(no subject) - [info]macros06, 2004-05-18 07:11 pm UTC (Expand)

[info]jaddziadax
2004-05-17 08:04 pm UTC (link)
good luck at fixing them!!

(Reply to this) (Thread)

[info]dopy
2004-05-18 05:49 am UTC (link)
cool icon

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]jaddziadax, 2004-05-18 09:51 am UTC (Expand)

(Deleted post)

(Deleted post)
(no subject) - [info]aedgeofeternity, 2004-05-18 04:08 am UTC (Expand)

[info]yum_cimil
2004-05-17 08:05 pm UTC (link)
You don't suppose it's weather related by any chance? I know we've been getting thunder and lightning storms in the Beaverton area earlier this afternoon.

(Reply to this) (Thread)

[info]bradfitz
2004-05-17 08:10 pm UTC (link)
Internap's in Seattle, and weather's not an issue. That place should be able to take an earthquake that would otherwise level Seattle.

I think we just didn't have enough power in there, and had no way to know. (We don't get graphs or data about that, and we've never planned for it.... just always assumed we had enough power. :-/)

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]abates, 2004-05-17 08:59 pm UTC (Expand)
(no subject) - [info]damnitnicole, 2004-05-17 09:06 pm UTC (Expand)
I smoked your uncle. - [info]expanding_soul, 2004-05-18 03:13 pm UTC (Expand)
Re: I smoked your uncle. - [info]damnitnicole, 2004-05-18 09:58 pm UTC (Expand)
(no subject) - [info]j9luvsmozart, 2004-05-17 09:23 pm UTC (Expand)

[info]risen_phoenix
2004-05-17 08:05 pm UTC (link)
Damn...

Best of luck, guys.

(Reply to this)


[info]ex_shattered767
2004-05-17 08:06 pm UTC (link)
Was wondering why I was having problems. Last time I checked (over a month ago) I thought I was on Porkchop, but a check 30 seconds ago revealed I'm on Madcow.

Thanks for letting us know. Good luck in figuring out what went wrong.

(Reply to this) (Thread)(Expand)

[info]pointclickkill
2004-05-17 08:34 pm UTC (link)
How do you know which server you're on? Probably a ridiculously stupid question, but I don't have a clue... =/

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]ex_shattered767, 2004-05-17 08:39 pm UTC (Expand)
(no subject) - [info]coolhappysteve, 2004-05-17 10:02 pm UTC (Expand)
(no subject) - [info]ex_shattered767, 2004-05-17 10:04 pm UTC (Expand)
(no subject) - [info]coolhappysteve, 2004-05-17 10:10 pm UTC (Expand)
(no subject) - [info]dopy, 2004-05-18 05:57 am UTC (Expand)

(Deleted post)
(no subject) - [info]ex_shattered767, 2004-05-17 11:09 pm UTC (Expand)

[info]madmade
2004-05-17 08:07 pm UTC (link)
So that's why lj is sucking for me.

(Reply to this)


[info]sunsplash
2004-05-17 08:08 pm UTC (link)
I knew I shouldn't have purchased a paid account today! I whammy'd it for everyone! Hah! :)

Hope my $25 helps keep lj up and running :)

(Reply to this) (Thread)

[info]damnitnicole
2004-05-17 09:07 pm UTC (link)
OMG u b0rked LJ.

*stabs*

(Reply to this) (Parent)

[info]chrishaas
2004-05-17 08:09 pm UTC (link)
Thanks for the explanation.

(Reply to this)


[info]kaimi
2004-05-17 08:11 pm UTC (link)
THAT's what happened....

(Reply to this)


[info]devlindreadwood
2004-05-17 08:11 pm UTC (link)
Was wondering why my client had a hard time connecting and i couldent access pages for the worth of dirt

(Reply to this)


[info]ooglie_wooglie
2004-05-17 08:11 pm UTC (link)
I think I'm GREEN but I couldn't access anything for about an hour either.

Good luck!

(Reply to this)


[info]arikia
2004-05-17 08:12 pm UTC (link)
So that's what was going on. I kept getting the "page cannot be displayed" and I couldn't log out of my other journal.
Thanks for the heads up. Good luck on fixing it!

(Reply to this) (Thread)

[info]finalconnection
2004-05-17 08:17 pm UTC (link)
hah i got the same thing too...i thought it was my computer

(Reply to this) (Parent)

[info]onedarkroom120
2004-05-17 08:12 pm UTC (link)
i was posting comments as it happened, lol

(Reply to this)


[info]poeticseduction
2004-05-17 08:13 pm UTC (link)
anyone have any idea of when it's going to be working again?

(Reply to this) (Thread)(Expand)

[info]ex_shattered767
2004-05-17 08:15 pm UTC (link)
"Will post more when we know more...."

I think they need to find out what went wrong before they can guess how long it will take to fix.

(Reply to this) (Parent)
as - [info]wonderland6, 2004-05-17 08:16 pm UTC (Expand)

(Reply from suspended user)

[info]ckd
2004-05-17 08:16 pm UTC (link)
I hope you find the lost power soon!

More seriously, that's a real problem (one we've had in my server room at work too) and I hope you find out what went wrong relatively quickly...I'm sure you will, though.

(Reply to this)


[info]imdwalrus
2004-05-17 08:17 pm UTC (link)
My journal seems to be working about half the time...Which half is down: the Pork or the Chop? Or does it not work that way? :)

I could've sworn that Pork and Chop were seperate, though, just like Mad and Cow...

In any case, good luck at fixing it up.

In other news, this post is an experiment: will I get any replies at all if I use Lakitu as my icon instead of the now-infamous Duck Hunt duck? :)

(Reply to this) (Thread)(Expand)

[info]schnee
2004-05-17 08:21 pm UTC (link)
You'll get a reply from me just because you used Lakitu. :) Yay for vintage gaming!

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]imdwalrus, 2004-05-17 08:38 pm UTC (Expand)
(no subject) - [info]mayhap, 2004-05-17 08:48 pm UTC (Expand)
(no subject) - [info]pavane, 2004-05-18 09:14 am UTC (Expand)
(no subject) - [info]bugtilaheh, 2004-05-17 08:51 pm UTC (Expand)
(no subject) - [info]astroasis, 2004-05-17 08:53 pm UTC (Expand)
(no subject) - [info]bossjock77, 2004-05-17 09:07 pm UTC (Expand)
(no subject) - [info]lucypevensie, 2004-05-17 09:14 pm UTC (Expand)
(no subject) - [info]bossjock77, 2004-05-17 09:25 pm UTC (Expand)
(no subject) - [info]lucypevensie, 2004-05-17 09:27 pm UTC (Expand)
(no subject) - [info]noda, 2004-05-17 09:49 pm UTC (Expand)
(no subject) - [info]dopy, 2004-05-18 06:25 am UTC (Expand)
(no subject) - [info]schnee, 2004-05-19 05:28 pm UTC (Expand)
(no subject) - [info]astroasis, 2004-05-17 08:47 pm UTC (Expand)
(no subject) - [info]moogleray, 2004-05-18 01:27 pm UTC (Expand)

[info]lizzie9208
2004-05-17 08:18 pm UTC (link)
ah, that explains why the site is so incredibly slow tonite.

(Reply to this)


[info]finalconnection
2004-05-17 08:21 pm UTC (link)
its TERRORISTS i tell ya!!!

*spaz attack*

(Reply to this) (Thread)

[info]dopy
2004-05-18 06:28 am UTC (link)
ha! ha!

(Reply to this) (Parent)

[info]crschmidt
2004-05-17 08:22 pm UTC (link)
Impressive that the site stays up. I think it's a testament to how well you guys have things running that you can lose 1/4th of your servers and still do just fine, albeit a bit slow.

Congratulations on such a great site.

(Reply to this) (Thread)

[info]bradfitz
2004-05-17 08:53 pm UTC (link)
Honestly I was a bit shocked.

I think we just got lucky, though. I'm not sure we could've survived as well with some of the other cabinets gone.

But if we did this well, I think total power loss in a cabinet is an almost-achievable goal... I want to rearrange some stuff now.

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]crschmidt, 2004-05-17 09:29 pm UTC (Expand)
(no subject) - [info]zarhooie, 2004-05-18 03:44 am UTC (Expand)

Page 1 of 6
<<[1] [2] [3] [4] [5] [6] >>

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…