LiveJournal Maintenance

MemcacheD Update

Thu, 16 Jul 2009 21:24:58 GMT

Just wanted to let everyone know, that a new version of MemcacheD has been released. We will be rolling this out to the memcache nodes during the week of July 20th to 24th. This should have very little impact on the stability of the website; however users may see a slight increase in load times as the cache is re-populated with entries.

The software has been tested and verified to be working just fine with the application; so we perceive this to be a very minimal risk in regards to updating, and the stability of the website.

Thanks...

User cluster

Thu, 28 May 2009 17:09:44 GMT

I'm posting here 'cause Ops/Eng is a little busy. This morning one of our user clusters just... fell over. This caused a plethora of error messages and some journals to be completely unavailable. The issue was resolved in less than an hour, and we've switched to the slave while Ops works on it.

Edit 12:40 p.m. PST/19:40 GMT: ~~there have been a few lingering issues - we'll do a web tier restart soon, which has the potential to clear those up.~~ web tier restart is done

LJ Maintenance: Memcache Upgrade (16:00-24:00 UTC / April 28th and 29th)

Thu, 16 Apr 2009 00:27:55 GMT

Hello LJ,

** Edit, had to push back a week due to shipping delays :( **

Just wanted to give our users a heads up, that next week we will be performing work on our memcache tier. During this time, you may experience a little wait time while loading up journals and communities. We will be monitoring our website health and response times closely to ensure that our user experience is impacted as little as possible, however with maintenance work of this size, the possibilities of site slowness / intermittent downtime can increase.

These Upgrades will be performed on our servers during 16:00 - 24:00 UTC Apr 28st and 29nd.

We will be upgrading 1 machine at a time to minimize problems a user may experience.

Just a bit of FYI Backend Tech stuff... Currently our memcache tier is around 60GB, which we will be doubling to 120GB. By doing this, we hope to reduce the load on our db servers, as well as increase the response time of people fetching information that helps to display journals and communities. We will also be taking this time to upgrade our memcached client to be more resilient against single memcache server failures. All in all, these two changes to our memcache tier should help increase our users experience across our entire site.

Thanks...

Network Maintenance - April 4, 04:00 - 07:00 UTC/GMT

Thu, 02 Apr 2009 08:20:29 GMT

EDIT@6:12AM UTC

We're done with our work. No downtime or network drops were detected! Happy! All your positive thoughts definitely helped out! Please tell us that the site seems faster (even if it doesn't), cuz you know, I get needy when I'm away from home. :D Seriously, if you see any problems that are out of the ordinary support is the best way to contact us.

----

It's that time again! In a little over 12 hours, I'll be heading to the airport, leaving on a jet plane, not quoting the rest of that John Denver song (bless his heart) and arriving in our frosty cold data center in Billings, Montana to do some NETWORK MAINTENANCE.

This maintenance window will be from April 4, 04:00 - 07:00 UTC. For the rest of us not in a zone that is even remotely UTC'ish, please check out this link and choose your city/timezone.

Though the window is 3 hours, I do not expect livejournal.com to be down anywhere near that time. Of course we'll be working during that time, and we will see network connectivity "blips" as well as potential slowness but the site should be up and functional.

Swap our second internet router with the replacement that has already been shipped; FPU slot 0, PIC 2 doesn't work and since the FPU is built into the chassis, out goes the chassis!
Upgrade our firewall OS to a newer version that doesn't have IPSEC bugs; our site-to-site VPN keeps dropping and we lose administrative access to our servers until I reboot BOTH firewalls. Yay.
Redo small portion of our internal network to go over fiber gigabig (ok, gigabit). -- yes, we will embiggen a small portion of our network! Have I used this joke before? If I did, please bear with me because I'm old and this is the THIRD time I've scheduled this work and I tend to repeat myself.

LJ Maintenance: Feb 4, 05:00 - 07:00 UTC/GMT SECONDED

Wed, 04 Feb 2009 03:22:14 GMT

EDIT@07:34 UTC/GMT.
We're 1 important step closer to our goal! Tomorrow we have to break the db replicating briefly, push new code, test it, then if it goes well we'll push it to production and enjoy a slightly snazzier LJ by night time. There won't be a separate lj_maintenance post for that work though as it should (hopefully) be transparent to us.

Remember to check status.livejournal.org just in case things don't go according to plan; we'll make sure we update that.

---

This isn't a repeat of our previous post, rather an update. I got tired of doing all that "EDIT1", "EDIT2" hoopla. New posts all around!

The main focus originally was to get the network fixed. That changed, we changed, *I've* changed. We're going to try the network changes but AFTER we try the database changes. The focus is trying to get the db changes in place for basically one community, one very large community, one very hyper community. And also to get my friend to stop IM'ing about it. :D

This means that there *will* be complete downtime on our website, probably an hour or more. The time window has stayed the same.

(I'm talking about databases, so I'm using the common terminology. No dirty or inappropriate allusions are being made here. I leave that for my personal journal. Huh huh huh.)

We've been running the alter statements on the slaves, then flipping the master/slave after replication is caught back up. We've still got 4 user clusters to finish before we flip them.

* Fix the database errors we got on user cluster 01 when we flipped it earlier today and OHNOTHEYDIDNT hummmm, "pre-released". Those were air quotes and I actually took my fingers off the keyboard to wiggle them in the air. Wiggle. We've been saying that today would be the day that they're back up and running 100% and we've had to slip on the date we've given them. But we're trying and we'll be trying through tonight.
* Run the alter statements on the global cluster master and all slaves, 'cept one. That one is special.
* Do the network stuff I've been planning since December of 2008.

We're going to try to stuff all 3 things in a maintenance period originally scheduled for just 1. But have no fear, mhwest and nicholaskurjan are actually IN Billings doing the work. I told them they can't come home until everything is fixed. They responded that they'd bring me back some good BBQ. Hugs and kisses abounded.

Network Maintenance: Feb 4, 05:00 - 07:00 UTC/GMT

Sat, 31 Jan 2009 00:09:52 GMT

This isn't related to ONTD, that saga is still continuing.

Time converter here. Find your city and watch what day it is since for those of us in the USA it's actually going to be on Tuesday night that the work wil occur.

Even though the window is going to be 2 hours in length, LiveJournal will not be down that entire time. Just be aware that *during* this 2 hour window, connectivity may be slow or timeout completely.

Details are in this lj_maintenance entry, see point #1 which we had to postpone until we got the extra parts in.

bt

Minor database maintenance -- ONTD!!!1!

Thu, 29 Jan 2009 07:09:00 GMT

EDIT@03:03:03AM UTC/GMT (7:03PM PST): Hey, I was having a Westmalle last night with my dinner when I was called, so that's probably why I rambled more than usual. I'll try to be more sensical tonight. is that even a word? sensical?

I think a wise man once said, "you better check yo'self before you wreck yo'self" or something along those lines. ONTD is great because not only do they provide hours of fun, they also push our application and highlight the pressure points. No, ONTD didn't break LJ, and no they didn't break themselves (well, not really), what they did is point out that "hey! there might be a journal or community that has over 16 million comments! no, really!".

Here's the poop.

Problem: LJ was designed with a 24 bit limitation on number of comments -- in the database and in crucial parts of our commenting code. This may have been done for speed optimization, resource conservation, for the children, whatever. It was done with good intentions and now we have BAAAD intentions because the theoretical limit for a journal's comments is 16,777,216 (2^24). And ONTD just did *that*.

Solution?: Our crack team of crack teamsters is picking through code and releasing the 16 million limit (see here) as well as modifying the databases.

Risk?: We haven't deployed ANY of the new code to the website yet, and we've only started testing the alter statements on one of our secondary databases, but if things continue to test well, there is a SMAAAAALL risk that when we attempt to modify the rest of the databases, LJ could drop bits. All over the floor. But we're really trying to make sure that doesn't happen, and in spite of burr86 and dormando's help (hahaha, got you guys) tupshin, henrylyne, gariev, nicholaskurjan and mhwest will continue to work at getting rabid ONTD fans their due fix. It might be a bumpy weekend.

---
EDIT@08:16AM UTC/GMT (12:16AM PST): The problem is even more insidious than we initially thought! That means the original ONTD community is going to be in read-only mode until at least tomorrow when a more comprehensive solution can be attained. In the meantime, if you hurry to (the temporary) ontd you might be able to get in your comment before the 60th page. Like I said though, you gotta hurry.

---
So we hit a limitation on one of our databases and we're going to do some emergency maintenance to move ONTD (ohnotheydidnt, please don't go there because it's uhhhh, overloaded, PLEASE, THINK OF THE CHILDREN!) and well, I'll edit this post when we're done. Again, this is only going to affect ONTD. We're going to put it in read-only mode so no new entries and no new comments until we're done. Thanks for your patience.

Network Maintenance: Dec 17, 2008 02:00 - 04:00 UTC/GMT

Tue, 16 Dec 2008 18:34:10 GMT

EDIT3: Maintenance for tonight is done as of December 17, 0400 UTC/GMT. Thank you everyone! Since the right parts didn't come in, we'll try to do the final bit of maintenance tomorrow. Even though we're having the parts overnighted I'm not sure when exactly they'll arrive. Once they arrive, I'll put up another post but it'll probably be within an hour of us doing the work.

EDIT2: The correct parts did not arrive in time. We will proceed with part of our network maintenance but will have to take another maintenance window tomorrow. Sorry everyone! The times for this window will *not* change though as we'll take the time for extra testing.

EDIT: To convert UTC/GMT to your local timezone, you can check out http://www.worldtimeserver.com/convert_time_in_UTC.aspx or http://timeanddate.com/worldclock/meeting.html. There's lots more sites out there if you don't like the above 2, just search for something like "convert UTC time zone" in your search engine of choice.

The site will be intermittently unavailable during the times of December 17, 02:00 to 04:00 UTC/GMT time.... which is about 7 hours from this posting!

Sorry for the relatively late notification, we're still waiting on some last minute parts to arrive so it's possible that this might not even happen today; I'll edit this post with updates as well as status.livejournal.org so check there for what's going on if you can't reach the main livejournal.com site.

Our main goals tonight are:
1) ~~redo our core network infrastructure for performance.~~ Parts didn't come in time.
2) enable BGP with another ISP for redundancy.

Remember right after the move when LiveJournal was really slow in loading pages? A large part of that was due to choosing (ok, ok, *I* chose... < insert chagrined and very sad face here >) a switch module that had gigabit interfaces but *shared* a gigabit connection to the backplane amongst every 4 interfaces! That's akin to having 4 large pipes funnel down to just 1 large pipe, which equals a big bottleneck as our traffic all of a sudden fights for effectively 1/4 of the capacity it needs. Input queues were constantly overwhelmed and packets were getting dropped on the floor and having to be retransmitted; it was ugly.
We had a temporary work around where we bypassed these modules and plugged the distribution switches directly to each other (ether-channelled/trunked), and ran just specific VLANs. That seemed to work ok, but we need to get rid of these cables running back and forth and go back to our original design.

BGP. Well, we had uRPF checks enabled but they were in strict mode, and we were getting asymmetric routing going on with our use of path prepending inbound and local pref outbound, so I really think loose mode is what will solve the problem of some of you not being able to reach us when we were multi-homed.

Hotmail success and notifications caught up

Fri, 28 Nov 2008 21:21:59 GMT

It was a little later than we originally were led to believe, but yesterday Hotmail did unblock our new IP addresses and the floodgates were opened. Over the next 10 hours or so, we delivered upwards of 800,000 mails just to Hotmail, in addition to our regular load.

During that time, some notifications and delivery to other providers got briefly backed up again, as processing that volume all at once is not exactly what LiveJournal is designed to do. But everything that we can deliver is now delivered. Our logs are showing no major email carrier is blocking us, and the Great Notifications Saga of 2008 is hereby declared over.

Post-move update

Wed, 26 Nov 2008 23:26:10 GMT

We completed our data center migration on November 18, 2008, but the work hasn't ended. All of us have been impacted by problems after the move such as delayed email notifications, site uptime problems and well, I won't outline ALL of them here but you can check http://www.livejournal.com/support/ for the "Big Blue Box" (of despair! No, I jest... slightly). The only good news is that almost every problem is a "new" one; we're not seeing the same root causes ones over and over again.

I just wanted to let you know that the ops and engineering crew at LiveJournal feel your pain, probably in slightly different ways than you, for slightly different reasons. I'm not going to pretend to even start to comprehend what LJ means to you personally, but we're also LJ users too. I've found real friends through LJ, found it a source of release during my good times and bad times, and have discovered endless hours of lulz, information and insight in these pages, from people just like you.

In short, you don't like downtime, and we don't like it either. I'm very proud of the dedication and ability of the support, ops and engineering teams here (I respect all our groups here but since this is lj_maintenance); the recent results for site uptime though, let's just say we've got a lot of room to improve and lots and lots of ways to make you happy with us. Understatement. And we're continuing to work through our growing and learning pains, involve those that know the intricacies of this site better than we do and... not give up. And I hope you don't give up on us either.

Enough talk, here's what the operations group (with help from a whole lot of smart people) is working on right now.

1) Due to our data center move and IP renumbering, a lot of the mail providers are looking at us suspiciously. This is not their fault. Due to the amount of UCE ("spam") nowadays, they have to protect their users (you). Some of the bigger ones have throttled the rate of email LiveJournal sends them. This has caused our Schwartz queues to back up. This delays emails for EVERYONE. In our efforts to work the queues -- which are not just for email but also for other notifications and events -- we've overloaded our Schwartz database a couple of times. When the Schwartz database chews on its own fat for too long, it hangs our web servers. Hence, downtime. Weird, right?
-- We're working on the emails RIGHT NOW. Nick, the man, has been wrangling the queues with Abe and Dormando's help all morning and we're hoping to get the emails waiting to be sent out from around 300,000 to zero. It'll take many hours though and it's a balancing act so unfortunately I can not give an ETA on this. It was over 600,000 earlier this morning!

2) We're trying to improve our network resiliency and perhaps even better connectivity to non-USA locations. Our 2nd internet provider started announcing our route earlier today which caused a loss of connectivity for anyone going through them. Which is weird because (1) I was prepending the hell out of our ASN through them, and external route servers like route-views.oregon-ix.net showed it was correctly making it through and (2) even if routing was asymmetric, hey it's the internet, TCP/IP deals with it. I tried to troubleshoot it for a good 5-10 minutes and eventually just withdrew our announcement through the 2nd provider for now.
-- Will have to revisit this though I'm very hesitant to do it until rest of the stuff stabilizes. Look for an announcement for some network maintenance in mid-December. Worst case scenario -- which I don't like at all -- is that I manually fail over (i.e., start announcing) in the case of problems with the primary path.
-- 2 more providers will be coming up within the next 2 months.

I'm opening this up to comments as lj_maintenance used to be, but I hope you understand we won't be able to personally address or answer all your comments; it's cuz we're working on the site! :D

ScrapBook emergency maintenance

Thu, 20 Nov 2008 00:26:15 GMT

EDIT: 4:50PM Scrapbook (aka pics.livejournal.com) is back up! Thank you.

---------------

Sorry guys for the late notice! We've had to bring down the ScrapBook databases for some emergency maintenance. This means pics in your LiveJournal gallery are going to be unavailable. They should be back up within the next hour. I'll update this post when they're back online and we've checked them out.

bt

We've moved! (but still have to finish unpacking)

Wed, 19 Nov 2008 00:46:32 GMT

EDIT@6:54PM PST

We just implemented a temporary fix for our internal network and it *looks* promising. We'll let it bake in overnight and see how well it performs tonight and tomorrow.

------

Hi hi!

We've completed the majority of the move and had the site back up by 20:00 UTC (hey, look, I can post! You can read! Yay!) , but there's still some lingering issues that we need to clean up and are aware of so please be a little more patient while we wrap up the odds and ends.

I want to echo the lj_releases post where we thank burr86 and dormando for their ceaseless efforts, skill and knowledge. The LJ Ops and Engineering teams have/are *still* working their butts off but without burr86 and dormando's expertise, we could not have pulled this off.

For you techies, the slowness on our website is known, and the issue is currently isolated to oversubscribed ports on our core switches. That's direct from our vendor. Yeah, I know... Anyways, we're trying to implement a temporary workaround and we'll be looking to upgrade our hardware as a permanent solution. Please, hang in there a little longer!

bt

We're moving Tuesday, November 18, 2008 at 16:00-20:00 UTC

Fri, 14 Nov 2008 20:53:22 GMT

We're pulling the plug! And putting it back in a different location! Ok, that was bad.

Seriously though, our data center move -- from San Francisco to Montana -- is scheduled for THIS TUESDAY, November 18, 2008 at 16:00 UTC (this equates to 8AM Pacific timezone) and will last for 4 hours. Unfortunately it'll be total downtime for all our sites and services -- no posting, no reading, no email.

Another news announcement will be posted on Monday November 17, including links to updated FAQs that you'll want to check out before the move.

As always, status.livejournal.org will have any updates during the move so bookmark it and check it out for the latest news.

All of us at LiveJournal have been eagerly looking forward to this move and I want to thank you for your patience and understanding as we put the finishing touches on something that will allow us to bring you an even better product in the future.
Though for us in the Ops and Engineering teams, we'll be REALLY happy if, on Tuesday, everything just works the same way it did on Monday. Heh.

bt

8/30/2008 (UTC) maintenance DONE

Thu, 30 Oct 2008 04:49:32 GMT

burr86 decided that he did NOT want the site going down and prepped it so that we didn't have *any* interruption in service! AND the data center power maintenance was also completed successfully; truly a win-win situation. But hold the accolades for burr86 on the power stuff since the data center personnel completed that part and besides, we don't want him to get an even bigger head. hee hee.

bt

October 29, 2008 PST/October 30 UTC Scheduled Maintenance REMINDER

Thu, 30 Oct 2008 00:12:41 GMT

This is just a friendly reminder that TONIGHT we'll be taking our scheduled maintenance window!

www.livejournal.com will be unavailable from Thursday, October 30, 2008 at 03:30:00 UTC to Thursday, October 30, 2008 at 07:30:00 UTC.

Make sure you check status.livejournal.ORG for updates as they happen. That is a different web server on a different network which is why it will be up even when the rest of our web servers are being... maintained.

As a little note, even though you won't be able to post when we're offline, during that time you can *still* create entries that will be ready to be uploaded once we're back! How? Check out our Client downloads page if you haven't already! Clients, or little programs designed to work just with LiveJournal, are available for all sorts of computers including Windows, Macs, Linux, and handhelds. They're really cool and a lot of the clients also keep a local copy of your journal entry on your computer (just remember to save your work as you go along). That way you won't lose an entry if you accidentally close your browser or get disconnected from the internet *just* as you're posting. Don't ask me how I know about that. :(

Hope this tip helps and we'll see you back online at 07:30:00 UTC time.

bt

Downtime Sunday, October 26th and Monday, October 27th

Tue, 28 Oct 2008 05:36:55 GMT

On Sunday and Monday of this week, LiveJournal experienced some downtime due to a distributed denial of service (DDoS) attack. This resulted in LiveJournal being inaccessible for about an hour each morning, followed by some short periods of slowness or brief inaccessibility directly after.

For the past 12 hours all services and loading times have returned to normal, and we will continue to gather additional information for the purposes of preventing and mitigating future attacks.

This downtime wasn't related to the scheduled downtime on Wednesday, or our upcoming move to a new data center next month.

We'll continue to update http://status.livejournal.org/ as soon as reasonably possible once our operations team confirms downtime or interruptions in service. We're currently working on improving the process so that updates there can be posted more quickly.

October 29, 2008 PST/October 30 UTC Scheduled Maintenance

Fri, 24 Oct 2008 19:03:01 GMT

This community is still alive!

To have a slightly adjusted echo from our news post from yesterday:

Our San Francisco data center is taking a little downtime for power maintenance, which means so are we. You won't be able to access the site on October 29 from 8:30PM to 12:30AM (October 30) PST. We apologize for the interruption in service and we'll work hard to make sure we're back up and running as soon as possible!

Eh, one final edit: The maintenance window will be Thursday, October 30, 2008 at 03:30:00 UTC to Thursday, October 30, 2008 at 07:30:00 UTC. In the future I'll probably keep the time in UTC; I did it in the Pacific time zone since that's where our admin team is located.

I've found www.timeanddate.com to be a handy timezone converter if anyone wants to find out what time our maintenance window will be in their local time.

bt

database maintenance tonite, 12/22/07

Sat, 22 Dec 2007 23:38:23 GMT

We're going to be doing some database maintenance tonight, beginning at 10:00pm Pacific.

Due to the nature of this maintenance, we're expecting a very brief period of downtime, followed by a little bit of slowness as the site gets back up to speed. Apologies in advance for the inconvenience. All should be well by at least 11pm, if not before.

HAPPY SOLSTICE!

Re: scrapbook DB maintenance -- one more time

Wed, 22 Aug 2007 06:23:14 GMT

This maintenance is complete. Scrapbook is up. Thanks again for your understanding.

scrapbook DB maintenance -- one more time

Tue, 21 Aug 2007 06:32:33 GMT

Tuesday night we will be performing a maintenance on the scrapbook database. We need to take one more step to complete this. The Scrapbook application will be unavailable for approximately 70 minutes. This will not impact the main LJ site. The maintenance window will begin on Tuesday August 21st, at 10:00pm PDT. We're expecting that this will take no more than seventy minutes. We'll update this post once the maintenance is completed.

Thank you for your understanding, and apologies in advance for the inconvenience this will cause.

scrapbook DB maintenance

Tue, 21 Aug 2007 05:25:05 GMT

This maintenance is complete. Scrapbook is up.

scrapbook DB maintenance

Mon, 20 Aug 2007 21:57:26 GMT

Tonight, Monday night, we will be performing a brief maintenance on the scrapbook database, resulting in brief scrapbook unavailability. This will not impact the main LJ site. The maintenance window will begin on August 20th, at 10:00pm PDT. We're expecting that this will take no more than fifteen minutes. We'll update this post once the maintenance is completed.

Thank you for your understanding, and apologies in advance for the inconvenience this will cause.

scrapbook DB maintenance

Tue, 14 Aug 2007 00:26:49 GMT

Post-poned. We'll reschedule. As you were!

scrapbook DB maintenance

Mon, 13 Aug 2007 20:04:12 GMT

Tonight, Monday night, we will be performing a brief maintenance on the scrapbook database, resulting in brief scrapbook unavailability. This will not impact the main LJ site. The maintenance window will begin on August 13th, at 10:00pm PDT. We're expecting that this will take no more than fifteen minutes. We'll update this post once the maintenance is completed.

Thank you for your understanding, and apologies in advance for the inconvenience this will cause.

power outage

Wed, 25 Jul 2007 05:08:18 GMT

We're working on recovering from a power outage that affected LiveJournal for the past few hours. For more information about this event, please see news.