|
IMPORTANT: RackForce has scheduled a maintenance window for Tuesday
January 22, 2008 beginning at 11:00 PM until approximately 12:00 AM
PST (GMT -7). RackForce will be performing a network cut-over that
may briefly effect some segments of our network. We do not anticipate
any service interruptions to exceed 5 minutes in duration. Category:
Network Date: Tuesday January 22, 2008 Time: 11:00 PM to approx.
12:00 AM PST (GMT -7) We sincerely apologize for any inconvenience or
impact this maintenance may cause you or your customers. We have
reviewed and organized the technical procedures to minimize the
impact on our customers and senior technical staff will be onsite for
the entire duration of the maintenance window.
|
|
Everything's set with the migrated server. For some reason mySQL didn't start up on boot but I have started that and set it to autorestart in case it fails.
Let me know if there are any problems. (Drop me an email to my gmail account.)
|
|
I've tracked down some intermittent problems I've been seeing with the server and it turns out we've been hitting our 512M of memory. I've put in for an upgrade from the DDS-200L to the DDS-400L and they're working on that now. Services may be intermittent until the work is complete. Nothing should get lost in transfer unless it's been posted in the last 15 minutes (or until the migration is complete).
Assuming I'm awake, I'll post when it's done.
|
|
(I'm posting this for my husband since LiveJournal is blocked at his workplace. --Nonny)
Original Post by Nathan Clemons:
I am able to ping my server, but I am unable to SSH in (connection timed out).
I am also seeing a lot of "Too many DB connections" on the CSC, leading me to believe that there is likely something going on on your end. Please advise.
Responses:
Posted By: Denis S.
Date: Sat Oct 27 2007 4:00 am
We are currently experiencing a network issue which appears to be related to an inbound DDOS attack. Our networking staff is aware of the issue and is working to mitigate it.
We sincerely apologize for any inconvenience this may cause. |
|
Web services may be interrupted as I add an additional support module to the webserver. I will edit this post when everything is complete. (Adding DAV support for Subversion server support, for the curious.)
EDIT: forgot to update this before I left yesterday, but this is complete.Current Music: Lullacry - Head Like A Hole (Nine Inch Nails Cover) |
| » Update... |
|---|
Posted By: Denis S.
Date: Fri Oct 12 2007 6:04 pm
Hello,
It appears as though we will be unable to recover from this particular hardware failure. I have provisioned you a new server and am currently in the process of restoring your most recent complete backup from: Oct 10 03:56
This will unfortunately take some time and i will be updating you when it has completed. We sincerely apologize for any inconvenience this has caused.
--Denis S.
RackForce Support
https://support.rackforce.com
Posted By: Denis S.
Date: Fri Oct 12 2007 8:08 pm
Hello,
Unfortunately your most recent backup failed to restore. It appears to have corrupted during creation.
I will mount what i was able to extract inside your new environment in the /mnt/old directory. I sincerely appologize for any inconvenience this may cause.
I will update you when i have completed this.
--Denis S.
RackForce Support
https://support.rackforce.com
Posted By: Denis S.
Date: Fri Oct 12 2007 8:26 pm
Hello,
Due to a complete data loss hardware failure on your hardware, we have taken the liberty of provisioning you a new server with the same IP addresses:
209.97.214.222
209.97.214.223
You can access your server via WHM http://209.97.214.222:2086/ with root password censored
Unfortunately your full backup could not be completely recovered, but i was able to mount what was successfully backed up in your /mnt/old directory in your new environment. If you require assistance with any problems restoring from any personal backup, we would be happy to assist. We apologize for any inconvenience this may cause.
--Denis S.
RackForce Support
https://support.rackforce.com
Posted By: Denis S.
Date: Fri Oct 12 2007 8:26 pm
this ticket has been closed
Posted By: Nathan Clemons
Date: Fri Oct 12 2007 9:53 pm
This ticket has been re-opened in the CSC.
Posted By: Nathan Clemons
Date: Fri Oct 12 2007 9:59 pm
Please provide more details on the failure. One of the reasons I picked the DDS offering was that it was advertised as something that could be easily moved from one server to another in the event of a physical hardware failure. This was down for approximately 7 hours before a new server was provisioned. Was the hard drive / array the cause of the failure, or if not, why could the existing hard drives not be placed into another server to bring it back online without data loss?
Also, what caused the corruption of the backup? This is an _immense_ source of frustration for me, as I had just recently finished moving over from my old server and now I have to go through and set everything back up again. I thought that I had ordered the DDS Backup service; apparently I had not and that is my mistake. However, I would assume that you are making some backups in the event of physical failure. Is this something that would have occurred even with the DDS Backup service? None of the databases survived the restoral.
These are things I need answers to if I am going to continue to do business with your company. I have been very happy with your service up to this point, and I am very unhappy about this disaster and needing to roll my customers back about a week or more. Will there be a service credit due to the downtime and data loss?
Posted By: Denis S.
Date: Fri Oct 12 2007 10:13 pm
Hello,
We suffered an additional array failure during a RAID rebuild, this has corrupted all the data on the server, this is something we cannot recover from. We apologize for any inconvenience this may cause.
The DDS backup failed as it appears the backup was stopped during the last attempt. I recovered the backup data to the best of my ability and mounted it in your new environment.
Please contact our sales team with any concerns about credit issues.
--Denis S.
RackForce Support
https://support.rackforce.com
Oct. 13th, 2007 @ 03:07 am
|
| » Server down |
|---|
Hello,
Unfortunately the hardware that your virtual server is hosted on has encountered a problem and is down at the moment. Our technicians are working on the problem and hope to have it resolved soon.
I do not have an ETA yet, once we know more from our provisioning team we will update this ticket.
Thanks
--Alistair S.
RackForce Support
https://support.rackforce.com
Oct. 12th, 2007 @ 06:43 pm
|
| » Rackforce Maintenance |
|---|
RackForce will be performing critical updates to specific gear on
Thursday January 25, 2007 between the hours of 22:00 and 23:30 PST.
Service interuptions are not expected to exceed 5 minutes in
duration.
What: Critical Service Window
When: 22:00 - to - 23:30 PST
Date: Thursday January 25, 2007
We apologize for the short notice and sincerely appreciate your
understanding in this matter.
Jan. 25th, 2007 @ 07:11 am
|
| » Upgrade... |
|---|
The upgrade to the DDS-400 was performed yesterday (this is why chat and the MUD had to be restarted).
Everything appears to be running fine... and a bit faster. Though it's entirely possible we're the only ones on the physical server at the moment as they had to order more when I put in the upgrade request.
Dec. 17th, 2006 @ 12:41 am
|
| » FYI... Upgrade |
|---|
I've put in a request with RackForce to upgrade from the DDS-200L to the DDS-400L.
This will upgrade from 256M RAM to 384M, from 5 environments on the server to 3, and from 10GB hard drive storage to 20GB.
Sale dept. isn't staffed on the weekend so no word on when the upgrade will occur. I'll post when I hear further news.
Virtually everything should be off HostTech at this point. I think the only thing left is email for windsofstorm.net and web/email for camelotcreationsonline.com. DNS should all be updated at this point.
Dec. 10th, 2006 @ 02:21 am
|
| » HostTech |
|---|
The IPs at HostTech are down again. The nameservers should have been updated by now but apparently haven't made it into the root nameservers yet. I've sent an email off to HostTech to have them reset the box again.
The HostTech service is due to be shut down soon so anything else on the boxes will be moved shortly.
MUD users can just connect to the IP address (209.97.214.223) on 9000.
EDIT: Gary was in the office and just didn't hear the phone ring. It's back up now.
Dec. 8th, 2006 @ 04:30 pm
|
| » Rackforce's reply... |
|---|
EDIT: And a further response...
Oct. 1st, 2006 @ 08:46 am
|
| » Back up. |
|---|
See that RackForce is back online. Actually after further checking, RackForce is actually located in Canada, though who knows where their data center(s) are located.
I opened a ticket with them looking for an RFO (reason for outage).
Oct. 1st, 2006 @ 08:31 am
|
| » Status... |
|---|
Can't reach the RackForce hosted stuff or the RackForce support page. However, given where I work, I know that telcos do maintenance on the weekends (particularly Sunday AM). And I did just see this come in from Sprint:
MULTIPLE DS3 OUTAGE IN STOCKTON, CA
Since RackForce is West Coast, this is probably what's causing the routing failure. Should resolve on its own in a bit.
Oct. 1st, 2006 @ 03:26 am
|
| » Server... |
|---|
TLS was having an issue with the server so I went to reboot it. It hasn't come back up yet. I've opened a ticket with RackForce.
EDIT: Rackforce found a problem with the hardware node. They restarted it and everything's back up now.
Sep. 18th, 2006 @ 06:29 pm
|
| » Latency / timeouts... |
|---|
Seeing a bit of packet loss going to the RackForce server right now. Looks like it's a backbone issue at Level 3... I was going to post MTR results but it looks like the route has changed and is no longer routing through Level3 but through UUNet.
Looks like RackForce has their connection through Shaw Cable, who has hookups to UUNet/ALTERNet and Level 3. It looks like at the moment, the UUNet link is the active one.
This is something beyond RackForce's control, and looks like it's fine now... so Shaw Cable is doing a good job by RackForce.
Aug. 27th, 2006 @ 10:37 am
|
| » Done. |
|---|
Thanks to some help from the RackForce guys, I got it working. Compile took a bit but it's all good now and the RSS feeds should be working again, as well as the spell checker (which I'll check later).
Aug. 16th, 2006 @ 06:01 am
|
| » Bleh. |
|---|
Apparently there are some dependancies that I don't see a way to resolve at the moment to get PHP5 installed. I've posted a question in the RackForce forums, we'll see if I get an answer. This isn't a ticket, which I'd be billed for as outside the scope of my support range, but we'll have to see.
Aug. 16th, 2006 @ 02:46 am
|
| » Sunspot... |
|---|
Going to try to upgrade PHP from PHP4 to PHP5 on sunspot.
During the process the sites hosted on it will not be available/working.
Will post when done.
(Had an issue accessing cPanel -- filed a support ticket -- response in 10 minutes, requested feedback from me. I answered in 3 minutes, they fixed the issue in 7. I really like these guys.)
Aug. 16th, 2006 @ 01:54 am
|
|
![[info]](http://l-stat.livejournal.com/img/community.gif){kind=small}
windsofstorm![[info]](http://l-stat.livejournal.com/img/userinfo.gif){kind=small}