Plaxo OPS Blog tag:opsblog.plaxo.com,2007://5 2006-08-30T09:05:46Z Movable Type 3.2 Databases offline tonight tag:opsblog.plaxo.com,2006://5.175 2006-08-30T09:02:51Z 2006-08-30T09:05:46Z At midnight we had 2 databases go offline because of a SAN storage issue. The downtime lasted 29 minutes, ending at 12:29 AM Pacific time. We apologize for the downtime.... Ethan Erchinger http://opsblog.plaxo.com At midnight we had 2 databases go offline because of a SAN storage issue. The downtime lasted 29 minutes, ending at 12:29 AM Pacific time. We apologize for the downtime.

]]> eCards site downtime tag:opsblog.plaxo.com,2006://5.166 2006-06-28T18:35:14Z 2006-06-28T18:38:14Z This morning from 3:30 to 8:00 AM Pacific time our eCards site was experiencing issues, which caused failures in sending eCards, as well as a temporary complete outage of the site. The problem has been remedied and measures have been... Ethan Erchinger http://opsblog.plaxo.com This morning from 3:30 to 8:00 AM Pacific time our eCards site was experiencing issues, which caused failures in sending eCards, as well as a temporary complete outage of the site. The problem has been remedied and measures have been taken to ensure the symptom behind the outage will not occur again. Sorry for any inconvenience.

]]> Forums being upgraded tag:opsblog.plaxo.com,2006://5.137 2006-05-09T18:54:14Z 2006-05-09T18:57:58Z At 10:30 AM PST on Tuesday (5/9/06) we'll be upgrading our forums site. We expect downtime of about 6-8 hours. Sorry for the inconvenience.... Ethan Erchinger http://opsblog.plaxo.com At 10:30 AM PST on Tuesday (5/9/06) we'll be upgrading our forums site. We expect downtime of about 6-8 hours. Sorry for the inconvenience.

]]> (Resolved) Issues with User Database tag:opsblog.plaxo.com,2006://5.109 2006-04-05T01:40:34Z 2007-01-29T23:47:09Z At 4:00PM PST on April 4th we started experiencing issues with one of our user databases. We are working to resolve this issue, and expect a recovery within 1 hour. This user database has been brought back online, and should... Ethan Erchinger http://opsblog.plaxo.com At 4:00PM PST on April 4th we started experiencing issues with one of our user databases. We are working to resolve this issue, and expect a recovery within 1 hour.

This user database has been brought back online, and should be fully operational, as of 5:20 PM PST.

Please talk with support@plaxo.com if you continue to notice any issues.

]]> Issues with user database tag:opsblog.plaxo.com,2006://5.108 2006-04-05T00:13:37Z 2007-01-29T23:47:34Z At 4:00PM PST on April 4th we started experiencing issues with one of our user databases. We are working to resolve this issue, and expect a recovery within 1 hour. Please reference the following FAQ for more details on how... Ethan Erchinger http://opsblog.plaxo.com At 4:00PM PST on April 4th we started experiencing issues with one of our user databases. We are working to resolve this issue, and expect a recovery within 1 hour.

Please reference the following FAQ for more details on how this will affect users on that system.
http://support.plaxo.com/bin/answer.py?answer=1224

]]> Intermittent access to website tag:opsblog.plaxo.com,2006://5.101 2006-03-28T19:43:12Z 2007-01-29T23:48:17Z Between 9:45 AM and 11:00 AM PST we were expieriencing instability with some networking equipment, which caused either slow or no access to certain portions of our site. This problem has been resolved.... Ethan Erchinger http://opsblog.plaxo.com Between 9:45 AM and 11:00 AM PST we were expieriencing instability with some networking equipment, which caused either slow or no access to certain portions of our site. This problem has been resolved.

]]> UPDATE: Service Outage - 3/27/06 tag:opsblog.plaxo.com,2006://5.98 2006-03-27T22:04:56Z 2007-01-29T23:48:36Z We were able to restore all systems affected by the service outage around 12:45pm PST. During the outage, some users may have experienced problems accessing their Plaxo Online account and some new users were unable to sign up. We apologize... We were able to restore all systems affected by the service outage around 12:45pm PST. During the outage, some users may have experienced problems accessing their Plaxo Online account and some new users were unable to sign up. We apologize for the inconvenience and we are still investigating the root cause of the outage. If you continue to experience difficulties accessing your Plaxo Online account, please contact Plaxo Support

]]> Service Outage - 3/27/06 tag:opsblog.plaxo.com,2006://5.97 2006-03-27T18:47:41Z 2007-01-29T23:48:05Z Some users may experience problems accessing their Plaxo Online account due to an outage located in our datacenter. The operations team is currently investigating the outage and working to restore access to those affected users as soon as possible.... Stacy http://www.plaxo.com Some users may experience problems accessing their Plaxo Online account due to an outage located in our datacenter. The operations team is currently investigating the outage and working to restore access to those affected users as soon as possible.

]]> (Closed) Issues with user database tag:opsblog.plaxo.com,2006://5.95 2006-03-23T19:46:39Z 2007-01-29T23:48:48Z We are working to replace the hardware that houses this database, and have an ETA of mid-tomorrow before this problem is fully resolved. While we're working hard to get the replacement system into place, we're being very cautious with this... Ethan Erchinger http://opsblog.plaxo.com
We are working to replace the hardware that houses this database, and have an ETA of mid-tomorrow before this problem is fully resolved.

While we're working hard to get the replacement system into place, we're being very cautious with this migration as we want to ensure there is no data loss. Because of this it looks like the migration to the new hardware, which will close out this issue, won't happen until late tomorrow. Sorry for the inconvenience.

We've finalized the migration to our new hardware late yesterday. Sorry for the delay, but we were able to provide some additional stability to the current system, so took that time to perform some heavier data validation on the new system. Please contact support@plaxo.com if you continue to have any problems.

]]> Issues with user database (followup) tag:opsblog.plaxo.com,2006://5.90 2006-03-17T19:09:01Z 2007-01-29T23:48:59Z We are working to replace the hardware that houses this database, and have an ETA of mid-tomorrow before this problem is fully resolved. While we're working hard to get the replacement system into place, we're being very cautious with this... Ethan Erchinger http://opsblog.plaxo.com We are working to replace the hardware that houses this database, and have an ETA of mid-tomorrow before this problem is fully resolved.

While we're working hard to get the replacement system into place, we're being very cautious with this migration as we want to ensure there is no data loss. Because of this it looks like the migration to the new hardware, which will close out this issue, won't happen until late tomorrow. Sorry for the inconvenience.

]]> Issues with user database tag:opsblog.plaxo.com,2006://5.88 2006-03-16T19:30:14Z 2007-01-29T23:49:12Z At 06:30 PST on March 16th we started experiencing issues with one of our user databases. We are working to replace the hardware that houses this database, and have an ETA of mid-tomorrow before this problem is fully resolved. This... Ethan Erchinger http://opsblog.plaxo.com At 06:30 PST on March 16th we started experiencing issues with one of our user databases. We are working to replace the hardware that houses this database, and have an ETA of mid-tomorrow before this problem is fully resolved. This should be affecting ~3% of users, that will see periodic failures.

Please reference the following FAQ for more details on how this will affect those users.
http://support.plaxo.com/bin/answer.py?answer=1224

]]> Recent slowness and outages tag:opsblog.plaxo.com,2006://5.87 2006-03-16T19:14:03Z 2007-01-29T23:49:24Z One of our backend database servers has recently been experiencing corruption. This corruption has caused us to failover to our secondary system, which isn't as powerful. The switch, while providing stable performance, proved to be an issue because it caused... Ethan Erchinger http://opsblog.plaxo.com One of our backend database servers has recently been experiencing corruption. This corruption has caused us to failover to our secondary system, which isn't as powerful. The switch, while providing stable performance, proved to be an issue because it caused a seemingly never ending string of slow requests and timeouts for those affected users.

Yesterday morning (around 9AM PST) we swapped back to our primary system after a restore and those users were once again online. We are making changes to that back-end server pair to provide better redundancy and a more stable environment.

]]> Welcome to the Plaxo Ops Blog tag:opsblog.plaxo.com,2006://5.86 2006-03-16T19:07:02Z 2007-01-29T23:49:34Z This blog will contain the most up-to-date information about the status of our service. If you're interested, please subscribe to our RSS feed. Thanks!... Mark Jen www.plaxo.com This blog will contain the most up-to-date information about the status of our service. If you're interested, please subscribe to our RSS feed. Thanks!

]]>