CCBot/2.0 ( British Columbia Geocaching Association —» Forums —» General Caching —» General —» Jeremy's assessment
Home Newest Caches Forums Your Account About BCGA Search Blitz  
Predator Ridge
Predator Ridge:  [?] . Breathtaking views near Vernon. [M | C
Toggle Content=  Register or Login  Please Join our Association 
topleft topfill topright
Toggle Content Our Sponsors
Landsharkz Coins and Geocaching Gear
Worldcaching.comLandsharkz Coins and Geocaching GearGold CountryCaching Containers
bottomleft bottomfill bottomright
topleft topfill topright
Toggle Content Waypoints
bottomleft bottomfill bottomright
topleft topfill topright
Toggle Content Coming Events
bottomleft bottomfill bottomright
Forums —» General Caching —» General —» Jeremy's assessment
topleft topfill topright
Please be advised that we are currently having problems with some of our users not being able to log into the site. We apologize for this, we are working to address the situation. Thank you for your patience

Thank You
BCGA Executive

bottomleft bottomfill bottomright
Jeremy's assessment
If you have questions/comments about geocaching in general or anything that doesn't fit anywhere else, post them here.
Post new topic   Reply to topic   Printer Friendly Page    Forum Index —» General

View previous topic :: View next topic  
Author Message

Joined: Jan 24, 2004
Posts: 85
Location: Nanaimo

PostPosted: Sat Jul 04, 2009 11:07 pm    Post subject: Jeremy's assessment Reply with quote

For everyones information, here is Jeremy's blogged assessment of the 29 hours of hell at Groundspeak:

Our servers are a bunch of primadonnas. They demand to be pampered in the greatest colocation facility in the world (if you agree with the video of Fisher Plaza touting that fact), resting on pillows of AC and fed power in Waterford crystal goblets. We literally pay more for the 5 cabinets that house the servers* than we do our entire Groundspeak office - and then some.

Around 5am Pacific today, all of our grumpy but lucid Groundspeak servers woke from their slumber to greet geocachers** who were, as one user wrote, scratching their arms in search for their next geocaching fix. Most were just happy to have the servers back online but others were asking questions about disaster recovery and communication in a crisis. Instead of finger pointing, although cathartic, I'd like to focus on what worked, what didn't, and how we can try to avert some issues if (and when) this happens again.

To set the stage, we have been hosted at Internap in the Fisher Plaza since 2002 and in that time have only had 2 significant events that related directly to facility issues. The last issue lasted around 8 hours while this one is, by far, the most signficant downtime in the history of the web site. In total we had 29 hours of downtime. Unfortunately the 29 hours were during the geocaching peak season on the busiest weekend of the year and, to compound things, a day off from work for many. The Fates were definitely conspiring to pick the worst day to bring the site down.

What Worked

The usefulness of Twitter and Facebook became obvious for this crisis. All our web servers and email servers were all located at Fisher Plaza. We had very few options for posting updates, so we had to rely on outside systems to communicate with our community and our partners. I switched from Groundspeak emails to my Gmail account, and my iPhone running Tweetie helped me to get information out as I was "on the scene." By the end of the day I added an additional 800+ followers on Twitter which, in the past, was used as a toy for logging geocaching finds with my family and for the random Groundspeak update.

Also, although we didn't have the need for backups this time, we have daily backups of all our systems. Since this happened before our nightly backups occured it was close to the worst time for a data failure. At the most we would have lost a day of data. In a catastrophic event this isn't a total Fail. It just sucks.

What Didn't Work

Although I won't finger point at the cause of this issue, I will point out that Fisher Plaza people lacked any official communication with the first responders at the scene. Many clients of the building were in the dark, both figuratively and literally, while we were waiting outside for news of what really happened. Instead we had to join in on Twitter to figure out what happened. Was it a fire? (yes) Did the sprinklers turn on? (yes) OMG! Our machines are fried! (no. just the generator) If someone walked out of the building with some authority and told us what they knew - we could have passed that information on to our customers. Internap did a relatively good job at giving status updates though they were sparse and sometimes repeated. I'd give Internap a C and Fisher Plaza an F for communication.

I'll be just as hard on us and say that we should get an F for communication preparedness. Although I think we did a good job at working around our own issues with Facebook and Twitter (and this blog), we were unable to make updates available on our web pages and our iPhone application. The reason why some sites could do this and others could not is that our entire server infrastructure was in the Fisher Plaza basket. The other companies likely had better ways to switch over to a new location. Our only alternative, pointing DNS to another server, would have made it harder to get back online since many people would continue to point to the wrong machine when the servers were back with power. Since we only anticipated a ~12hr outage it made no sense to do something that could take another 24 hours to correct for some users.

What Next?

There are some obvious things to do to correct what didn't work, and some solutions that will require some thought. I'll highlight a couple of high level things we'll consider and implement.

We're not a bank, so although 29 hours is a long time to be down, we do not plan to duplicate our infrastructure so we are completely redundant. It is just too expensive to make fiscal sense. Instead, we'll ensure that in the case of a catastrophic event that we'll have the best backups and the best steps for restoring those backups to a new system. We already have a good system but we'll make it even better.

We'll have a better system for communicating with our customers, so these systems will be the focus for redundancy planning. This includes rerouting web servers and email. Even streaming my Twitter account on the front page of would have been helpful for letting people know what is happening.

Lastly, we're going to create an official disaster recovery plan so everyone knows what to do at Groundspeak in the situation where there is a catastrophic event. We should always understand the worst case scenario and how to recover from it. We owe this to our customers.

For those in the US, have a Happy 4th of July! And thanks to everyone for your ongoing support of Groundspeak and the geocaching activity. From the Tweets and Facebook posts you definitely enjoy geocaching. Now go out and find a cache!

Rock crawlin' in my Bronco and Geocaching with xstitcher. Life is good.
Back to top
Display posts from previous:   
Post new topic   Reply to topic   Printer Friendly Page    Forum Index —» General
Page 1 of 1
All times are GMT - 8 Hours

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

topleft topfill topright
bottomleft bottomfill bottomright

:: Copyright 2003 - 2009 BC Geocaching Association ::

RSS Feed:
Follow us on Twitter:
Interactive software released under GNU GPL, Code Credits, Privacy Policy
Theme by British Columbia Geocaching Association.