CCBot/2.0 (http://commoncrawl.org/faq/) British Columbia Geocaching Association —» Forums —» General Caching —» General —» Groundspeak Network Down!!!
Home Newest Caches Forums Your Account About BCGA Search Blitz  
Spotted Lake
Spotted Lake:  [?] . Rapid evaporation of Spotted Lake during the summer months results in crystallized rings surrounding multicoloured pools. [M | C
Toggle Content=  Register or Login  Please Join our Association 
topleft topfill topright
Toggle Content Our Sponsors
Worldcaching.com
Worldcaching.comLandsharkz Coins and Geocaching GearGold CountryCaching Containers
bottomleft bottomfill bottomright
topleft topfill topright
Toggle Content Waypoints
bottomleft bottomfill bottomright
topleft topfill topright
Toggle Content Coming Events
bottomleft bottomfill bottomright
Forums —» General Caching —» General —» Groundspeak Network Down!!!
topleft topfill topright
LOG-IN DIFFICULTIES
Please be advised that we are currently having problems with some of our users not being able to log into the site. We apologize for this, we are working to address the situation. Thank you for your patience

Thank You
BCGA Executive

bottomleft bottomfill bottomright
toplefttopfilltopright
Groundspeak Network Down!!!
If you have questions/comments about geocaching in general or anything that doesn't fit anywhere else, post them here.
Go to page 1, 2  Next
Post new topic   Reply to topic   Printer Friendly Page    Forum Index —» General

View previous topic :: View next topic  
Author Message
Xtremeskier1978



Joined: Jul 03, 2009
Posts: 16
Location: Surrey, BC

PostPosted: Fri Jul 03, 2009 6:45 am    Post subject: Groundspeak Network Down!!! Reply with quote

Does anyone have any updates??? I heard there was a power outage at their server. I lost geocaching.com at around 11:55pm PST.
Back to top
911turbos



Joined: Sep 14, 2005
Posts: 242
Location: Anmore, BC

PostPosted: Fri Jul 03, 2009 7:21 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

From the Calgary forums:

After not being able to get to geocaching.com nor the Groundspeak forums I figured something was up. Sure enough, Jeremy posted this on his Twitter page about 20 minutes ago:

"Geocaching.com is down. Internap power outage - a fire at Fisher Plaza. Machines are fine but no power to the building."

"Geocaching.com servers are hosted at the Fisher Plaza building near the Space Needle. www.fisherplaza.com is down too"

And, as of one minute ago:

"Someone at the scene said 6 inches of water in the basement from putting out the fire. Both generators are there."

----------------------------

More updates from Mr. Irish continue to roll in...

"No ETA when the power is back on. People are hand carrying critical servers out of the building."

"OH "They said they'd be ecstatic if the power is back on by 6 tonight" - Geocaching.com likely to be down all day."

"Fire at Fisher Plaza was put out hours ago. They're now dealing with a melted generator and flood damage from putting out the fire."
Back to top
geodug



Joined: Apr 13, 2004
Posts: 161

PostPosted: Fri Jul 03, 2009 8:00 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

911turbos wrote:

"Geocaching.com is down. Internap power outage - a fire at Fisher Plaza. Machines are fine but no power to the building."

"Geocaching.com servers are hosted at the Fisher Plaza building near the Space Needle. www.fisherplaza.com is down too"



"No ETA when the power is back on. People are hand carrying critical servers out of the building."

"OH "They said they'd be ecstatic if the power is back on by 6 tonight" - Geocaching.com likely to be down all day."

"Fire at Fisher Plaza was put out hours ago. They're now dealing with a melted generator and flood damage from putting out the fire."

<sarcasm> I guess their mirror site is in the same building </sarcasm>

How can we get through the weekend without our PQs and other info?
Back to top
_canadianbacon_



Joined: Jan 05, 2004
Posts: 1154
Location: Surrey, BC, Canada

PostPosted: Fri Jul 03, 2009 8:06 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

geodug wrote:


<sarcasm> I guess their mirror site is in the same building </sarcasm>

How can we get through the weekend without our PQs and other info?

Geez, I guess that sarcasm tag is not WC3 compliant yet!

I was thinking the same thing, even said so to Mrs. Turbo in an email.

1) No emergency power
2) No redundant back-up site

Guess we know where our fees for membership and tb tabs is NOT going!
Considering it's an internet based business, pretty bad.

If it was a shoe shop, I'd not care so much.

Saw a comment somewhere else about "Carrying out servers".... Then what?
Plug them into your Home Cable?

_________________
~~~~ _CanadianBacon_~~~~
Back to top
doingitoldschool



Joined: Jan 07, 2008
Posts: 176

PostPosted: Fri Jul 03, 2009 8:18 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Once again, I'm impressed that I can come to these forums to find out what's going on when Groundspeak has problems. Thanks for the updates!


Last edited by doingitoldschool on Fri Jul 03, 2009 10:51 am; edited 1 time in total
Back to top
geodug



Joined: Apr 13, 2004
Posts: 161

PostPosted: Fri Jul 03, 2009 9:17 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

If it makes anyone feel better, there is another company affected by the fire that is in worse trouble.

Authorize.net claims to be the Internet's largest credit card processing company with over 200,000 merchants around the world processing their payments through them.

en.wikipedia.org/wiki/Authorize.net

All these merchants are now close to 12 hours down with no prospects for an early fix. It sounds like authorize.net did not have a redundant location either.

I don't get it. This kind of backup is not so expensive anymore. To not have this kind of disaster plan makes you wonder who is in charge.

Authorize.net clients are well beyond upset about this.
Back to top
LordEd



Joined: Feb 03, 2008
Posts: 68

PostPosted: Fri Jul 03, 2009 9:21 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Quote::

1) No emergency power
2) No redundant back-up site

Guess we know where our fees for membership and tb tabs is NOT going!
Considering it's an internet based business, pretty bad.

There are only so many situations you can cover when planning for disaster recovery. "Building Burned Down" is one of those things that can happen.

Lets say your shoe store is in the building next door to groundspeak and also has no power. Should they have a second store in case the first one is in a crisis? Would you blame them for being closed for a day?

Quote::
geodug: This kind of backup is not so expensive anymore.
Care to provide a price on an off-site data center with a high-end database server capable of storing and querying a significant amount of spatial data, photos, and serving up to a large community?

I would say their techs are having a rough Friday today and are in for a long weekend. Give them some time to address the problem.
Back to top
geodug



Joined: Apr 13, 2004
Posts: 161

PostPosted: Fri Jul 03, 2009 9:36 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

LordEd wrote:
Quote::

1) No emergency power
2) No redundant back-up site

Guess we know where our fees for membership and tb tabs is NOT going!
Considering it's an internet based business, pretty bad.

There are only so many situations you can cover when planning for disaster recovery. "Building Burned Down" is one of those things that can happen.

Lets say your shoe store is in the building next door to groundspeak and also has no power. Should they have a second store in case the first one is in a crisis? Would you blame them for being closed for a day?

Quote::
geodug: This kind of backup is not so expensive anymore.
Care to provide a price on an off-site data center with a high-end database server capable of storing and querying a significant amount of spatial data, photos, and serving up to a large community?

I would say their techs are having a rough Friday today and are in for a long weekend. Give them some time to address the problem.

I would truly expect a company that runs a critical application like authorize.net to have a redundant plan. That they do not will cost them customers and revenue after the smoke clears on this.

As far as geocaching.com goes, it might be debatable whether inconveniencing 100,000 members access to vital information just before a weekend is a big deal or not.

The real question is the price tag you need to pay to keep your customers happy. If this downtime extends into the weekend, I'm guessing the geocaching community will clearly show they were expecting better.
Back to top
_canadianbacon_



Joined: Jan 05, 2004
Posts: 1154
Location: Surrey, BC, Canada

PostPosted: Fri Jul 03, 2009 9:45 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

1) I'm sure there is an other shoe store 2 doors down, matter of fact I'm sure there are several shoe stores within several blocks. Not sure where I'd go for my latest pocket query though?

2) When almost your entire business relies on the internet to function, wouldn't you have a plan in case 'the building burned down?'.

Probably there is an business continuity plan that involves restoring the service at a backup location, and I agree that what probably is a 1 day outage is not the event that will trigger recovery at a backup-facility.

I have worked on large internet based B2B sites that have near real time off site redundancy, but their income was reliant on selling their products 24 hours a day, a somewhat different situation.

Delaying a few memberships or TB tag sales a day or two, possibly a bit in lost advertising revenue would not impact Groundspeaks bottom line too much, so a 1-2 day outage probably would not result in a site-move. A 'burned to the ground' situation is very different, I'd be surprised if their hosting company did NOT have something in place for that situation.

All in all it's an interesting situation, and probably has Groundspeak's Managers adding a 'todo' item about 'Hosting Failure Review' on their list of things to do. This mid-level 'disaster' might be a good thing if it triggers some thinking about the issue.

_________________
~~~~ _CanadianBacon_~~~~
Back to top
Teddy2k



Joined: Sep 12, 2008
Posts: 358
Location: Rosedale, BC

PostPosted: Fri Jul 03, 2009 9:46 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

geodug wrote:
If this downtime extends into the weekend, I'm guessing the geocaching community will clearly show they were expecting better.

The Groundspeak forums are littered with posts complaining when they take the site down for 30 minutes to do updates, let alone this long. Groundspeak will be taking some heat for this one, guaranteed!

Personally, it doesn't bother me.... $hit happens. I'm just lucky to have had a few PQ's running for that past few days, so I am all ready for caching to and from the Gold Rush events. There are, however, going to be more than a few cachers that will have waited until today to run their PQs and now may or may not be able to cache to and from the events.


Last edited by Teddy2k on Fri Jul 03, 2009 9:48 am; edited 1 time in total
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 9:47 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Interesting related report.

isc.sans.org/diary.htm...d=6721&rss


Given that Groundspeak probably (pure guessing) is only renting collocation space versus running their own dedicated data-centre due to the massive cost difference. Power, Environmental Control, and Network redundancies are what they are paying for (or should be).

I have been through a similar issue myself when I worked for a data-centre in Vancouver and the power room in the building literally blew up. UPS only lasted so long. We ran on emergency generators for 5 days.

Still, multiple locations / mirror site is the answer to this issue. Once again this leads to increased costs.
Back to top
anakerose



Joined: May 26, 2007
Posts: 107
Location: Kelowna

PostPosted: Fri Jul 03, 2009 10:06 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

I'm one of those unfortunately Sad
I knew I should have done this a few days ago! I hope the event organizers have the GPX files for the new caches. We're bringing our laptop. It'll just make it easier than going every which way wasting time trying to figure out what order to do the caches in.

Teddy2k wrote:
geodug wrote:
If this downtime extends into the weekend, I'm guessing the geocaching community will clearly show they were expecting better.

The Groundspeak forums are littered with posts complaining when they take the site down for 30 minutes to do updates, let alone this long. Groundspeak will be taking some heat for this one, guaranteed!

Personally, it doesn't bother me.... $hit happens. I'm just lucky to have had a few PQ's running for that past few days, so I am all ready for caching to and from the Gold Rush events. There are, however, going to be more than a few cachers that will have waited until today to run their PQs and now may or may not be able to cache to and from the events.
Back to top
Xtremeskier1978



Joined: Jul 03, 2009
Posts: 16
Location: Surrey, BC

PostPosted: Fri Jul 03, 2009 10:11 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Organizers for the Gold Rush...at least in Lytton...have the files loaded on a flash drive. Anyone with a laptop will be able to load those in via this method. That was the last thing I remember reading on the cache event page before the outage.
Back to top
Teddy2k



Joined: Sep 12, 2008
Posts: 358
Location: Rosedale, BC

PostPosted: Fri Jul 03, 2009 10:15 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

They also posted that on the Merritt event page, so I assume all locations will have it, although I'm sure it will only be the Gold Rush caches and nothing else.
Back to top
_canadianbacon_



Joined: Jan 05, 2004
Posts: 1154
Location: Surrey, BC, Canada

PostPosted: Fri Jul 03, 2009 10:18 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Xtremeskier1978 wrote:
Organizers for the Gold Rush...at least in Lytton...have the files loaded on a flash drive. Anyone with a laptop will be able to load those in via this method. That was the last thing I remember reading on the cache event page before the outage.

If reading into GSAK make sure your import does NOT have 'delete file after loading' set to YES!!!!

_________________
~~~~ _CanadianBacon_~~~~
Back to top
gotlost001



Joined: Nov 06, 2006
Posts: 95
Location: Abbotsford

PostPosted: Fri Jul 03, 2009 10:50 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

I have most of the caches from Chilliwack to Cache creek then from Cache creek to Squamish and Squamish to Abbotsford in GSAK. If people are at the Chilliwack kick off or Lytton I can load them up. Only because Groundspeak is down. Just find me and ask!
Back to top
grafinator



Joined: Nov 12, 2006
Posts: 195
Location: Coruscant

PostPosted: Fri Jul 03, 2009 11:46 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Poor american cachers not being able to geocache on a long weekend.
I took the day off work today to geocache, that was bad enough.

I guess they will probably come up with a back-up plan after this.

_________________
I find your lack of faith disturbing. - Darth Vader
Back to top
LordEd



Joined: Feb 03, 2008
Posts: 68

PostPosted: Fri Jul 03, 2009 11:49 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

twitpic.com/966ee
Looks like they're bringing in a generator
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 11:50 am    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Here is the generator arriving

twitpic.com/966ee

Here is Jeremy Irish's Twitter page where he wrote:

Elias cuts his vacation short to fly to Seattle to help with the #fisherfire. Hopefully servers are up when he arrives ~3pm

When power comes back the servers will boot automagically. I'll follow up to check the machines. 7 minute drive away.26 minutes ago from TwitterFox
Back to top
Cyclepath_Cacher



Joined: Mar 31, 2008
Posts: 41
Location: Hope B.C.

PostPosted: Fri Jul 03, 2009 12:00 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

I sure hope they get this fixed up. I'll keep checking here for updates. I was hoping to load up/ print off some caches for the weekend. It's been a bad week for crashes. First my Colorado, then my BB and now this. Smack
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 12:54 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

More from Jeremy Irish

I posted a blog entry for my expriences during the #fisherfire #geocaching tinyurl.com/kntjcn


Also

Temp repair is "on schedule to be completed by late this evening" for #fisherfire East. We're in Fisher Plaza East #geocaching2 minutes ago from TwitterFox
Back to top
anakerose



Joined: May 26, 2007
Posts: 107
Location: Kelowna

PostPosted: Fri Jul 03, 2009 1:29 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

yay so there's hope. Beanie

I'm sure they're doing everything they can to get gc.com back up and running asap.
Back to top
geodug



Joined: Apr 13, 2004
Posts: 161

PostPosted: Fri Jul 03, 2009 4:37 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

As we wait for geocaching.com to come back up, there is a success story coming out of the same fire. Redfin real estate was knocked out of service too but was up and running 4 hours later.

From the story it sounds as if Redfin built redundancy in another building. It doesn't sound like it was across the country which would be my preference but this time it saved them.

While they say they didn't "spend too much money", I think they mean this in the context of an enterprise cost as opposed to Aunt Gladys' monthly budget for corn flakes.

Hopefully, a lesson learned for all concerned including Groundspeak.

www.techflash.com/vent...91132.html
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 5:10 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Latest as of 6:10 pm PST

@locuslingua Will Groundspeak be considering a standby secondary site? Lightning does seem to strike twice in that building.19 minutes ago from web in reply to locuslingua

@KnightOrc We'll consider a bare minimum scenario if we have another catastrophic event, but we can't afford 2 datacenters.10 minutes ago from TwitterFox

repairs done late this evening means within the next 4 hours from this tweet.8 minutes ago from TwitterFox

@rockhawk says 2nd generator is on site and thinks the early evening forecast (~2hrs) is spot on for power4 minutes ago from TwitterFox
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 6:04 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

As of 7:04 pm PDT

RT @bluegeckodba The ETA for power to be online is 19:30 PDT. Power loads for cooling will be carefully restored in stages. #fisherfire5 minutes ago from TwitterFox
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 6:12 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

< 30min is confirmed for power at Fisher Plaza. Elias will be on site to check servers when they come online. #fisherfire4 minutes ago from TwitterFox
Back to top
RachelR0cks



Joined: Jul 04, 2009
Posts: 5

PostPosted: Fri Jul 03, 2009 6:19 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Wish there was some way to garner info on the subject...want to cache my weekend away but have none recorded right now... if you hear anything more please let us know!! thanks RachelR0cks.

refresh, sigh, refresh, sigh, refresh, sigh....
Back to top
RachelR0cks



Joined: Jul 04, 2009
Posts: 5

PostPosted: Fri Jul 03, 2009 6:22 pm    Post subject: Reply with quote

Smack


Last edited by RachelR0cks on Fri Jul 03, 2009 6:31 pm; edited 1 time in total
Back to top
Rokman



Joined: Aug 15, 2008
Posts: 2
Location: Kamloops BC V2B 2V4

PostPosted: Fri Jul 03, 2009 6:29 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Bejeweled on facebook was also down due to a fire in Seattle, no specifics.
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 6:31 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Rokman wrote:
Bejeweled on facebook was also down due to a fire in Seattle, no specifics.


I read somewhere that popcap games had systems in the building.
Back to top
scruffster



Joined: Sep 29, 2004
Posts: 1207
Location: Burnaby

PostPosted: Fri Jul 03, 2009 6:50 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

KOMO TV 4 out of Seattle was doing their newscast from outside. They are in the building too.

S.
Back to top
RachelR0cks



Joined: Jul 04, 2009
Posts: 5

PostPosted: Fri Jul 03, 2009 6:58 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Thanks fo the update Scruffster!
Back to top
vanislelady



Joined: Jan 20, 2007
Posts: 230
Location: Campbell River

PostPosted: Fri Jul 03, 2009 7:03 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

yes popcap games was affected..looks like a lot of affected networks...
Back to top
RachelR0cks



Joined: Jul 04, 2009
Posts: 5

PostPosted: Fri Jul 03, 2009 7:22 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Popcap and authorize.net are both up and running...
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 7:24 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

From Jeremy Irish

twitter.com/locuslingua

Elias is hanging out with others in the Fisher Plaza lobby waiting for power #fisherfire10 minutes ago from TwitterFox
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 7:50 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

More from Jeremy Irish

Internap still doesn't feel comfortable with an ETA but my guess is we'll be online by 10:30pm (1 1/2 hours) #fisherfire 3 minutes ago from TwitterFox
Back to top
RachelR0cks



Joined: Jul 04, 2009
Posts: 5

PostPosted: Fri Jul 03, 2009 8:12 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Right on Knightorc, thanks for keeping us posted!
Back to top
SOLONM



Joined: Apr 12, 2008
Posts: 125

PostPosted: Fri Jul 03, 2009 8:22 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Wow!! Reality bites…….get a grip folks, as to how geocaching began with a can of beans…take a beat & think about it. This not a perfect world & stuff happens….did any of you have a house burn down & lose everything? That is a reality check!! Having the game messed up is sad but not nearly as sad as losing your real world. I’m sure those folks at Groundspeak are traumatized. Lets not be giving them more grief than they need just because their trauma is interfering with our fun……as Red Green says, “we are all in it together” LOL!!
Back to top
Philatsea



Joined: Jun 17, 2008
Posts: 118

PostPosted: Fri Jul 03, 2009 8:26 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

Let's see I leave for Europe tomorrow to Cruise and Geocache along the way. I only got my GPS back from Garmin and I need to load the caches from Europe. This isn't good..... Smack

Phil
Back to top
knightorc



Joined: Apr 18, 2009
Posts: 22
Location: Salmon Arm, BC

PostPosted: Fri Jul 03, 2009 8:34 pm    Post subject: Re: Groundspeak Network Down!!! Reply with quote

RT @rockhawk Full power restored at Fisher Plaza but data centers still not allowing servers until rooms are cooled <2-4 hours #fisherfire 26 minutes ago from Tweetie

According to Elias, @rockhawk news is bogus. Power and lights are still off and colo only slightly warm. #fisherfire 12 minutes ago from Tweetie

Elias doubts they'll require a colo cooldown at Fisher before restoring power to servers. #fisherfire 10 minutes ago from Tweetie
Back to top
Display posts from previous:   
Post new topic   Reply to topic   Printer Friendly Page    Forum Index —» General
Page 1 of 2
All times are GMT - 8 Hours
Go to page 1, 2  Next



Jump to:  


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum


bottomleftbottomright
topleft topfill topright
Advertisement
bottomleft bottomfill bottomright

:: Copyright 2003 - 2009 BC Geocaching Association ::

RSS Feed:
Follow us on Twitter: http://twitter.com/BCGeocaching
Interactive software released under GNU GPL, Code Credits, Privacy Policy
Theme by British Columbia Geocaching Association.