FYI: Disney network outage - Most systems back up!

[Figments Friend]

Thanks go out to the original poster for changing the title of this thread once the system was back up.
I came here first to check the status early this morn and in a instant saw that this had been resolved.
I was then able to book all of my FPs this morn for a upcoming trip.
So I thank you!

 

[WhatJaneSays]

Thanks go out to the original poster for changing the title of this thread once the system was back up.
I came here first to check the status early this morn and in a instant saw thugs had been resolved.
I was then able to book all of my FPs this morn for a upcoming trip.
So I thank you!

♪ ♩ You're welcome ♩ ♪

In the few news worthy things I get to post I try to do my best. Hope you got all the FPs you wanted!

 

[James J]

It seems to have gone down again for me, at least partially. The front page comes up but as soon as I try to go onto MDE or any of the options across the top (Places to Stay, Things to Do etc) I'm getting the page eaten message...

 

[TheGuyThatMakesSwords]

Three words....

Amazon Web Services.

Disney? PLEASE get serious...

 

[Tom P.]

I hear that Disney has finally moved up to Novell NetWare 3.12 and soon they'll upgrade all their workstations to MS-DOS 6.22. There was an initiative being discussed to utilize OS/2 Warp, but I'm afraid cost cutting measures killed that.

 

[Figments Friend]

♪ ♩ You're welcome ♩ ♪

In the few news worthy things I get to post I try to do my best. Hope you got all the FPs you wanted!

Yes, I did indeed.
Flight of Passage, Slinky Dog Dash, Dwarfs Mine Train, Space Mountain, 'Frostrom'......everything I was hoping to get in the four days I will in the Parks this time around.
Quite pleased.

Thank you again!

 

[aaronml]

Three words....

Amazon Web Services.

Disney? PLEASE get serious...

So it really isn’t that simple.

Among other things, several newer Disney sites / systems are actually hosted in AWS, including FastPass for DLR I believe.

But one of the key things you have to plan for when hosting stuff in the Cloud is failure. You need to build systems that are incredibly fault-tolerant (see Chaos Monkey from Netflix for example), and Disney moving everything from their current infrastructure to AWS isn’t going to magically solve their issues. In fact, it has the potential to make some things much worse.

What Disney actually needs to do is build systems with a high-level of redundancy, and ability to fail gracefully, in ways that minimize impact.

 

[spacemtnfanatic]

It's a good thing technology on the Disney network isn't vital to experiencing a day in a Disney theme park. Oh wait....

For the amount of money folks pay for Disney vacations, it is simply not acceptable for the amount of failures this stuff has.

 

[DisneyJoe]

So it really isn’t that simple.

Among other things, several newer Disney sites / systems are actually hosted in AWS, including FastPass for DLR I believe.

But one of the key things you have to plan for when hosting stuff in the Cloud is failure. You need to build systems that are incredibly fault-tolerant (see Chaos Monkey from Netflix for example), and Disney moving everything from their current infrastructure to AWS isn’t going to magically solve their issues. In fact, it has the potential to make some things much worse.

What Disney actually needs to do is build systems with a high-level of redundancy, and ability to fail gracefully, in ways that minimize impact.

High level of redundancy, graceful failover, and proper testing of their entire architecture including performance/stress testing.

IMHO they're very weak in all of those areas.

 

[RustySpork]

High level of redundancy, graceful failover, and proper testing of their entire architecture including performance/stress testing.

IMHO they're very weak in all of those areas.

Psh, everything has greatly improved since the banyan vines to lanman/token ring upgrade.



Seriously though, they're not that weak in any of those areas. They probably do have some critical legacy component, but I'd challenge you to name a single enterprise at their scale that doesn't.

 

[DisneyJoe]

Psh, everything has greatly improved since the banyan vines to lanman/token ring upgrade.



Seriously though, they're not that weak in any of those areas. They probably do have some critical legacy component, but I'd challenge you to name a single enterprise at their scale that doesn't.

When a new Disney offer is released, phone systems crash, web systems crash, and hold times can be many many hours - and this has been happening for 5-8 years or more - why?

Name one other enterprise company that has those types of issues multiple times per year for the last 5+ years.

 

[RustySpork]

When a new Disney offer is released, phone systems crash, web systems crash, and hold times can be many many hours - and this has been happening for 5-8 years or more - why?

Name one other enterprise company that has those types of issues multiple times per year for the last 5+ years.

All of them do, you just aren't paying attention when it happens to others because you have less of a vested interest. Differing operating models mask a lot of it as well, as there are few companies that have the sort of die-hard following that Disney does.

Just peruse downdetector.com, I recommend visiting some of the major sites like the Apple App Store, and Amazon.com.

You're talking about a suite of web applications and APIs that receive hundreds of thousands of hits per second, if not millions. They have a pretty amazing level of uptime for the amount of traffic their consumer facing apps see.

 

[JoeCamel]

When a new Disney offer is released, phone systems crash, web systems crash, and hold times can be many many hours - and this has been happening for 5-8 years or more - why?

Name one other enterprise company that has those types of issues multiple times per year for the last 5+ years.

Same as many companies - bean counters don't see the return from good IT so what they fund is minimal expertise and often pull the plug on useful features. The problem is endemic in for profit companies and the public systems are not immune. How many years has the VA and postal service been paying out millions for systems that don't perform the functions they were designed for.

Not limited to TDO or even TWDC. We "should" be getting secure functional systems but we end up with massive data breaches and system failures that impact a company's ability to do business. Same story everywhere in the IT game we just focus on the failings of MDE here.

 

[DisneyJoe]

All of them do, you just aren't paying attention when it happens to others because you have less of a vested interest. Differing operating models mask a lot of it as well, as there are few companies that have the sort of die-hard following that Disney does.

Just peruse downdetector.com, I recommend visiting some of the major sites like the Apple App Store, and Amazon.com.

You're talking about a suite of web applications and APIs that receive hundreds of thousands of hits per second, if not millions. They have a pretty amazing level of uptime for the amount of traffic their consumer facing apps see.

I understand, I worked in IT for 28 years.

It's just very frustrating now, when working in travel, that it sometimes takes us days to finish a task for a client that should take minutes if the systems were working properly.

 

[RustySpork]

I understand, I worked in IT for 28 years.

It's just very frustrating now, when working in travel, that it sometimes takes us days to finish a task for a client that should take minutes if the systems were working properly.

It's always frustrating when something you rely on is down, especially when it affects your ability to make money or experience the magic.

 

[esskay]

When a new Disney offer is released, phone systems crash, web systems crash, and hold times can be many many hours - and this has been happening for 5-8 years or more - why?

Name one other enterprise company that has those types of issues multiple times per year for the last 5+ years.

You clearly dont keep up with the tech industry anymore if you need to ask that question.

To name a few, in the last 5 years Amazon (who power most of the web, with AWS) have had at least 10 high profile outages in 5 years, as has Microsoft, Apple, Google, Facebook etc.

Outages WILL happen, you simply cannot avoid that. All you can do is have the right people in place to deal with it, and ensure that type of outage cant happen again.

I really dont get why people even reported on this. MDE had a big outage in May that lasted a week and nobody even noticed. During the week no table side magic band readers worked, the server tills were locked out in most restaurants (Epcot being the worst affected), and fastpass usage caches weren't refreshing automatically, meaning you were often locked into only being able to do 3 fastpasses in total on most days.

It happens, it sucks, get over it, move on and understand that things like this are not a freak occurrence and will happen from time to time.

 

[RSoxNo1]

My kids were having an issue with DisneyNow yesterday.

I deleted it yesterday. Unrelated.

 

[DisneyJoe]

You clearly dont keep up with the tech industry anymore if you need to ask that question.

To name a few, in the last 5 years Amazon (who power most of the web, with AWS) have had at least 10 high profile outages in 5 years, as has Microsoft, Apple, Google, Facebook etc.

Outages WILL happen, you simply cannot avoid that. All you can do is have the right people in place to deal with it, and ensure that type of outage cant happen again.

I really dont get why people even reported on this. MDE had a big outage in May that lasted a week and nobody even noticed. During the week no table side magic band readers worked, the server tills were locked out in most restaurants (Epcot being the worst affected), and fastpass usage caches weren't refreshing automatically, meaning you were often locked into only being able to do 3 fastpasses in total on most days.

It happens, it sucks, get over it, move on and understand that things like this are not a freak occurrence and will happen from time to time.

This most recent outage for Disney affected 30 sites/systems from MDE to shopDisney to ESPN.

 

[RustySpork]

This most recent outage for Disney affected 30 sites/systems from MDE to shopDisney to ESPN.

That along with it being reported that internal CM access to the same systems was still working leads me to believe that it could have been external. A vendor outage could have caused it. What happened when Cloud Flare went down a while back? 10% of the entire internet went down.

 

[GhostHost1000]

So it really isn’t that simple.

Among other things, several newer Disney sites / systems are actually hosted in AWS, including FastPass for DLR I believe.

But one of the key things you have to plan for when hosting stuff in the Cloud is failure. You need to build systems that are incredibly fault-tolerant (see Chaos Monkey from Netflix for example), and Disney moving everything from their current infrastructure to AWS isn’t going to magically solve their issues. In fact, it has the potential to make some things much worse.

What Disney actually needs to do is build systems with a high-level of redundancy, and ability to fail gracefully, in ways that minimize impact.

this is very true... however one benefit moving to the cloud would give them is being able to ramp up (and down) when needed especially when new discounts or things happen/open up that generally crash and take down their systems. I still cannot believe how poorly they plan (or that they don't plan at all) for this type of thing in today's world