MDE Down all morning 4/12/17

[Laketravis]

I can keep listing outages if you'd like.

AWS outage knocks Amazon, Netflix, Tinder and IMDb in MEGA data collapse

Wow. Are you following me? LOL!

Okay, apparently AWS isn't infallible either. Back to my original point that sites like Amazon and Netflix don't seem to suffer from the same amount of downtime that Disney sites do.

Now stop following me. Seriously. It's creepy.

 

[Chef Mickey]

Nah. I just let them know at the front desk of my resort. My time has value. Which is why this "minor" inconvenience was such a pita. Spring break crowds and crap tech do not mix well.

It worked out fine for me. But I'm not self centered enough to not realize that for hundreds even thousands of other guests, it may not have.

Dis forces you to rely on MDE. If MDE fails, that's a Dis problem, not a guest problem.

I never said it was a guest problem and not a Disney problem. I said technology breaks and they'll fix it. While that's happening, do other things. It is 100% their fault. Agreed our time has value and that's why you should not waste any time handling it there. Complain after the fact like I did.

If MDE experience is down, you can still do things...unlike Visa not being able to process the payment. Visa is a one trick pony.

I just don't understand the outrage that their system went down a few hours. It's a system that does some good, but the real issue is the money spent on that should have been used to improve the parks physically with restaurants, attractions, improvements, and general expansion. It being down a few hours seem super minor and predictable to me. Just roll with it and address it later.

I disagree you're totally reliant on MDE while at parks. I guarantee rides were still functioning, restaurants were still serving, and entry was still granted. Maybe slower and not ideal, but I'm sure they did their best and didn't just shrug their shoulders. There are processes in place.

 

[Chef Mickey]

Furthermore, Disney can continue operations with this system down. There are companies that literally can't work without their systems working 100% of the time and even those go down.

Recall Southwest and United had software issues that grounded flights and Amazon had web services down...no shopping on Amazon. Yahoo had servers go down. It happens. Meh.

 

[HRHPrincessAriel]

Disney has the ABSOLUTE WORST website availability of any company I have ever seen large or small.

In this day and age and with elastic cloud resource technology as well they need to plan and architect better. It's embarrassing and sad how bad it is. I don't know if the issue here is web server resources but when they have people flooding their sites for discounts or fp openings for new attractions etc they always crash and burn.

Magically pathetic

You haven't tried six flags site. I have been trying for days to purchase a ticket online and can't. When I call to try and fix it my call "can't be answered because of technical difficulties"

 

[Bandini]

You haven't tried six flags site. I have been trying for days to purchase a ticket online and can't. When I call to try and fix it my call "can't be answered because of technical difficulties"

That stinks. But at least once you get to the park, you won't be reliant on their subpar IT for ride reservations!

 

[GhostHost1000]

I do recall where website upgrades were included in that project. Would have been an excellent opportunity to address chronic downtime issues.

Website updates not server/hardware/network/infrastructure I believe

 

[GhostHost1000]

I just don't understand the outrage that their system went down a few hours.

Because this is a trend for Disney across the board with their sites and they won't fix it

 

[ford91exploder]

I can keep listing outages if you'd like.

AWS outage knocks Amazon, Netflix, Tinder and IMDb in MEGA data collapse

Heres the thing about Amazon AWS they have cheap services like S3 which provide limited redundancy and outage protection to services which specify five and six nines of availability and are priced accordingly and these services are specified to have minutes or seconds of unplanned downtime PER YEAR

Unlike Disney who has days of unplanned downtime across all systems

 

[GhostHost1000]

Heres the thing about Amazon AWS they have cheap services like S3 which provide limited redundancy and outage protection to services which specify five and six nines of availability and are priced accordingly and these services are specified to have minutes or seconds of unplanned downtime PER YEAR

Unlike Disney who has days of unplanned downtime across all systems

and what's even worse is when they KNOW their web traffic is going to large at a given time (new discounts, etc.) they don't plan for it... and things crash all around them. it's incredibly poorly managed

 

[ford91exploder]

and what's even worse is when they KNOW their web traffic is going to large at a given time (new discounts, etc.) they don't plan for it... and things crash all around them. it's incredibly poorly managed

Its all part of a mindset which devalues IT I would wager Iger does not even do email his assistant does it for him.

This leads to situations where IT is procured and managed along the same lines as Janitorial service, i.e. Cheapest vendor wins because IT is all the same some just overcharge... and having US based workers is too expensive so lets have a daylight savings time outage because where systems are managed there is no DST...

 

[ford91exploder]

Who said it wasn't enough?

I had to work for it though. Thousands of others got nothing. Tough luck on them I guess.

Disney needs to do better with their tech. Losing FP ability for 4 hours on a spring break morning is unacceptable.

Like I've repeatedly said, I came out of this fine, in part thanks to this thread, in part bc I made my FP 60 days ago, and in part bc I'm experienced here. I am not an average guest. Average guests got the shaft this morning.

Average guests get the shaft everyday at wdw

 

[ToTBellHop]

What's the question you need to have answered? When MDE goes down, the walk up kiosks still operate in a mode where you view existing ones and you can book one more. Though your app is down, the system reading it at the rides still works and is still active.

Thank goodness they removed half of the kiosks.

 

[drizgirl]

I said technology breaks and they'll fix it. While that's happening, do other things.

Yes tech breaks. But bad tech breaks more.

Because this is a trend for Disney across the board with their sites and they won't fix it

That's exactly it.

and what's even worse is when they KNOW their web traffic is going to large at a given time (new discounts, etc.) they don't plan for it... and things crash all around them. it's incredibly poorly managed

Wanna make bets on whether the system crashes when free dining is announced later this month?

 

[Chef Mickey]

Yes tech breaks. But bad tech breaks more.



That's exactly it.



Wanna make bets on whether the system crashes when free dining is announced later this month?

No one really knows their down time % other than their anecdotal experience. I have actually never experienced any down time with MDE. My guess is it's pretty minimal because it is now woven into the Disney experience and they ultimately rely on it to work for money collection.

I have only been talking about MDE in this thread and still believe a few hours of down time is not really a big deal and guests have options while they fix it. Is it "fine?" No, but it's going to happen as I mentioned with other companies experiencing down time on MUCH more critical systems to their business. It just happens. Down time doesn't automatically mean a bad system. They certainly spent plenty (too much) money on it.

Regarding their other IT like their website, I agree it's terrible. I don't think MDE is as bad as their website (Reservations, tickets, dining, etc).

 

[Bandini]

Does Disney lose money when their websites crash? They certainly don't act like it affects them.

 

[RustySpork]

No, but it's going to happen as I mentioned with other companies experiencing down time on MUCH more critical systems to their business. It just happens. Down time doesn't automatically mean a bad system. They certainly spent plenty (too much) money on it.

Other companies learn from their mistakes and implement tooling, automation, and redundancy to help reduce the pain of outages. Netflix created their own CDN, when they had the big Amazon outage it was not a total outage for Netflix because the traffic and content is distributed.

Many companies anycast their public DNS terminating traffic at multiple points of presence, so a consumer never actually sees when a site goes down.

Disney's public / consumer facing systems quite often do not meet even basic capacity requirements. How often you see stitch eat the page or MDE not work properly? Often.

Downtime does mean a bad system, every time. It may be acceptable downtime, but it's still bad even when it's planned because it impacts customers and that is never good.

Regarding their other IT like their website, I agree it's terrible. I don't think MDE is as bad as their website (Reservations, tickets, dining, etc).

MDE and their reservation/ticketing/dining websites are all served by the same service bus, the same databases, and more than likely the same front end load balancers. The only difference is more than likely that MDE traffic passes to their API servers, and their website traffic passes to their reverse proxies and content servers.

 

[HauntedPirate]

Other companies learn from their mistakes and implement tooling, automation, and redundancy to help reduce the pain of outages. Netflix created their own CDN, when they had the big Amazon outage it was not a total outage for Netflix because the traffic and content is distributed.

Many companies anycast their public DNS terminating traffic at multiple points of presence, so a consumer never actually sees when a site goes down.

Disney's public / consumer facing systems quite often do not meet even basic capacity requirements. How often you see stitch eat the page or MDE not work properly? Often.

Downtime does mean a bad system, every time. It may be acceptable downtime, but it's still bad even when it's planned because it impacts customers and that is never good.



MDE and their reservation/ticketing/dining websites are all served by the same service bus, the same databases, and more than likely the same front end load balancers. The only difference is more than likely that MDE traffic passes to their API servers, and their website traffic passes to their reverse proxies and content servers.

This post is just music to my IT ears.

I'd love to know what they use on the back-end. Having worked for an airline in the not-too-distant past, and a travel management company these days, I'm relatively familiar with having load-balanced, highly available systems. In my airline days, any downtime meant planes didn't fly, and if planes don't fly, the company loses millions of dollars each minute [I used to tell my kids that I kept the airline running, because what I was primarily responsible for, making sure everyone could log in, meant they could do their jobs and planes could continue to fly ]. While Disney is not in that same category, they have no less of a responsibility to provide near-100% up-time for their customer-facing systems. As mentioned, they market themselves as a premier product. Downtime like what happened the other day is not acceptable for such a company.

 

[Chef Mickey]

Other companies learn from their mistakes and implement tooling, automation, and redundancy to help reduce the pain of outages. Netflix created their own CDN, when they had the big Amazon outage it was not a total outage for Netflix because the traffic and content is distributed.

Many companies anycast their public DNS terminating traffic at multiple points of presence, so a consumer never actually sees when a site goes down.

Disney's public / consumer facing systems quite often do not meet even basic capacity requirements. How often you see stitch eat the page or MDE not work properly? Often.

Downtime does mean a bad system, every time. It may be acceptable downtime, but it's still bad even when it's planned because it impacts customers and that is never good.



MDE and their reservation/ticketing/dining websites are all served by the same service bus, the same databases, and more than likely the same front end load balancers. The only difference is more than likely that MDE traffic passes to their API servers, and their website traffic passes to their reverse proxies and content servers.

NFLX and DIS are not the same in their IT needs. Again, NFLX can't have their streaming service go down because the streaming service *IS* the company. Amazon is an online company. That's all they do...no alternatives. Disney exists in the physical world and existed before MDE. It's an inconvenience and shouldn't happen at a $175b company, but it's still relatively new and not as critical.

Disney needs better tech and less down time....I agree. But acting like Disney's IT needs are the same as NFLX and AMZN is just incorrect.

 

[Chef Mickey]

This post is just music to my IT ears.

I'd love to know what they use on the back-end. Having worked for an airline in the not-too-distant past, and a travel management company these days, I'm relatively familiar with having load-balanced, highly available systems. In my airline days, any downtime meant planes didn't fly, and if planes don't fly, the company loses millions of dollars each minute [I used to tell my kids that I kept the airline running, because what I was primarily responsible for, making sure everyone could log in, meant they could do their jobs and planes could continue to fly ]. While Disney is not in that same category, they have no less of a responsibility to provide near-100% up-time for their customer-facing systems. As mentioned, they market themselves as a premier product. Downtime like what happened the other day is not acceptable for such a company.

The reason it's not the same is because Disney doesn't' lose millions each minute like airlines. Their whole business is contingent upon booking systems and flight plans working flawlessly. Disney might tick off a few guests and it shouldn't happen, but they'll probably just go back to doing something else in the parks.

 

[RustySpork]

NFLX and DIS are not the same in their IT needs. Again, NFLX can't have their streaming service go down because the streaming service *IS* the company. Amazon is an online company. That's all they do...no alternatives. Disney exists in the physical world and existed before MDE. It's an inconvenience and shouldn't happen at a $175b company, but it's still relatively new and not as critical.

Disney needs better tech and less down time....I agree. But acting like Disney's IT needs are the same as NFLX and AMZN is just incorrect.

Right, that's why I specifically said that downtime may be acceptable, what I meant by that is that Disney management made a decision that the risk does not outweigh the cost. That doesn't make it OK to consumers, but it does make it OK that some consumer impact is acceptable. The fact that they're a $175b company just makes it worse that it happens at all. From a consumer perspective it means they aren't investing in the right areas.

This post is just music to my IT ears.

I'd love to know what they use on the back-end. Having worked for an airline in the not-too-distant past, and a travel management company these days, I'm relatively familiar with having load-balanced, highly available systems. In my airline days, any downtime meant planes didn't fly, and if planes don't fly, the company loses millions of dollars each minute [I used to tell my kids that I kept the airline running, because what I was primarily responsible for, making sure everyone could log in, meant they could do their jobs and planes could continue to fly ]. While Disney is not in that same category, they have no less of a responsibility to provide near-100% up-time for their customer-facing systems. As mentioned, they market themselves as a premier product. Downtime like what happened the other day is not acceptable for such a company.

They should at a minimum promise 3 9's.