Entrance Scanners/FastPass+ down at DHS

[dstrawn9889]

huh, IMHO someone allowed it to go Batt backup for test of cascade shutdown. and it did not shutdown properly. i'd bet it was bad battery packs reading charged in the UPS not allowing processes enough time to complete. ought to see a truck at the docks with new battery packs for the UPS system. i would think a clearing of the cache and a proper re-start and shutdown of the systems to correct.

 

[dstrawn9889]

huh, IMHO someone allowed it to go Batt backup for test of cascade shutdown. and it did not shutdown properly. i'd bet it was bad battery packs reading charged in the UPS not allowing processes enough time to complete. ought to see a truck at the docks with new battery packs for the UPS system. i would think a clearing of the cache and a proper re-start and shutdown of the systems to correct.

http://data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBwgHBgkIBwgKCgkLDRYPDQwMDRsUFRAWIB0iIiAdHx8kKDQsJCYxJx8fLT0tMTU3Ojo6Iys/RD84QzQ5OjcBCgoKDQwNGg8PGjclHyU3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3N//AABEIAG0AwgMBIgACEQEDEQH/xAAbAAACAgMBAAAAAAAAAAAAAAAEBQMGAAIHAf/EAEUQAAIBAwICBgYFCQcEAwAAAAECAwAEEQUhEjEGEyJBUWEycYGRobEUIzTB0QckQmJyc6Ky8BUmUmPC4fEzU4LSJTZE/8QAGgEAAwEBAQEAAAAAAAAAAAAAAQIDBAAFBv/EACQRAAICAgICAgMBAQAAAAAAAAABAhEDMRIhBEEiURNhcTIF/9oADAMBAAIRAxEAPwCz2Q/NLcH/ABufj/tRd3j6Q3kq/wBfGhbL7LZjyb+Y0TKc30w8OH+UV5kvZ6CEU68GpXR8XyfcBRuknMg9dD3q4vpvPJ+NEaT6QoDi9JcdOdbj/wAu2b+EirM3agIPhVTfs/lE1bHfaW5+Bq0xnMRo5diQ0Ik7L8Pg1Wazb8wkI7lBquzrw3JPic09sznT5h/ln5UIPtBnoQ9KbUT6RcLw5MRLD31yOWBwSccq7jexh0uFYdktv6jXNr2x4LqSIrjhYg+w0UqVlsUr6KyB2Rnuo2y55GBWXEDCZgFOM4qe3g7uW9TZpQzt24jhqIIIO9QW8JUjBzmmEcEjqB1bnzxSj2a2GYr23l8Jk+Yq+kbkedUv6PJHuy8OPGrhb3Cy28c2QAyA5PjVcXVmTyVpnjgqSfKlWrxoLMHfsup+OPvpm0qyA8DKx78d1DXMPX28sZ5MpHq8DTyVkI9OytzPxFGAPVw9gbc2Yjc/Ae+p47aZ5mR42VRg8bd48q30qdEi4CBx9Xx89ycV41xcyOyx8RadOKIjBGDjceoH31n4WrNjm06R5HE9vd8DRktJIWTwYZ29wx7q2vWf6ZKXTiZCFVTyC4zy896lN+9skaTKzy8faHIhd+0fcRQ120cty8ltMxMpBYcHIgAcz3V0kknR0HJtckas7yTI0gHWTNkkHlyHLwxt7KP1GFheW1yi24SBWAeaYpwuSMYAG/L40DZgz34J3C7Ljltz+Pyo7Uwq3EDTWj3UIjkTgWPjw54cZHdkZGe7202H2yfk9UiZJ44UWJ7uMsg4SWUk5HjvWVvZRTw2cEUjOXSNVY57wK9q9mUOsxiKzH6mfexNSvvqNx+0B/CK0tv/AMo5fUr+Neqc6jc+Uo/lWll7GQvvxw3TZ8Cfia30r0hWupfaPPg++ttJ9JfXSsb0KLw8H5RL/wA7GD76s8Ldj2VVtUPD+UK68TYQ/NqsUDdj2UcwsEDXW8mfOnGmqWtZABnKHb2Uml7Untp9pQ+qI5ZXFLDaDPRRuk3SHUR0lt+jmkwqstx1Uklw2+EIycDyAO9E63YpHfzOBnrAGHrPOibiPTLTpB/bN7LGswtI4YyX3XY8W3uoDWOlWhPIrtO8mFwAuB7fjWtqPDrYuFyU7ehedOjYjjbC8yMZzUtvp1oj8Qj4v2jmls/TTSoiWhti5/Xf8MUruOnsjti2iijH6q71H8dmv8qLwkKKBwRAD9Vak6xY/SKoB4mqAOkGrXfORkB78VIkU1yPziV38i1DiHmWy71qxizxzBvJd6TXHSqNVMVoy8OeWe/1VtYdG5pgH+jsq7HJU71rc9EoTfwGNlXiOHBXB9lCvsWT6sY9DnvL27ku5ZG6lFKc9mY/hVvx51BY28VpbR21rB1cSDZc8/Op2dQpYDOATjlRSozSlyZXb7TxFdqYnGdyFGS2OeCO8eHKvEa7tFSS6jQLGpSMk7tn9X2V7b3TpEs7hgS/WTN3MuDjHvGBU0LIX+nXhHGezEnPhHgB40josr9iwyFmBmdmZmCcRHMgnYnx3qZ0KWryAbcaxnfB3ONqYRyWtzJJC1u0byEMRIpUtjvH+1BzAJBJbOcSxS8aZ3D7kg/P1VNwV2UWR6qjIJxaMZFjLYXIRaLmuuvihlJkREmxIIyfRweeNyM4pYvHJMIeBhJ4FdseOeWKOuXls7OHqJuDLFGKgEMzAhf4sUuPknQfI4NWtkdtp0n0eLNu/oD02PFy7/Ospe/Sa2Rik0kyyqcOBIRg9+1ZWm39GCv2XO33miXwiQfwitUb/wCRu8/94/IVtBtdqP1VHwFDK+L+7/fH7q56ZReiLUc/Sjn/AAkfE1vpPNd++o9TJMqty2YfGt9J/Q9dTG9CfWjw/lBkPjp0J/iantq/1dI+kIx07z46bH/O1Nrcncd1PNWKiVFLTZp9puwApNHsRimdrNwKu/a8KSHTOno510r6I3F7eS3kd4eFlVhCd8eIHuJ9tUi96Pz9c6qxHb+Hn511PXbx42YFiqvtgeHI1WblkeLrWHCQOwiLufAZrWugdso8PRq5nmSKJWkdic4/RHjVt0PoFHHiTUbrHfwxeHmT91H9FE613fYlcq2B355Uy1AXwJlijkZRsArYx+NByb6KwgkrGum6H0eswvHbo+B6UpLfOrBaQ6XFhra3t8folEWufXWj39/gutyjgAlut2IzyB5GrF0esZ7O2ntpZDI6plSTXSTiFfIfX2oW6dmSQA+Gd6qXSWVY4obnOEEoyQdxkGgLno7emdbq8mEsjtkoX7EQ8cZGaguLOZNLlN4I+FmGApyNjtSyj1bOuuhnpPSSG5mCKSVA5d5ppDcTXdrLJAirKchBJ6Oe7NVHSrZLWR5YQqcR32z7qtungrbZVuy3axUxGhWtpLFJHp0kokj4FcgEjhbfb1d/soxprW2lH0iZeuAwAeY9QFCmR5Lu8mjH1iq3APMDA+Vb6fNa8EUcXaldck4ySduZqXJWVafEIDNdX0U4jZIokYAyDBbPl4bVFAhu5C17Dnq8gK64G5z7eVESypEgkZgR6+dCPfylyUjCjuB3pZSS2GEJS0iTU7vqz1MTDYdvHMd+K9uJra1sEe6jfqkKsAg5EdofKlMrmS6lLjDSdoe4CnMywT6ejzxNKo4SEDYyTtj48q6EuUg5YcMaIjb2ExMv9lF+PtcXEvaz386ymFvY2kkEbxx4RkBUcZ2GNqyrGQPi+3t5ED4UEPt11+/ajoft8v7ZoFftl3+/ai9SCto01dj1sf7P4VtpX6I8601b04D3lCPcf9620w44fXSIcW9Ih/faM+Omr/OaYwnApf0j/wDuUB8dO/1mjoNwKf7FWkGxZJFMIY1kXhI4iRtvyoGBaPtiOMCkjs6RVekkeIMsDlCfWap13O0tu0keCE7Lrj+u6r30ilWKMtKAyBmyPaa4xrWq3CatPNbNwqxwVxlSO6tKjyBz49s6H0OAS0fJGC+fX51c7NElyku6tzGa5r0GmZrYzNKx4zjqxyWrxbXTREFdxStUXi010N10eyt2MgllfG4V5OyK8spQb9R+i+xFK5bqS7kw2ViHcObH8KWnpQlndrDq9lJbshzHMm6SD1/jXdjUWJ7TTrido7oN16tnAkK5Gcdx3FL+lKwxacVCjhyAoHrpVqGqvqMyvp8fC8QLoTsT4+zFR6reXDwAXaActs53pWc10FWFghgUMw4hhsU0jAjYlWCrj0RyFDaZ27WNzsMbGjzGvCM4w3fSrRBvsR6NIJJhI3NxxH1net4E+j6j9dGEWVnKYOy7j8fjQunkQmIrklQOVNJ3jN0hZ1Cwg7+LHkPcM1CHdo05Li017QoaM2100RXbJZGznsk7fD5V7BdOEd2KK5DJxkbpvjIHjip9WH5ykwOVeMAHu2J/9hUETBcMscRfPNowSKlOlM0Y7ljCNSF08pM6jgyDE23huPHJ350RGyjTJetjWVYwTwMPSPMD18qwSyXVpPHMUMigshHeRuPjiptJRZYnVkDIxGxGc1RO2mjPPqDi/QvGrzqOEx2S424SWGPLGNqymUcEYjUG2OcDP1VZWijIN7Q8V2/7ZoAH89vB/nt91G2P2hj4sT8aCAzfXf78/IV3pnezNViLQxTgjsEoR68fhWabyU+dE6iM6RMB3AN7iKF07kn9d9CvjYyYD0iH97LVvGwP89Fw7YofpCP70WJPI2Tj+KiI/RU0VtgekMITtmi7TtTAGgYmwKNsf+sD4mk9nPQt16y6+xuWCcQQsd9+8/7VyU2Canq8tkIQrgEEZ3AHMnPuzXZtQvrSztr5bqYJxB1QDcsTywK57q+q3ev69p2m2VskUFxcRiRExxSKGBYuR3YB28q9Dx8EskXL0icm6BX0qTow6Rgs0ZCs2+SvEAy588HFPtP1CKTDZ2q69MOjv0+x+kQxcbovDIgG7L3H1iuV9VLp0/AxymdmxSTjTKY8iaTRaZ7fUbj6yzvbeJRskU0Zwf8AyB291RTaRqlxEEuWs2HPLKOz4YOSfhRWmj6dagRug2wfOpF0WeRiJLqQJ/hVzip+jSqANF0tdJM19eTNO6qQHLdjnk4Xl3c+dIdT1FtRumWM9lDxOfEnu9lWHpnFPD0cu1s2AS1gMkjEe5fX3+w1RtCurWa3URbSndkJyc/fSyhJxtEpzV8TpOgjrNLiJ3xtU+pQTPZlLcdsHiAB3OO4edV7SdeNhamARB2zlSTsKbab0iW9uhbyxBGPIg7GkVURt3Yq6h0kgkw8CDK/WIVBGPPmeVY86uGhXIRHBwRuT4576fatFm+VrpGa0kg6kuu/VEnn8t/Kq/Lbv1mJiVmjPC4Xk2Pu7/Hes2SHDR6GLJ+TZJJPxQpC6bFgyP55xgez51q0L2rjjU8Eu6ADdccx5ePtplFqCQwBI4QpHj6PD3mgEUAzOZCSm8b42PEw4ifMEjPqruKa6YFJxeqPVmSORS+fIj4b00s1MWltg4JjyDnh7tjnupdNBELTghTgij7MZIwZJO4/15+VPkiEVoBIMgJuM89uVdCNMTPNuJErQqoU3IGBjnWVvCkaxIqq5UKACQCce+sq9mOgnTz9aT/XOgx9vvP35+QouwPboQfbrz98f5Vpl/lhew6YcVjKP1D8qX6S3FFEx58j76YMfzOT9ml+ngLGncOI/M0a+By2Q9JFxrenP3fR5B8jW8Z+rGKW67rthc3kUlu5f6OGQkjAJ8qrF/0jmmcxpJwRj/BtW7B/zs2R2+kF6LTc9J9PtHeFWaeZCVZIyBg+G5FBSdKtQYcUCw2ychkhm9pPyxVA1CI3xEkMRz6TSFse3NDYWzz1tw00o2C52Xx/o+6vQh4OHFLVk5ZGnosmpapI0rSSzGSY7szHOacfkzikN/PrZPEYX6mMN35GW9WxHvNc6Wd5HJdixzzJrqfRq2v26KWUuj/RpURSZYWBRy+SW7WT8QKp5eTjjpC8uR02DpHaIFW4V4iCASvaUe7f4Uj6U9GLXVYGvtKMcinJdYjnB8R+FUWW400XHG9zNo2qncpeEiOQ+v0SPbVj6M6tc2cxlnK24VcMgcFHHiCOfLY591eQ3y6YFHi7iU6T+09BuGMUcksOcZRSceG45H100s+l11ciO3lhZGbYsRvj1A1Z9V16CS2+j3EtusE0peRIoQCSTkBt8Dcjc4oC2kSGTittDu1zyk4YYyfbxZNT4IqssvRPr9us/RPULSxV55Z7dsmTsFiR59+w2rjVlZRIysXdnG4YbAV3a0kaf6prV4etUjDt2uXhXFdVhOlaxc2k2FaKVgFz3Z2+GK9DwuLbTE6u2NraZmISZhg/pY5eumtgnBewl2AXiHbHKq0t2kNuJZQSnf8Aq1tB0ktk7Kni8sVTyfDwZHa6Y/KPtnYOsW4gC9YCAwbIPPBzSrXIsNHdKMD/AKbj5H7vbSvolfrNZmUNsx9BjuPfVjljW5t5YjsHXY+B7j78V42bFKLqSHg3BplckbsAY4UP6RHo9+/wFS2MVtLcyR3Lu8MUTFYxnfGC2fhRWkvGXZJUBDx9/I+NCANCouRII+JQIncemoLdknxwR68VkhHo2ZJW+IdYi0htluAk8shfhijkIYgnuXy86ZwXLTQyK9uUlUdqNtwcjI9YpbpzwszXbMoUHsxouAM9+PE4plZZaKa4kDAzHOM7qoGBTxM+Q3R7kIoNvggeOPvrKhintuqTMiMeEZJh3NZVTOS6e44h50MD+fXv7/8A0rXlg546wfbr399/pWuj/ljPYax/M5P2TVf1ec22hXLqcMUKg+s4p+/2OT9k1TOl0rLoiINgzEn2A1p8SHOcV+zvZSrS6LQS5Po5JqLiS3j66bJZ/RGOdCIxW0u8eC/OvdROY4G5HB+6vpZP0Rb6PLjUp5eyrcCg/o8/f3eyh4rdny5Ox8e+trWMYLt2sd1ayTO5GTgZwFHKlUUu2T3s8XhDcKc66l+SPV0MNzpUmA8bdYh8VPP3GuawwqIgRzPfTzoCkj9MLSOKYxl2IZgM5GDkfAVn8mF42FHaJrW1ROtl4AqEuzvgD10jOkp0sKX0d0BZOCqFQeJhnngjbbljuNMrnS7GWcSX8LXgByscz5jB8eDkT681N1ptbqJ4gAk2FaMDbwBHsx7q8ehyt3XR220i4EljJNLPhkYXEnGHXbsYO2MHb/jB+mXqwZit7dgp5xq+VB9R5U91aBJo5FfvUHIpZZJBeQI88CmTHprs3vHqoBT6B9e1g6Dot5qjwqkwjIj4myXc+iPLeuEpPLdvJJcsXeVyzue9ick+81fPyw3UwuNO0wOfo7KZTnclhtVJeJRDttw7CvQ8LHuQjfyN4ZmtyFce/vooQK2JLVQD3oNseqhIvroir81GxrIWeMgo2MVvr2MhpZX80BzG7Ajuq3aZ0smQKJ8Mo25DIqlviS2W55OcZHjn/ipLRzwv5DNCcIZY1JFUy+Wk7hBcQgScKl+IcsAd9MtJvljtXjk4iE3XhHieQ9pGKWdB1EulyGXtAuUweWDzotI1tlmVBnq7hFBPhxgV8t5GF4czivs184zTtBk0Mlrcx3kijEr/AFiruEbu+/fx9dM4uqQEpjDHJweeah1J+GwfKhhjO/kM/dSzTo43uoJEjWPCty5kjvJ9VSfxf9ES5x/gfFJZCJB9R6I7lH3V7UJ0C3Jz1sm/nXlVMtn/2Q==
stop... reverse it...

 

[asianway]

Twitter is blowing up, to no one's surprise.

Word from @lentesta: Word is that FP+, entrance turnstile systems didn't restart correctly after a planned power test last night at DHS. ^len

Handing out paper guest recovery FPs as a stop-gap measure, apparently. More to come, I'm sure, as well as the 673rd argument on MM+.

Isn't that how Chernobyl happened?

 

[matt9112]

Isn't that how Chernobyl happened?

yes and no....there are literary dozens of small things that go wrong in Chernobyl. (most notable is the shutdown of SMRT)

but yes a scheduled shutdown was the initial plan.

 

[yellowstone7123]

huh, IMHO someone allowed it to go Batt backup for test of cascade shutdown. and it did not shutdown properly. i'd bet it was bad battery packs reading charged in the UPS not allowing processes enough time to complete. ought to see a truck at the docks with new battery packs for the UPS system. i would think a clearing of the cache and a proper re-start and shutdown of the systems to correct.

I would hope that Disney regularly stress and load tests those batteries, if not, what a stupid mistake.

 

[Nubs70]

My 180 day window opened last night 8/21. I could not schedule any ADR's ( on mobile or laptop) as the last open day was stuck on 8/19. This morning the same issue still existed. This afternoon the issue was resolved.

Additionally, I attempted to link my reservation to MGE by the phone app but my reservation could not be found. I then used the laptop and was able to link. Checked the phone app and reservation was linked

Did they need to do a reset MM+?

 

[-em]

Was at DHS for a bit today (Saturday) and turnstiles were still down which caused a pretty nice backlog as they had Guest Relations CMs having to clear everyone one at a time while the main entrance cast stood there bored...

fun times

 

[matt9112]

f

Was at DHS for a bit today (Saturday) and turnstiles were still down which caused a pretty nice backlog as they had Guest Relations CMs having to clear everyone one at a time while the main entrance cast stood there bored...

fun times

fast times.....richmond high?

 

[tissandtully]

Was at DHS for a bit today (Saturday) and turnstiles were still down which caused a pretty nice backlog as they had Guest Relations CMs having to clear everyone one at a time while the main entrance cast stood there bored...

fun times

Wait, they are STILL down??

 

[Olijimcha]

huh, IMHO someone allowed it to go Batt backup for test of cascade shutdown. and it did not shutdown properly. i'd bet it was bad battery packs reading charged in the UPS not allowing processes enough time to complete. ought to see a truck at the docks with new battery packs for the UPS system. i would think a clearing of the cache and a proper re-start and shutdown of the systems to correct.

Interesting, do you know what the UPS configuration is there? I would have expected N+1 at least.

 

[ford91exploder]

Where I work, the generators are offline until the power fails. Then they come up automatically, and critical systems are protected by UPS during the gap. At least that it how it is supposed to work but because failures happen in the real world, things get tested periodically.

This would require IT competence which Disney does not have.

 

[ford91exploder]

Interesting, do you know what the UPS configuration is there? I would have expected N+1 at least.

N+1 costs extra MONEY, I'd be surprised if DHS even HAS a single UPS protecting the gear

 

[ford91exploder]

Most competent organizations have a weekly UPS/Generator test - Mains are shut down and load transfers to UPS which then runs until generator is up to speed in about 60 seconds,

Interesting things happen during these tests, One site found the electrician mislabeled the Mains/Protected power outlets in a new set of racks... Ooopsie when all the servers crashed at once...

 

[ford91exploder]

The President will hold a press conference at 2...

In all seriousness though GrumpyFan is right. Over the summer we get so many storms and so much wind that the power flickers all the time. Disney has their own grid, but it's subject to the same enviroment. It's better they test it now and find out what may or may not happen with the hurricane season approaching. This way they have 3+ months to fix it (if they can).

That said, if it's as bad as people are saying, this really shouldn't have happened.

Agree even in my home office where I have a pair of 5000KVA units running office systems I have a weekly test where load is transferred to battery, This is simply a unacceptable level of performance for a major theme park.

 

[some guy]

I found leaked info about their backup plan for a power outage!