[Stategism] AGOL Outage Report

[Bill Farnsworth]

Agency GIS People,

You might have noticed that Esri had an outage in the AGOL Services on February 28th.
Below is the report from Esri about the outage for your information.
(sorry for the delay, Esri got me the report in a timely manner, but I was on vacation)
If you have any questions please let me know.

Thank you for your patience as we worked through the ArcGIS Online outage to restore all services, maps and apps.  ArcGIS Online is a resilient, well architected system and events of this type are rare - we have not had an outage like this since 2011.  At this point, we are confident that everything is back to normal and no data was lost.

Here is a summary of what happened and what's planned going forward:

We saw an outage of most arcgis.com<http://arcgis.com/> sites and services between 9:39 am and 12:34pm on February 28, 2017 with lingering issues with our Hosted Tile Services until 4:50pm. The impact was widespread as our primary www.arcgis.com<http://www.arcgis.com/> site was down causing users to be unable to use sites and applications that depend on ArcGIS Online logins and maps.

The root cause was an Amazon Web Services (AWS) S3 outage which was widely experienced across the Internet. Many of you have asked if ArcGIS Online is redundant and the answer is yes.  It is a multi-data center deployment but was impacted because the AWS outage affected the whole US-East-1 region. Because AWS uses S3 within many other services many of the services we rely on were also impacted.

Things we will be working on :
- We already have redundancy and fail over across data centers for our central web site and its content mgt and sharing api that were affected by the outage.  The underlying storage service we rely on from Amazon for this is already replicated across data centers.  We will investigate and consider additional improvements that might be made as we receive details on the outage.
- Additional communication channels such as email notification are being considered as an escalation should events like this occur in the future.  We want to ensure you get the message when communication is essential.
- We will continue to work with our partner providers on the details of the root cause of the incident so we can examine our engineering and operational processes.


Additional resources:
1.       Status information about ArcGIS Online services disruptions are updated throughout events, and also serve as a historical availability reference http://status.arcgis.com<http://status.arcgis.com/>.

2.       Additional information about the Amazon Web Services outage may be found within the Status History February 28th portion of the http://status.aws.amazon.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__status.aws.amazon.com_&d=DwMFAg&c=n6-cguzQvX_tUIrZOS_4Og&r=AZdr0XWlk6fN6767KnD5zrf5p7tTLe3E5j99KldRjIY&m=lZZAkkk4Jf__oyCkChcFktCmlh3OX2PFmDgh6MB3sfE&s=rSugs6ySBMIVDE3u-2Id0yQ3Dwo1BNyf_h9z_ZGDI3c&e=>  website.  The specific response on this from Amazon is here: https://aws.amazon.com/message/41926/<https://urldefense.proofpoint.com/v2/url?u=https-3A__aws.amazon.com_message_41926_&d=DwMFAg&c=n6-cguzQvX_tUIrZOS_4Og&r=AZdr0XWlk6fN6767KnD5zrf5p7tTLe3E5j99KldRjIY&m=lZZAkkk4Jf__oyCkChcFktCmlh3OX2PFmDgh6MB3sfE&s=WYi5AvEzmwTHhGx7bi3Wu-qrF4nYkS44qIxm7GHI5AE&e=>



Paul Ross
Sr. Product Manager, ArcGIS Online
pross at esri.com<mailto:pross at esri.com>




Cheers,
Bill

bill.farnsworth at cio.idaho.gov<mailto:bill.farnsworth at cio.idaho.gov>
208 332-1878 (w)
208 867-5007 (c)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://admws.idaho.gov/pipermail/stategism/attachments/20170315/a1c8d1ce/attachment.html>