[OCIO_CUCM] FW: Trouble

[Cheryl Dearborn]

Good morning.  Below are the latest updates from CenturyLink regarding their outage.

While they are continuing to work through a complete recovery of services impacted by the outage our phone services appear to be restored. Please let us know if you experience any trouble making local calls or get reports from customers reporting trouble calling in to local extensions.  Thank you.

From: Teater, Martin <Martin.Teater at CenturyLink.com>
Sent: Friday, December 28, 2018 6:44 AM
To: Cheryl Dearborn <Cheryl.Dearborn at its.idaho.gov>
Cc: Wade Douglass <Wade.Douglass at its.idaho.gov>; Zaine Tarpo <Zaine.Tarpo at its.idaho.gov>
Subject: Re: Trouble

Good morning. Here is the latest update:


12/28/2018 13:18:47 GMT - The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Services continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. CenturyLink NOC management remains in contact with the equipment vendor to obtain updates as restoration efforts continue.


Good progress has been made. The restoration of the nodes is taking longer than anticipated. Many of the CenturyLink ticking and flow through systems have been restored. Another update will be provided again soon. Thanks for your support and patience.


Marty T

On Dec 27, 2018, at 11:51 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
Hi here is the latest update. Thanks

12/28/2018 05:54:05 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT<x-apple-data-detectors://1> remains.

0900 GMT<x-apple-data-detectors://2> would be 2:00AM Mountain time<x-apple-data-detectors://3>.

Marty T

On Dec 27, 2018, at 9:45 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:

Our engineers and technicians have identified the network element that has affected our customer services. Services are restoring, and the current estimated time for full recovery is four hours.

CenturyLink will be conducting an extensive post-incident investigation and root cause analysis to provide follow-up information to our customers impacted by this event. We will provide one more Sales Alert from this address once services are fully restored.

Marty T

On Dec 27, 2018, at 9:21 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
Latest event update:

12/28/2018 04:06:54 GMT - The Equipment Vendors Tier IV Technical Support team in conjunction with CenturyLink Tier III Technical Support continues to remotely work to remove the secondary communication channel tunnels across the network until full visibility can be restored, as well as applying the necessary polling filter to each of the reachable nodes.

Marty T

On Dec 27, 2018, at 8:34 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
Latest event comm:

12/28/2018 02:52:35 GMT - The Transport NOC has confirmed that cooperative efforts remain ongoing to remove the secondary communication channel tunnel across the network until full visibility can be restored, as well as applying the necessary filter to each of the reachable nodes. It has been confirmed that both of these actions are being performed remotely, but an estimated time to complete the activities is not available at this time.

Marty T

On Dec 27, 2018, at 5:38 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
Here is the latest update:


12/27/2018 23:56:42 GMT - The Equipment Vendor is currently reviewing the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root cause.

Marty T

On Dec 27, 2018, at 3:17 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
CenturyLink efforts to regain visibility to sites remain ongoing.
Marty T

On Dec 27, 2018, at 12:39 PM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
Additional update and details


On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs to further assist with isolation. Once visibility was restored to the site in Kansas City, MO a filter was applied to the equipment to further alleviate the impact observed. All of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership.

Marty T

On Dec 27, 2018, at 10:54 AM, Teater, Martin <Martin.Teater at centurylink.com<mailto:Martin.Teater at centurylink.com>> wrote:
Here is the latest update :

Tier IV Technical Support continues to work with all of the necessary troubleshooting teams to further alleviate the issue being experienced.  Field Operations have arrived at multiple sites including Kansas City, MO and New Orleans, LA to assist with the investigations.  Resolution efforts are currently ongoing with the necessary teams investigating equipment in Kansas City, MO and New Orleans, LA.  Field Operations have been dispatched to the site in Atlanta, GA as remote access to the equipment was been lost.

Marty T

On Dec 27, 2018, at 7:11 AM, Cheryl Dearborn <Cheryl.Dearborn at its.idaho.gov<mailto:Cheryl.Dearborn at its.idaho.gov>> wrote:
sip outage? we appear to be down. i

Get Outlook for iOS<https://aka.ms/o0ukef>
This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://admws.idaho.gov/pipermail/ocio_cucm/attachments/20181228/6b540153/attachment-0001.html>