Today’s network events

 

To our customers and partners,

As you know, Intermedia experienced a significant network interruption today. As CEO of Intermedia, I want to offer a profound apology.

I also want to explain what happened, what we’ve learned, and what we’re going to change.

First, you should know that our services are now stable. There has been no data loss and there were no security breaches. And all of the emails that were sent to you during this issue will be delivered as the backlog is processed.

Here’s what we know.

Around midnight on Wednesday night, August 28th, we experienced an issue with our core routers. We implemented a fix to this issue, and we believed that it was completely resolved by 3 AM EST on Thursday.

At 7 AM EST on Tuesday, September 3, we observed some further anomalies on our core routing equipment. Our attempts at intervention were unsuccessful, and corruption in our core routing table propagated to the edge routers in all our US datacenters. This created significant packet loss between the edge and core in each datacenter and it prevented delivery of service to our customers. In addition, as our communication systems reside in the same datacenters, our ability to communicate with customers and partners was disrupted.

At 8 AM EST on Tuesday, we started restoring network connectivity by systematically rebooting each of the affected network devices. Our network and services were brought back online throughout the day, with complete network restoration by 3:30 PM EST.

As connectivity was restored to each datacenter, our system administration team focused on resolving any residual server issues caused by the network outage. The final server-related issues were resolved by approximately 6:30 PM EST. Our support team was able to take phone calls beginning around 3 PM EST.

What comes next.

As of now, all our services are back online. We now have three tasks ahead of us.

The first task is to complete our RFO (Reason For Outage) report to fully identify the root causes. We’re working with our network equipment vendor to complete this report, and we will share it with our customers and partners as soon as we can.

Second, we will take the recommendations from this RFO and make any changes necessary to improve our stability and resiliency.

Third, we will also improve the responsiveness and robustness of our customer notification tools and systems. Although we were successful in notifying many of our customers about the issues via alternate email addresses, text messages and HostPilot, not all customers were reached. Going forward, we will make more timely use of our website and social media—especially Twitter and Facebook.

I hope you accept my apology.

On behalf of all of us at Intermedia, I offer a profound apology. We understand how essential our services are to your business. We fully appreciate the breadth of our responsibility. We live and breathe it every day.

I’d appreciate any feedback you can share. Please email feedback@intermedia.net. This mailbox is monitored by Intermedia’s senior management team.

Sincerely,

Phil Koen
CEO, Intermedia

About Phil Koen, CEO of Intermedia

Phil Koen is the CEO of Intermedia.

More news from Intermedia

New video: How “My Services” empowers Intermedia’s end users

Posted on August 1, 2014

I know we focus a lot on how Intermedia products p...
Read More

7 ways that financial services firms use Intermedia products

Posted on October 30, 2014

With the specter of hacking constantly in the news...
Read More

Intermedia extends Hosted PBX to a new set of partners: ADI dealers

Posted on July 22, 2014

You've probably heard of ADI. It's North America's...
Read More