System outage on October 7th 2014

On Tuesday October 7th 2014 a scheduled maintenance of our data center caused an unforeseen interference with our firewall and network infrastructure.

The technical background revolves around the Spanning Tree Protocol (STP) which is a family of protocols for Layer 2 switches. The function of these protocols is to prevent "loops" in network traffic. Such loops can create "Broadcast Storms” which make the infrastructure unusable. To prevent these Broadcast Storms the STP protocol layer disables the affected port when it detects a loop. So, STP protocols (there are many versions of them) are useful in low controlled switching networks and are not required in well controlled and seldom changed networks like ours in the data center. STP protocols can become a problem if not configured properly.

In the present case, our Ethernet ports in the data center had not been removed from the default list of "STP enabled" ports as per our requirements, and detected our network as entering in to a loop after the scheduled maintenance, and the STP protocol layer blocked the port to our network. After configuring the port correctly everything returned back to normal.

We have been able to resolve this together with our service provider and brought back all systems to normal operational status by 11:50am on October 7th 2014.