URL Shortener and SMS services outage on June 9th, 2016

Overview

We experienced a service disruption on Thursday, June 9th, 2016, which affected URL Shortener and SMS services in VoC Feedback. During this incident the shortened links were not accessible by end-users across all VoC Hubs. Additionally, certain customers' SMS channel was affected: it wasn't possible to answer surveys or to receive questions via SMS channel during the period of outage.

The period of disruption began at ~13:40 CET on June 9th, 2016 and lasted until ~17:50 CET the same day.

Cause

The cause of the service disruption was the damaging of a fiber optics cable of one of our network operators, UPC Cablecom. This was a global network disruption that affected both business and home customers.

Post Mortem

Our operations team investigated the issue. After the root cause was determined we were monitoring the situation and were analyzing the updates from our network partner. Please note that this accident was due to network connectivity issues and is not related to SandSIV's recent move to the new URL Shortener domain.

Update: To mitigate this issue in future, we have moved our URL Shortener services to our main data center with network redundancy.