What happened?

SMS response channel suffered an outage on 15.06.2019 due to an issue with parsing of specific message. Specifically, incoming messages were not processed during the affected period of time.

We did not lose any requests coming to our API webhooks or via SMPP channel during the affected period of time. Processing of such messages, however, was delayed until the SMS response channel was brought up at ~19:06:00 CET.

Affected period of time: 15.06.2019 14:39:57 CET - 15.06.2019 ~19:06:00 CET

Total: ~5 hours, 27 minutes 

Why did it happen?

The issue was caused by a specific message which caused the system to generate an empty internal message which, in turn, caused the processing of incoming SMS messages to stall.

What are we doing to avoid such issues?

1. The specific message was found and we will investigate the cause of resulting empty internal message which caused the outage.

2. Internal messages processor will be made more robust to make sure empty messages don't cause it to fail.

3. Long-term, our legacy SMS channel interaction layer is undergoing a rewrite which will bring more stability and speed to the SMS response channel. We will share more details in upcoming announcements of new releases of VoC Hub.