Overview
We experienced a brief service outage on Wednesday, July 27th, 2016, which affected all VoC Hub products. During this incident VoC Hub back-end (*hub.sandsiv.com) was not accessible as well as the front-end which means that customers might have experienced problems answering the surveys.
At the time of writing, no existing customer data is at risk of being missed. The systems are currently up and running.
The period of disruption began at 16:29 CET on July 27th, 2016 and lasted until 17:49 CET the same day.
Total measured downtime during the incident: 34 minutes.
Cause
The cause of this disruption was an unintended consequence of a change in configuration for networking hardware. This change caused clustering feature malfunction of border network equipment which resulted in intermittent downtime loop of all related services.
Post-Mortem
We underestimated the network clustering behavior during usual maintenance and as a result we plan to perform the following changes to avoid such incidents in future:
1. Move scheduled maintenance events outside the business hours window.
2. Look into stabilizing clustering behavior of network equipment.
We apologize for the inconvenience this incident have caused.
Comments
0 comments
Article is closed for comments.