Overview
We experienced a brief service outage on Wednesday, July 20th, 2016, which affected all VoC Hub products. During this incident VoC Hub back-end (*hub.sandsiv.com) was not accessible as well as the front-end which means that customers might have experienced problems answering the surveys.
As at the time of writing, no existing customer data is at risk of being missed. The systems are currently up and running.
The period of disruption began at 09:40 CET on July 20th, 2016 and lasted until 10:15 CET the same day.
Cause
The cause of this disruption was identified soon. This was a network disruption caused by the misbehaved internal server causing a DoS-like effect on our proxy server by accessing it at a very fast rate. Once the cause was determined, actions followed including shutting down the connectivity between the networks so that the proxy server is able to forward the requests as intended.
Post-Mortem
We underestimated the probability of internal services causing an effect on our production systems and as a result we plan to perform the following changes to avoid such incidents in future:
1. Move internal services (Proxy, DNS, etc.) related to development to other servers/networks not related to production environment.
2. Revise proxy server settings to ensure accidents such as this one don't occur in future.
We apologize for the inconvenience this incident have caused.
Comments
0 comments
Article is closed for comments.