We experienced a couple of outages on 17.04.2017, 26.04.2017 and 27.04.2017 which affected VoC Feedback backend usage. During this incident VoC Feedback backend (allows to create new and edit existing surveys) was unavailable. This outage didn't affect customers' ability to answer existing questionnaires, new questionnaires, or to receive invites.
Existing questionnaire data was not lost. We're not seeing any evidence of data loss whatsoever.
Total measured downtime during business hours*: 1 hour 1 minute
Total measured downtime: 11 hours 54 minutes
Why did it happen?
With VoC Hub 8.1.3 which was delivered on 07.04.2017 we began to see unexpected shutdowns of the service which is responsible for delivering API calls for VoC Feedback.
What are we doing to resolve this issue?
This particular part of the system was not monitored explicitly but rather accessibility of VoC Hub as a whole was under checks. This was improved by adding specific monitors for availability of the API service. Additionally, we're going to investigate what caused the abrupt shutdowns and meanwhile a temporary solution to resume the service quicker in case of an unexpected failure was deployed to production environment.
We apologize for the inconveniences these outages have caused.
* Business hours: 9:00 - 17:00 CET