During the affected period of time issues with the database server of VoC Hub and the API server of VoC Feedback module of VoC Hub were observed:
- Issue with the VoC Hub database server on September 29 affected both frontend (survey answering system) as well as backend (survey management and analytics apps) parts of VoC Hub.
- Issue with the VoC Feedback API server on September 30 affected only backend (survey management only) part of VoC Hub.
Existing customer data was not affected by this issue.
Issues occurred during non-business hours.
Downtime period (VoC Hub Database): September 28,
Downtime period (VoC Feedback API): September 29,
Why it happened?
Due to an unexpected issue, VoC Hub database server started to misbehave on September 28 which caused performance degradation and intermittent outages of all VoC Hub services.
Another unexpected issue caused VoC Feedback API server to shutdown abruptly on September 29. An automated process which should've launched the server didn't succeed in recovering the server.
What has been done to avoid such issues?
Changes were made the OS environment of VoC Hub database server to prevent the class of issues which caused the server to fail.
Our teams are investigating the reasons behind VoC Feedback API server shutdown. Further investigations will be performed to make sure the automated process works properly in case of API server disruptions.