We have identified a significant decrease in overall VoC Hub performance across all its applications that results in the complete unavailability of the service.
This issue was related to VoC Hub backend survey management system, thus Frontend survey answering system wasn't affected, meaning the customers were able to complete the surveys they have been invited to.
Affected period of time: 05.11.2020 ~10:30 CET - 05.11.2020 13:10 CET.
Why did it happen?
We were able to identify that the root cause of VoC Hub unavailability is related to the malfunctioning Network File System server with shared disk resources.
What were we doing to resolve the issue?
After fixing the problem with the NFS server and the team restarted the system's main core that restored the availability of VoC Hub.
What are we doing to avoid such issues?
The team will continue working on analyzing the causes of the issue and will take all the necessary measures to avoid such issues in the future.