We have identified a significant decrease in overall VoC Hub performance across all its applications that results in the complete unavailability of the service.
This issue is related to VoC Hub backend survey management system, thus Frontend survey answering system isn't affected, meaning the customers were able to complete the surveys they have been invited to.
Affected period of time: 13.01.2021 ~07:51 CET - 13.01.2021 ~21:11 CET.
Why did it happen?
We were able to identify that the root cause of VoC Hub unavailability is related to the malfunctioning Network File System server with shared disk resources.
What were we doing to resolve the issue?
The team fixed the problem with the NFS server and restarted the system's main core that eventually restored the availability of VoC Hub.
What are we doing to avoid such issues?
The whole SANDSIV team will work on improving infrastructure stability and reaction time for such cases.