VoC Feedback survey answering system limited outage on 15.05.2017 - 17.05.2017

What happened?

We experienced a limited outage of VoC Feedback survey answering system for certain customers which use custom survey domains in invites for WEB channel between May 15, 18:34:00, 2017 СЕТ and May 17, 12:07:00, 2017 СЕТ. This outage resulted in error messages such as "This site can't be reached" and "[domain_name].customer_name.tld refused to connect" when clients of certain customers were attempting to open a survey from an email invite. This outage didn't affect VoC Feedback backend usage (allows to create new and edit existing surveys) and the possibility to receive invites.

We're not seeing any evidence of existing questionnaire data loss.

Downtime period: May 15, 18:34:00, 2017 СЕТ - May 17, 12:07:00, 2017 СЕТ

Total measured downtime during business hours*: 11 hours 7 minutes

Total measured downtime: 41 hours 33 minutes

Why did it happen?

After restarting one of the services that supports custom survey domains in invites a web server has failed to launch due to an unexpected issue.

What has been done to avoid such issues?

A monitoring module was implemented that will perform required checks and will monitor the accessibility of VoC Feedback answering system for custom domains to avoid such outages in the future.

We apologize for the inconveniences this outage has caused.

* Business hours: 9:00 - 17:00 CET