What happened?

Update: The issue has now been resolved. We've updated the original article to reflect changes to affected period of time and reasons behind the downtime.

We're experiencing network connectivity issues across all production instances of VoC Hub and survey answering systems.

Our upstream service provider has updated us that traffic levels had returned to normal and the incident is resolved.

Please see the latest update here: https://www.cloudflarestatus.com/incidents/46z55mdhg0t5

Affected period of time: 24.06.2019 12:39 CET - 24.06.2019 14:02 CET

Total: 1 hour 33 minutes

Why did it happen?

The issue was caused by network connectivity issues of an upstream service provider, Cloudflare. Specifically, a route leak issue caused certain Cloudflare IP ranges to experience packet loss due to improper routing of packets. Our services were within the affected IP ranges.

What are we doing to avoid such issues?

We were in contact with Cloudflare along the entire duration of the incident.

Route leaks are common on the internet but their impact is usually minimal as the entire ecosystem works 24x7 to monitor and react to them as soon as possible. This time, however, the scale of this specific bad route announcement was much larger than previous one.

We continuously monitor the availability of our services. However, in cases like this one, unfortunately, we have no ability to act on the network issues outside the scope of our reach and influence.

If you're interested, you can read more about route leaks here: https://blog.cloudflare.com/why-google-went-offline-today-and-a-bit-about/