All systems are operational.

RFO (Reason for Outage) Event Report

• Issue summary

  On 2nd April our hosting platform was subject of an outgoing DDoS attack that partially affected VPSes and hosting services.

• Incident timelines

  • 2022-04-02 - 22:22 - Attack starts making some VPSes and hosting services unavailable.

  • 2022-04-02 - 23:15 - VPSes are fully available as well as e-mail, shared hosting websites are partially unavailable

  • 2022-04-03 - 00:50 - All shared hosting is up

  • 2022-04-03 - 09:00 - Small amount of shared hosting websites present unavailability due some changes in the infrastructure

  • 2022-04-03 - 19:37 - All shared hosting is up

• Root cause

  A vulnerability on the wordpress tatsu plugin was exploited to launch a DNS amplification attack through vulnerable sites.

• Affected services

  • Partially cloud virtual machines
  • Partially Hosting services
  • Partially Mail services

• Corrective/Preventative actions

  • Upgrade the vulnerable plugin on the affected websites
  • Improve platform restrictions to avoid future attacks
  • Improve monitoring

AMS-ix confirmed the exchange is currently stable. We will continue to monitor the connections closely.

Around 10.23 CET we discovered network disruptions towards our infrastructure. This caused packet loss or slowness on some routers/paths. After investigating we discovered issues at the Amsterdam Internet Exchange (AMS-ix),however, at 10.45 traffic recovered. We are waiting for clarifications from AMS-ix, meanwhile we monitor traffic closely and will reroute traffic if the problems at AMS-ix reoccur.

Yesterday, 2 April, around 22:25 our systems we subject to a large attack. We managed to block most of it quickly. Around 23:15 it seemed the attack was defeated fully.

In blocking the attack we made some configuration changes that caused other problems. We inadvertently blocked our internal DNS servers, resulting in (after a small delay) websites not being able to connect to databases, and webmail not being able to connect to it's database. These problems became clear a little past midnight. At 01:10 these issues were fixed too.

We apologize for the disruption. We will evaluate what went wrong and see what we can do to prevent similar incidents from happening again.