RFO (Reason for Outage)

Issue summary

Between December 17th to December and 20th we suffered several DDoS attack affecting Greenhost platform availability at different levels.

Incident timelines

• 2020/12/17 - 09:10 UTC DDoS attack started

  The attack was targeted at a Greenhost customer. The attack unexpectedly also affected also other parts of the network. A significant amount of Virtual Servers had to be rebooted afterwards, including some providing services for shared services.

  Management and core infrastructure were not impacted.

  • 10:24 UTC DDoS attack was deflected and network stabilized. Hosting and Cloud VPS were partially unavailable at this point. Mail was not available. Service center was working.
  • 11:00-11:15 UTC Hosting platform coming back to capacity
  • 11:30 UTC Mail fully operational starting to process the backlog of emails.
  • 15:00-15:30 UTC Cloud virtual machines which didn't seem to be operational were brought up

• 2020/12/19 - 23:11-00:12 UTC DDoS attack

  It may have caused unavailability of some systems or long latency on connections

• 2020/12/20 - 12:37-13:37 UTC DDoS attack

  Systems were unavialable

• 2020/12/21 - 16:30-17:16 UTC Issue with DNS resolution

  Systems were unavailable due to a change on our DNS setup which made DNS resolution not work properly

Root cause

The main cause was a large scale DDoS attack which targeted at least one of our customers and after that towards Greenhost servers as well. This included the Greenhost website as well as our nameservers were targeted.

Affected services

• Cloud virtual machines
• Hosting services
• Mail services
• Service center

Corrective/Preventative actions

• Make DNS infrastructure more resilient and fault tolerant
• Use DDoS mitigation
• Parts of our network design will be adjusted to limit the possibility of a distribution of the attack on the network.