Update on yesterdays outage:
Yesterday around 12:15 we were alerted to a problem with one of our customers' websites. Since the majority of our servers were still functioning properly, and there were no alerts in our monitoring software, it wasn't immediately clear that this issue was wider than only the hosting package of this customer.
Around 12:45 we discovered that an overloaded proxy server was the underlying cause of the issue. This had not been caught by our monitoring software, because while overloaded, the server was still responding. A restart of this proxy server initially seemed to solve the issue. We immediately improved the monitoring to also notice this state, and kept an eye on the proxy server.
Issues quickly reappeared. We noticed that the high load was caused by a DDoS targeting one of the websites served by this node. Around 14:30 we managed to identify the target of the DDoS. Around 15:00 the different measures to block and divert the traffic to the attacked site were starting to take effect and normal functionality was restored.
