Root cause analysis and incident report on the August DDoS attack

AMSTERDAM and PRAGUE, Aug. 21, 2019 /PRNewswire/ -- Servers.com and Qrator Labs share details on the recent attacks, so that others can prepare and be ready to face this in the future. The two companies provide details and analysis of the first significant in-the-wild DDoS attack employing one of the TCP amplification vectors. Below is the timeframe, analysis and conclusion as well as a series of steps being taken to significantly reduce the chances that any new kind of attack will significantly impact customer servers availability.

Timeframe

Sunday, August 18, 2019

07:00 UTC: Attackers started targeting Servers.com network infrastructure, producing regular traffic spikes at a high packet rate. Services availability within Servers.com network is unaffected.

07:10 UTC: Servers.com reached out to Qrator Labs intending to protect certain IP prefixes with the Qrator Ingress service, managed via BGP announcements.

13:07 UTC: A change in the attack: Servers.com monitoring system reports a combination of UDP flood and TCP SYN/ACK attack. Some Servers.com networks become affected by the attack.

13:20 UTC: Servers.com implements a tool which isolates networks affected by the attack from those unaffected to limit the impact of the attack on the customers inside the attacked networks. Customers inside the affected networks are advised to migrate to the unaffected ones. Since that moment and until attack mitigation, attackers have been periodically changing the list of targeted and untargeted networks, so the isolation-migration process has been reiterated.

Monday, August 19, 2019

15:08 UTC: Initial announcement of the affected Servers.com IP prefixes through every one of the Qrator MSSP network upstreams except CenturyLink/Level3 (more on that below).

17:48 UTC: First occurrence of the attack through the Qrator network is recognized as a mixed UDP amplification attack, with the prevalence of LDAP traffic. The attack is being mitigated successfully, continuing until 18:54 UTC.

19:01 UTC: First tactics swap: UDP amplification is being replaced with the SYN/ACK amplification attack. The attack is being mitigated successfully, continuing steadily at a stable volume until August 20, 2019, 06:33 UTC, for a total of 11.5 hours.

Tuesday, August 20, 2019

06:35 UTC: Second tactics change: SYN/ACK amplification is being replaced with a mixed TCP+UDP amplification attack. The attack is being mitigated successfully, continuing until 07:11 UTC.

07:53 UTC: Third tactics swap: a carpet-bombing amplification attack is targeting more of the Servers.com IP networks.

08:27 UTC: The rest of the affected Servers.com prefixes are re-routed through Ingress. The attack is being mitigated successfully.

10:45 UTC: CenturyLink/Level3 link becomes fully operational (see below).

Attack details

The attack mainly consisted of the LDAP amplification (with a significant portion of fragmented UDP datagrams) and the SYN/ACK amplification traffic, with other sorts of UDP amplification present periodically.

The SYN/ACK amplification traffic peaked at around 208 millions of packets per second, possibly excluding a significant portion of traffic originating in the Level3 customer cone due to static loops, suspected transit link congestions and other routing problems (see below).

The main issues behind a SYN/ACK amplification (and TCP-based packet floods in general) are:

1. This kind of a DDoS attack is almost untraceable due to low adoption of counter-spoofing approaches (such as BCP 38). As those approaches are assumed to require a substantial redesign of a network under certain circumstances, collaterally breaking things that are more important for a network, the adoption of those approaches is expected to stay low in the foreseeable future unless the IETF comes up with a different robust anti-spoofing design;
    
2. An ISP's or a datacenter's ability to handle this kind of a DDoS attack with BGP Flow Spec (or similar techniques) depends on what particular sort of equipment is deployed in their network;
    
3. For a considerable part of customers, the ability to connect to an external service or an API gateway via TCP is crucial. Web crawlers, technologies like OAuth or CDN, or enterprises such as credit scoring systems or insurance companies depend greatly on external databases (or data sources in general);
    
4. To ensure proper handling of spoofed SYN/ACKs while still maintaining a possibility to connect to an external service, a hosting company under attack would have to track all of the outgoing SYNs to match them against received SYN/ACKs later.
   
   The latter could be done in different ways, each is considered either complicated to design and deploy, or having a severe impact on network latency and RTT, or both;
    
5. The amplification factor for SYN/ACK amplification absent of relatively rare corner-cases is assumed to be between 1x and 5x. That is not a significant figure if compared to NTP's 500+ or Memcached' 9000+. However, taking into account the rest of the issues and complicated mitigation measures, five times the DDoS packet rate might be a turning point.

A series of unfortunate events further impacted the attack mitigation timeline:

• Qrator Labs network operates through many upstream Tier-1 (or regional Tier-1) Internet connectivity providers, one of which is the AS3356 CenturyLink (formerly Level3).

Qrator's autonomous systems (mainly AS200449) are perfectly multihomed and anycast-based and are designed to operate with one or more upstream networks being TITSUP or congested at any time. Having said that, 2019 marks one year for Qrator as a proud CenturyLink customer. CenturyLink provides Qrator with its excellent connectivity service in both the United States and Europe.

What severely affected the timeline of mitigation though, is the CenturyLink's 48-hours prefix filter update policy. Both Qrator and Servers.com have reached out to their respective points of contact in Level3 in an attempt to cut that waiting time, however (taking into account that the setup of the mitigation service for the production network started during Sunday in UTC) we were unable to reduce the queuing time significantly.

That is not intended to blame CenturyLink but rather to highlight that under certain conditions a BGP-enhanced DDoS mitigation service could take a significant time to connect to. Hence, it is crucial to prepare DDoS mitigation measures well in advance.

• The Qrator overlay network in specific modes may reduce MTU of the transmitted packets to 1448 or 1480 bytes. Therefore, MSS clamping setup also took place, with a few Servers.com customers being forced to implement it on their size.

Note: SYN/ACK amplification attack is also frequently being recognized as the "TCP amplification attack." While such a phrasing is obviously correct (the same way an NTP amplification attack by virtue of the fact could be called a UDP amplification attack), we believe it's important to make a distinction between SYN/ACK amplification attacks and other TCP amplification attacks, e.g. reflection attacks making use of vulnerable TCP-based servers with predictable sequence number generation algorithms.

Conclusion

With all these actions taken, Qrator Labs and Servers.com decided to form a partnership on an ongoing basis, effective today. Backed by this partnership, Qrator Ingress service would be available by default for the Servers.com global infrastructure, and all Servers.com customers can subscribe to the highest-standard DDoS attacks mitigation service provided by Qrator Labs. An important part of the partnership agreement is the extensive collaboration in research and development efforts between the two companies with the intent to share data for traffic analysis and approaches to network design improvement at scale.

Nick Dvas, CEO for Servers.com, says: "First and foremost, I sincerely apologize to every Servers.com customer whose infrastructure was affected by this event. Servers.com is committed to taking full responsibility for customer uptime. The scale of this unique attack was unprecedented, and we were not prepared for an attack of this kind.

Forming a partnership with Qrator Labs is one step we are taking in order not only to be able to mitigate the consequences of an attack of this particular kind - but more importantly to significantly reduce the chances that any new kind of attack will significantly impact customer servers availability. Servers.com strives to provide world-class services that deliver the best possible experience to our customers.

We are starting a 6 million USD investment program dedicated solely to Servers.com network design and implementation improvement in particular with regard to Servers.com ability to withstand DDoS. On a different note, Servers.com will initiate the investigation with the authorities to make sure this large scale cybercrime offense does not remain without due response."

Alexander Lyamin, COO and Founder at Qrator Labs: "This incident is a great showcase of how fragile current state of the Internet is. It's been nearly five years since the research paper describing SYN/ACK amplification was published. Since then, we've seen few public presentations or discussions on that topic and even less engineering efforts to alleviate it, finally culminating in a sequence of successful real-world attacks last week with absolutely devastating effects.

We believe that the industry should pick up the slack, taking a proactive stance this time against these threats, and more. We need better protocols, infrastructure, and technology in place today to prevent similar attacks in the near future. What can be done with the present-day tech is just the threat management and mitigation plans ready and tested for everyone, reviewed and amended at least once a year — more often is better, since threat landscape changes fast these days."

About Servers.com

Servers.com is a global IaaS hosting platform that provides automated server infrastructure. It offers a diversity of bare-metal and cloud computing services with business tools in one package. The company empowers businesses of all sizes by providing access to premium server hosting solutions around the world.

About Qrator Labs

Since 2010 Qrator Labs operates a globally distributed DDoS attacks mitigation and continuous availability network. Providing uninterrupted Internet connectivity to business customers, Qrator Labs also maintains and develops one of the biggest BGP analytics platforms - the Qrator Radar. Based on the data feeds from the DDoS mitigation network and Radar, Qrator is involved in discussing and upgrading network technologies via participation in key Internet engineering communities (RIPE, IETF, ICANN, NOGs and similar non-profits).

SOURCE Servers.com