A massive distributed denial of service (DDoS) attack on Wednesday, Feb. 28, left users unable to access the code repository GitHub for nearly 10 minutes. The 1.35 Tbps attack was the ever largest seen, according to the content delivery network services provider Akamai Technologies.
The attack on GitHub was delivered through a new method involving the memcached distributed memory caching system, which is designed to speed up performance of Web sites with dynamic, disk- or database-driven content. Attackers can flood such sites with huge volumes of traffic via memcached's use of the User Datagram Protocol (UDP), a core Internet Protocol transport feature.
Just a day before GitHub was hit, Akamai had reported that DDoS attacks using UDP-based memcached traffic had the potential to reflect and amplify traffic loads of 190 Gbps and more. Akamai warned that "organizations need to be prepared for more multigigabit attacks using this protocol and should plan accordingly."
The largest previously reported DDoS attack was a 1.2 Tbps attack on the domain name provider Dyn in October 2016. That attack temporarily knocked multiple large sites, including Twitter and Spotify, offline.
Amplifying Traffic by 51,000x
Wednesday's attack on GitHub left the site unavailable for five minutes shortly after noon Eastern Time, and only intermittently available for another four minutes after that. However, the attack did not at any point affect the confidentiality or integrity of users' data, GitHub engineering manager Sam Kottler wrote in an update on the site yesterday.
Kottler said the attack worked by taking advantage of memcached instances that are "inadvertently accessible on the public Internet with UDP support enabled." By spoofing IP addresses, the attacker or attackers were able to direct memcached responses to GitHub, multiplying the volume of data sent in the process.
"The vulnerability via misconfiguration described in the post is somewhat unique amongst that class of attacks because the amplification factor is up to 51,000, meaning that for each byte sent by the attacker, up to 51KB is sent toward the target," Kottler said.
He added that over the past year GitHub had been taking steps to boost its transit capacity to better withstand DDoS attacks, and planned to continue doing so.
Be Cautious with UDP
"Making GitHub's edge infrastructure more resilient to current and future conditions of the Internet and less dependent upon human involvement requires better automated intervention," Kottler noted. "We're investigating the use of our monitoring infrastructure to automate enabling DDoS mitigation providers and will continue to measure our response times to incidents like this with a goal of reducing mean time to recovery (MTTR)."
What steps can other organizations take to prevent coming under a similar DDoS attack? The content delivery network provider Cloudflare said one key is to "stop using UDP."
"If you must, please don't enable it by default," said Cloudflare team member Marek Majkowski in a blog post. "We've been down this road so many times. DNS, NTP, Chargen, SSDP and now memcached. If you use UDP, you must always respond with strictly a smaller packet size then the request. Otherwise your protocol will be abused. Also remember that people do forget to set up a firewall. Be a nice citizen. Don't invent a UDP-based protocol that lacks authentication of any kind."