alt Hacker NewsUnderstanding the recent DDoS attack against Read the Docs
Comments
Sorry for the stress that the team had mitigating this attack. Yet I'm always excited to see signs of competence on the attacking side: a targeted application-specific and adaptive attack at L7? Wow!
Here's another guess: a DDoS to steal your attention and mask other intrusion attempts.
> Defenses need to have broader rate limits across more than just IPs (ASNs, hostnames, etc.).
The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
> Attackers actively search for non-cacheable paths (e.g. dynamic redirects, search endpoints, and 404s).
To continue with my previous point. As long as individual server's resources permit (memory, socket limit) stall requests before processing them. Low reputation IPs get stalled for longer and the requests that exceed the queue get dropped. The idea is graceful degradation:
A good rep IP will not be stalled by sleep(). A poor rep IP will be stalled, but eventually receive its answer instead of some 429/403 (i.e. a user who opened many tabs at once). A bad rep IP will be slowed down by the wait times + rate-limits (queue exceeded) before getting completely banned for good. The point is to have more granularity before throwing errors at random users at the server-level.
> they were overwhelming a hardcoded Nginx redirect (a simple rewrite regex directive)
1. I wonder how much optimization ngx_http_rewrite_module has? Does it precompile the patterns? LLM said yes, this SO answer [1] says that a JIT config option must be on. I consider "Just in Time" to be a half measure when the config itself is static.
2. From looking at NGINX docs, it looks to me there are some pitfalls to writing these rules. Like you must manually make sure to short-circuit the rewrites to exit early?
3. The caveat of regex is that catastrophically backtracking regexes do look simple. I don't see this issue being talked about enough. See links, if you, the reader, haven't heard of it yet.
[1] https://stackoverflow.com/questions/59284921/how-much-impact...
[3.1] https://joshua.hu/nginx-directives-regex-redos-denial-of-ser...
[3.2] https://en.wikipedia.org/wiki/ReDoS
[3.3] https://infrafolks.com/blog/regex-backtracking-devops/
[3.4] https://www.regular-expressions.info/catastrophic.html
this might be a AI driven attack and readthedocs was just a test target.
What surprised me is how easy it was to evade the cloudflare defenses. I know it was easy to evade CF, but I would expected CF to do a better job at blocking L7 DDOS.
CF is really good in defending against the L4 DDOS, but not L7.
this means that cloudflare is really not useful much in the era of Agentic DDOS driven by thousands agents across the globe
There’s an assumption that turning on Cloudflare’s “under attack” mode would mitigate the attack.
Given how adaptive the rest of the attack was, I would be very curious to find out how it would approach that obstacle.
A more interesting question is, what exactly do the attackers gain from hitting read the docs? Most of their docs hosting is static/easily CDN cached. Unlike database bound sites, you would need a lot more traffic to overload pure/mostly static hosting. Maybe it's a malicious AI lab looking to deny their competitors training data? As far as infosec profiling goes, this is probably the oddest case I have heard of.
I am thinking it's probably an AI lab that misconfigured their data scraper (made it too agentic) and it ended up looking like a DDoS.
The new generation of scrapers are all agentic and self healing. (As an example see YC's https://parse.bot)
Interesting that the Under Attack Mode wasn’t used at all here. I understand not wanting to break APIs but I feel temporarily challenging non-API usage could have at least helped without impacting users too much?
I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
> One decision we made is to always give real users an escape hatch. Read the Docs very rarely issues outright blocks or bans to specific IPs or user agents. Instead, our "worst" is a JavaScript challenge, and if a user solves a challenge, they are very unlikely to get challenged again for the next day or so.
Finally, a competent response that doesn't leave the users hang out to dry.
I'm so tired of seeing incompetents with measures like "blackhole 2 continents" deployed even outside active attacks.
[flagged]
[flagged]
[dead]
I'd like to see more of a legal response.
First, find out who's on the other end of a few hundred IP addresses. Start with ones in the US. Sue for damages. Use discovery to find out what's on the other end. Sue the maker of that device. If it turns out to be an appliance or smart TV, it may be possible to consolidate cases into one case against the manufacturer. Criminal negligence, tort interference with contract, harassment, Computer Fraud and Abuse act violation... Maybe a restraining order prohibiting the sale of "smart TV" known to be able to host attacks. Have imports seized by Customs and Border Protection. That would get a manufacturer's attention.
The manufacturer's EULA will not help the manufacturer, because the plaintiff, the party being attacked, is not a party to the EULA at all.