I broke down the November 2025 Cloudflare outage, and the lesson was to validate your own data.
On November 18, 2025, Cloudflare had its worst outage since 2019, and I recorded a breakdown of it that same day. A configuration file that Cloudflare generated itself came out bigger than its code allowed, and the new proxy crashed instead of falling back. If you build apps, the lesson is to check data before you use it, even when it comes from your own systems.
The business case
In the video I called Cloudflare the world's front door for internet traffic. Even if your site runs on AWS, Azure or Google, chances are you're using Cloudflare to protect it. I put its share at "25% of the world's traffic or something like that." That day X, Canva and Coinbase all had problems. I build production apps on Cloudflare's developer platform, so I wanted to know what broke.
Cloudflare's own chart of 5xx errors peaked at close to 30 million per second, by my reading of it. I give them real credit for the write-up, though. Within about four or five hours they had a detailed post-mortem on their blog showing where they went wrong, and I can't say that for every cloud provider.
Disclosure: in April 2026 I said Cloudflare gave me a $25,000 credit as sponsorship. No affiliate links.
The bug was a missing check on data that came from Cloudflare's own database, and any of us can make that mistake in our own apps. I also said in the video that the AI race is pushing big providers to ship faster with fewer checks.
How a request flows
When you visit a site behind Cloudflare, your traffic goes through Cloudflare's network first. The first step terminates HTTPS. Cloudflare checks the certificate once, so the systems further down don't repeat that work. Then the request goes to what Cloudflare calls FL, short for Frontline. In my simplified diagram, it holds the web application firewall, rate limiting and bot analysis. If the request passes, Cloudflare hands it to your site, wherever it's hosted.
Bot analysis is the part that broke. It decides whether a request comes from a person or a bot, like the Google crawler or the AI companies crawling sites for data. Cloudflare's Bot Management gives each request a bot score from a machine learning model, and that model reads a "feature" file listing the signals it uses.
The file that grew
On the day, I said the code expected about 60 items and got more than 200. The post-mortem gives the exact version. Cloudflare uses about 60 features, and the module has a hard limit of 200 so it can set aside memory ahead of time. The bad file had more than 200 entries, the limit was hit, and the system panicked.
The file comes from a query against ClickHouse, the open-source column-oriented database Cloudflare uses for analytics. A permissions change on that ClickHouse cluster made the query return duplicate rows, which more than doubled the file. Cloudflare then pushes the file out to the servers running the bot code.
Jump to
- 0:00 Introduction and CloudFlare Outage Overview
- 0:12 Impact of the Outage
- 0:40 Fixing the Outage
- 1:08 Detailed Breakdown of the Outage
- 4:29 CloudFlare's Role in Internet Infrastructure
- 7:08 Technical Deep Dive into the Code
- 19:58 Lessons Learned and Mitigation Strategies
- 25:36 Conclusion and Final Thoughts
I walk through the post-mortem graph, the diagram and the Rust code on screen in the full video: my Cloudflare outage breakdown on YouTube.
Why the graph flapped
The error graph spiked, dropped back and spiked again, and I put that down to the change rolling out across nodes. According to the post-mortem, the file was regenerated every five minutes, and it only came out bad when the query ran on a part of the cluster that had already been updated. The network kept flipping between a good file and a bad one. Cloudflare's status page went down too, which turned out to be a coincidence, since it's hosted off Cloudflare. Both made their team suspect an attack at first.
The bug was in FL2, the new proxy written in Rust that Cloudflare was moving customers onto at the time. My service didn't go down that day, and my understanding was that we were still on the older FL. Cloudflare's post says customers on the old proxy didn't see errors, but every request got a bot score of zero, so anyone blocking bots by score saw false positives.
The one line
The failing Rust code calls unwrap(). In Rust that means panic if this fails, with no safe fallback. I'm no Rust expert and no expert on Cloudflare's network, so this part is my speculation as a developer. I called it a rookie mistake to let a bad bot list take down such a critical piece of the system. I'd want it to degrade gracefully or roll back.
I pasted the code into Gemini and ChatGPT. ChatGPT suggested checking the count, rejecting the file when it's over the limit, logging the error, warning the monitoring system and falling back to the last known-good state. I was clear that this isn't the fix for Cloudflare's problem. For the rest of us, it's defensive programming: when your code gets data, validate it before you continue. Cloudflare's first follow-up item says much the same, treating the config files it generates the way it treats user input.
A Hacker News commenter tied this to the AT&T long-distance network crash decades ago. That one also came down to a single line of code, and it sat in the system through the Christmas holidays before it failed.
Stop the rollout early
My biggest question for Cloudflare was why the change kept spreading after the first node started throwing errors. At this scale you need more than one fail-safe. I'd roll a change to one node, watch it, and stop the spread the moment errors run higher than usual. Cloudflare's counterpoint is that this file updates often because attackers change tactics fast. My idea was two lanes: a fast one for responding to attacks and a slow, tested one for regular updates.
What a small app can do
For a small app, the options are limited. The cheapest step is to keep your domain registrar separate from Cloudflare, say at GoDaddy, and point your DNS to Cloudflare. To go further, replicate your DNS records somewhere else and switch over during an outage. That can take about an hour and depends on how fast the rest of the internet picks up the change. Microsoft and AWS both offer DNS failover. Sometimes you're better off waiting it out.
Validate what you trust
Azure and AWS had outages in the weeks before this one, so that made three major outages in about 30 days. Putting on my tin foil hat, my theory was that the AI race is pushing these companies to deploy faster with fewer checks, while their systems now touch hospitals and other critical infrastructure. Take that with a grain of salt. The part you control is your own code: check the data, keep a known-good fallback, and log the problem instead of crashing. I've also written about how I map a SaaS architecture on Cloudflare. If you build on Cloudflare and want the foundations, take my free Cloudflare course.
About Dwain
I'm Dwain Browne, Founder of LeadScore AI and SnapSuite. I'm a Toronto-based software architect with 25+ years across software, sales, operations and entrepreneurship. SnapSuite helps contractors and trades teams get paid faster, and LeadScore AI shows you which leads deserve your next call. I also run Vibe Code to Profit, a community helping people build software with AI.