Cloudflare - Firethering

Imagine discovering that your infrastructure is using 100 terabytes of memory you don't actually need. You could buy more servers. Add more RAM. Or, as Cloudflare just showed, you could look much closer at the code. The company has freed up roughly 100TB of memory across its network by changing how its DNS caching system stores data. No new servers or bigger memory modules. Just a series of surprisingly small changes to the software powering the DNS infrastructure behind services like 1.1.1.1. The reason this became possible is scale. Cloudflare's DNS cache holds more than 250 billion entries at any given time. At that scale, a few unnecessary bytes attached to every entry aren't really a few bytes anymore. They're hundreds of gigabytes and eventually, terabytes. Also Cloudflare didn't just make the cache smaller. It made it faster too. So how do you find 100TB of RAM hiding inside a DNS cache? It turns out, the answer starts with some very ordinary pieces of Rust code. Table of ContentsIt started with 250 billion DNS entriesThe memory hiding inside a VecThen Cloudflare started removing pointersSometimes, the best data is no dataHow a 4-byte record ended up taking 144 bytesThe Result Was More Than Just Efficiency It started with 250 billion DNS entries The system Cloudflare had to optimize is called Big Pineapple, the DNS caching platform behind 1.1.1.1 and several of the company's other DNS services. At any given moment, Big Pineapple holds more than 250 billion DNS cache entries across Cloudflare's network. Each entry stores information about a DNS query and its response. That sounds simple enough, but when you have hundreds of billions of entries sitting in memory at the same time, the way each individual entry is represented starts to matter a lot. Cloudflare found that some of its data structures were carrying information they didn't really need. A single unused byte might seem irrelevant when you're looking at one cache entry. Multiply that by 250 billion, however, and that same byte represents more than 250GB of memory. So the engineers started looking at the cache one piece at a time and their first target was something almost every Rust developer has used: Vec. The memory hiding inside a Vec A Rust Vec is designed to grow. Along with the data it stores, it keeps track of things like where that data lives, how many items it contains, and how much room has been reserved for future items. That's useful when you're continually adding things to a collection. But Cloudflare's DNS cache had a different situation. Once a DNS response was stored in the cache, it wasn't going to grow anymore. Yet the Vec was still carrying the capacity information needed for a collection that could grow. So Cloudflare replaced those growable structures with fixed-size representations that didn't need to keep that extra capacity metadata around. The change sounds almost too small to matter. Across billions of cache entries, it mattered a lot. Cloudflare estimates that this change, along with eliminating the unused space that Vec could reserve, saved more than 15TB of memory on its own. And that was only the beginning. Then Cloudflare started removing pointers The next problem was hiding in the way Big Pineapple organized DNS responses. A cached response contains several sections, including the answer, authority, and additional records. Cloudflare's original design kept these in separate lists. That meant every list needed its own metadata including pointers to where the data lived and information about how much data it contained. There was nothing fundamentally wrong with this approach. It was straightforward, easy to work with, and perfectly reasonable for a much smaller system. But Cloudflare wasn't operating a small system. The engineers realized they could store the records together in a single block of memory and use small offsets to mark where each section began. Instead of maintaining multiple lists, they essentially kept one list and a few directions telling the system where to look. That removed unnecessary pointers and metadata from every cache entr and this is where the larger lesson starts becoming clear. Cloudflare wasn't finding some enormous memory leak buried deep inside 1.1.1.1. It was finding tiny pieces of overhead that had been multiplied hundreds of billions of times. And sometimes, the best optimization is realizing you never needed to store it in the first place. You May Like: Best Open Source AI Assistants You Can Run Locally Sometimes, the best data is no data Cloudflare found another opportunity by looking at what was being stored inside each DNS record. Many DNS records contain the same domain name that was already present in the cache key. For example, if the cache is storing the response for example.com, several records in that response may simply say that their owner is example.com. So the cache could end up storing the same information twice: once in the cache key, and again inside the individual records. Cloudflare changed that. When a record's owner is the same as the queried domain, the cache no longer stores the owner name. Instead, it simply remembers that the owner is the domain from the cache key and reconstructs it when the response is served. If the owner is different, as can happen with something like a CNAME chain: Cloudflare still stores the full name. It's a small distinction, but an important one: don't store something you already know. Removing those repeated names also meant fewer heap allocations, which reduced some of the overhead associated with managing all that data. By this point, Cloudflare's approach was becoming clear. They were asking a simple question over and over again: Does every byte we're storing actually need to be there? Then they found one of the strangest examples yet, a tiny 4-byte DNS record that could end up occupying 144 bytes. How a 4-byte record ended up taking 144 bytes DNS records come in different shapes and sizes. An IPv4 A record, for example, needs just 4 bytes to store its address. An AAAA record needs 16. But Big Pineapple was using a Rust enum to represent different types of DNS records. That created an unusual problem. A Rust enum has to be large enough to hold its largest possible variant. In Cloudflare's case, one of the less common DNS record types, NAPTR, was much larger than an A or AAAA record. The result was that the enum occupied 144 bytes. So an A record that only needed 4 bytes could effectively be sitting inside a structure with enough room for a 144-byte record. Most of that space was simply unused. And because A and AAAA records account for more than 80% of the traffic Cloudflare measured, this wasn't some obscure edge case. The system was repeatedly reserving space for large record types while usually storing much smaller ones. Cloudflare's solution was to move the larger record types out of the enum and store them separately, while keeping the small, common records inline. That dramatically reduced the amount of space wasted by the common record types. But Cloudflare wasn't finished. The engineers eventually took the idea one step further: instead of storing every record as a separate structured object, they packed the record data into a single block of raw bytes. That meant fewer individual allocations and better memory locality — the data Cloudflare needed was now sitting together instead of being scattered around the heap. And surprisingly, this didn't just save memory. It also meant the cache had less work to do when serving DNS responses. After all those small changes, the numbers started to look anything but small. Also Read: Not Everything Needs AI: Powerful Alternatives to the Apps Everyone Uses The Result Was More Than Just Efficiency Individually, none of these optimizations sounds like the kind of thing that should make headlines. Together, they transformed the economics of Cloudflare's DNS cache. In its benchmarks, Cloudflare reduced the memory footprint of a typical cache entry from 953 bytes to 420 bytes, a 56% reduction. The number of allocations per entry also fell from 1.1KB to 461 bytes. But the surprise was that the cache got faster at the same time. Insert throughput increased from 625,000 to 893,000 entries per second, while lookup latency dropped from 828 nanoseconds to 670 nanoseconds. And when the changes reached production, the savings became more. At the 99th percentile, memory usage per instance fell from 9.3GB to 5.3GB, a 43% reduction. Across Cloudflare's network, the company estimates that the changes freed roughly 100TB of memory. That's roughly the amount of RAM contained in 130 of Cloudflare's Gen 13 servers. And Cloudflare doesn't plan to simply leave that memory sitting unused. The company says it plans to use the recovered capacity to make its DNS caches larger. More cached records could mean more queries answered directly from Cloudflare's infrastructure, reducing the number of requests that have to travel upstream to authoritative DNS servers. So Cloudflare didn't really make 100TB of RAM appear out of nowhere. It found 100TB that was already there, hidden inside billions of tiny inefficiencies.

Tag: Cloudflare

Most Read

Are AI Chats Private? What the Claude Diary Case Reveals

openGym: Open Source Workout Tracker With Android App

Michael Burry Wants Markets to Tank Just to Stop OpenAI and Anthropic from Going Public

Frappe Books: Open Source Accounting Software Without the Subscription