Content Delivery Networks, from scratch

A CDN is easy to describe in one line — “it caches your static files closer to users” — and that line hides almost everything interesting about it. This is the whole thing from first principles: why CDNs must exist (it comes down to physics), how a request actually travels, how caching and invalidation really work, and then the concrete build — how you set one up on AWS for a real frontend.

The problem: you can't out-spend physics

Forget CDNs exist. You have one server in a data center in Virginia. Your users are everywhere — including one in Sydney. The first enemy is not load. It is the speed of light.

Light in fiber travels ~200,000 km/s. Virginia → Sydney is ~16,000 km:

16,000 km ÷ 200,000 km/s = 80 ms   (one way)
Round trip (RTT) ≈ 160 ms   — a physical floor, unbeatable

In practice it's ~200–250 ms because fiber doesn't run in straight lines. And one request is not one round trip — over HTTPS several sequential round trips happen before a single byte of content arrives:

1. DNS lookup            ~1 RTT
2. TCP handshake         1 RTT
3. TLS handshake         1–2 RTT
4. HTTP request/response 1 RTT

At 200 ms RTT that's ~1 second before your HTML even starts loading, and every asset can cost more. You cannot fix this by buying a bigger server — latency is dominated by distance, and distance is fixed as long as content lives in one place.

Three more problems stack on top:

• Bandwidth & origin load — every user everywhere hits one server. 10M downloads of a 2 MB image = 20 TB off your origin.

• Single point of failure — that data center hiccups, everyone is down.

• Flash crowds — you hit the front page; 100× traffic in 60 seconds melts the origin.

The core idea

Two moves, fused: caching (the same content is requested many times — store a copy and reuse it) and geographic distribution (put those copies physically close to users). A CDN is a globally distributed network of caching servers that serve copies of your content from a location near each user.

The library analogy: instead of one national library everyone must travel to, you build a branch in every city. The national library (your origin) still holds the authoritative collection, but 95% of visits are satisfied locally — people get books faster and the origin stops drowning in foot traffic.

Anatomy: origin, PoP, edge

Often there's a cache hierarchy to protect the origin: city edge → regional parent cache → origin. Thousands of edges collapse into a handful of parents, so your origin sees a trickle, not a flood. This is called origin shielding — remember it, it matters for invalidation.

A request, end to end

Here is the whole path. The user is routed to the closest edge; the edge answers from cache; only a miss reaches back to the origin — and the next user in that region gets a hit.

How a request finds the nearest edge

This is the part people wave their hands at. You type cdn.example.com/logo.png — how does your browser end up talking to an edge 30 km away instead of Virginia? Two mechanisms, often combined.

GeoDNS (DNS-based routing)

When you resolve the hostname, the request reaches the CDN's own authoritative DNS server, which looks at where the query came from and returns a different IP per location. Same hostname, different answer per region — DNS becomes the geo-router. It can also skip dead or overloaded PoPs.

Anycast (the elegant one)

With anycast the CDN announces the same IP from many PoPs at once via BGP. The internet's routers then forward your packets to the topologically nearest PoP — you don't choose it, the network's routing does, and it self-heals: if the nearest PoP dies, BGP reroutes you to the next. This is how Cloudflare works, and it naturally disperses DDoS traffic across the whole network.

The full lifecycle to memorize:

1. Resolve cdn.example.com → anycast/GeoDNS lands you on the nearest edge
2. Browser opens TCP + TLS to that edge (~5ms away, not 200ms)
3. GET /logo.png
4. Edge computes a cache key and looks in its local cache:
     HIT  → serve the bytes immediately. Origin never touched.
     MISS → fetch from origin (or parent), store a copy, then serve.
            Next user in this city → HIT.

Caching mechanics (the heart)

The cache key. The edge decides “is this the same as something I've cached?” by building a key, by default from host + path + query string. You can customize it — include Accept-Encoding so gzip/brotli are cached separately, or exclude a tracking param so ?utm_source=x doesn't shatter your cache into a million near-duplicates (a classic real-world bug).

Who decides what's cacheable — HTTP headers. Your origin's response headers are the contract with the edge:

Freshness & revalidation. The max-age value is the TTL. Fresh → serve instantly. Once stale, the edge doesn't blindly re-download — it revalidates with a validator the origin stamped earlier:

• ETag — a version fingerprint. Edge asks If-None-Match; unchanged → origin replies 304 Not Modified (empty body).

• Last-Modified — a timestamp. Edge asks If-Modified-Since; same 304 dance.

The win: a 304 costs one small round trip and zero body bytes. You revalidate a 2 MB image for a few hundred bytes instead of re-transferring 2 MB.

Pull vs Push CDNs

• Pull (lazy, the default) — the edge starts empty; on the first miss it pulls from your origin and caches. You just set headers. Cost: the first user per region pays the cold-miss penalty. 99% of use cases.

• Push (eager) — you upload content to the CDN ahead of time, distributed to edges proactively. For large predictable-hot objects (a game update, a launch video) where you don't want a cold first miss. More operational work.

Static vs dynamic, and edge compute

Classic CDNs cache static assets — bytes identical for everyone. Dynamic content (a personalized dashboard, a cart) can't be blindly cached — serving user A's data to user B is a disaster. Three modern answers:

1. Don't cache the dynamic parts (private, no-store) but still route them through the CDN to reuse the fast edge↔origin connection.

2. Micro-cache shared responses — a homepage that's the same for everyone for 10s: s-maxage=10 turns 10,000 req/s at the origin into ~1.

Cache invalidation: the genuinely hard problem

There are only two hard things in computer science: cache invalidation and naming things. You now have copies on thousands of edges. You update the logo. How do the stale copies get replaced? Two philosophies:

1. Purge / invalidate — you call the CDN's API: “purge /logo.png.” It propagates to all PoPs; next request re-pulls. But propagation isn't instant, and purging everything stampedes your origin as every edge goes cold at once (this is where origin shielding saves you).

2. Versioned URLs (the pro move) — instead of invalidating logo.png, change its name every time it changes:

logo.abc123.png   →  (you edit it)  →   logo.def456.png

The filename embeds a content fingerprint. The new HTML references a URL the edge has never seen → clean miss → new version. The old URL can be cached forever (max-age=31536000, immutable) because its content, by definition, never changes. You get instant updates and cache-forever assets with zero invalidation calls. This is why build tools fingerprint filenames — the same “new identity instead of mutate-in-place” idea behind soft-deletes and event logs.

The one thing you can't fingerprint is the HTML entry point (its URL is fixed). So the standard pattern is: HTML = short TTL; fingerprinted assets = cache forever.

What you get for free beyond caching

Even for uncacheable dynamic traffic, routing through a CDN helps, because the edge is a smart proxy:

• TLS termination at the edge — the expensive handshake happens ~5 ms away, and the edge keeps warm reused connections to your origin.

• Protocol optimization — HTTP/2 / HTTP/3 (QUIC), tuned congestion on the long edge↔origin hop.

• DDoS absorption — anycast disperses volumetric attacks; the CDN soaks up terabits.

• Availability — stale-if-error keeps your site up while the origin is completely down.

Tradeoffs — when it bites

• Invalidation complexity — “why is the old version still showing?” is a daily CDN question.

• Harder debugging — a layer between you and users. Read Age, X-Cache: HIT/MISS, CF-Cache-Status.

• Cost model — great for read-heavy global traffic; an extra hop for nothing if all your users sit next to the origin.

• Cache-key foot-guns — wrong Vary/param handling either fragments the cache (near-zero hit rate) or poisons it (serves the wrong variant).

The 30-second mental model

A CDN pushes copies of your content to servers near users, so requests are answered from ~5 ms away instead of ~200 ms and mostly never touch your origin. Requests find the nearest edge via anycast/GeoDNS; the edge serves a HIT or fetches on MISS; HTTP headers (Cache-Control, ETag) govern freshness; and the hard problem is invalidation, best solved by fingerprinted, immutable URLs rather than purging. For a frontend on AWS that's S3 (origin) + CloudFront (CDN) + Route 53 (DNS) + ACM (TLS) — no server involved.