How Requests Flow Through Systems

Building Blocks

• Recursive Resolver: The server that does the legwork of a lookup on behalf of the client, typically run by the ISP or a public provider (Cloudflare 1.1.1.1, Google 8.8.8.8). It iteratively queries root, TLD, and authoritative servers, caches the result for the TTL, and answers future identical queries from cache. A warm resolver cache is why most DNS lookups complete in under 10 ms.
• Root, TLD, and Authoritative Servers: The three tiers of the DNS hierarchy. Root servers (13 named logical services, actually hundreds of anycast instances worldwide) direct queries to TLD servers (.com, .org, country codes). TLD servers point to the domain's authoritative nameserver (e.g., Route53 or Cloudflare nameservers for your domain), which holds the actual records. Each tier only knows about the layer below it, which is what makes the system scale.
• Record Types (A, AAAA, CNAME): A records map a name to an IPv4 address; AAAA records map to IPv6. CNAME records alias one name to another name (e.g., www.example.com → example.cdn.net), which then resolves to an A record. A CNAME cannot coexist with other records at the same name, and you cannot put a CNAME at the zone apex in classic DNS — which is why providers invented ALIAS/ANAME pseudo-records.
• TTL and Caching: Every DNS record carries a TTL (time-to-live) telling resolvers how long to cache the answer. Long TTLs (hours) reduce lookup latency and load on authoritative servers but slow down failover — clients may keep sending traffic to a dead IP until the TTL expires. Short TTLs (30–60 seconds) enable faster traffic shifts at the cost of more authoritative queries and slightly higher first-hit latency.
• Anycast: A routing technique where the same IP address is announced from many physical locations, and the network routes each user to the nearest one. Major DNS providers and CDNs use anycast so a query to 1.1.1.1 is answered by whichever of hundreds of worldwide PoPs is closest. Anycast improves latency and provides natural resilience — if one location dies, routing simply shifts to the next nearest.
• DNS-Based Load Balancing: Returning different IPs for the same name to spread or steer traffic: round-robin DNS rotates a list of A records; geo DNS returns region-specific IPs; weighted records shift percentages of traffic. Its limits are fundamental: resolvers and clients cache answers past your control, DNS has no health awareness (it will happily return a dead server's IP until the TTL expires and the record changes), and changes propagate slowly and unevenly.

Definitions

DNS Resolution

The process of translating a domain name into an IP address by walking the DNS hierarchy.

• Full cold resolution path: client → recursive resolver → root server → TLD server → authoritative server → answer.
• Each hop adds latency, but caching collapses most real-world lookups to a single resolver query.
• The resolver queries iteratively: each server answers with a referral to the next layer, not the final answer.

TTL (Time-To-Live)

The number of seconds a DNS record may be cached by resolvers and clients before it must be re-fetched.

• Low TTL (30–300 s) enables fast failover and frequent traffic steering; high TTL (hours) minimizes lookups and latency.
• TTL is the main reason DNS changes are 'propagating' — stale caches keep serving the old answer until expiry.
• Some resolvers and ISPs ignore very low TTLs and enforce a minimum, so you cannot fully control cache behavior.

A / AAAA Record

Records that map a domain name directly to an IPv4 (A) or IPv6 (AAAA) address.

• Multiple A records on one name is the simplest form of load balancing: clients pick from the returned list.
• AAAA records matter increasingly as mobile networks and ISPs go IPv6-first.
• Because these hold literal IPs, changing them is how you move traffic between data centers or providers.

CNAME Record

A record that aliases one domain name to another, deferring the final IP lookup to the target name.

• Canonical use: pointing app.example.com at a CDN hostname like d123.cloudfront.net so the CDN controls the final IPs.
• Adds one extra lookup per cold resolution, which slightly increases latency.
• Cannot exist alongside other record types at the same name, and classic DNS forbids it at the zone apex.

Authoritative Nameserver

The server that holds the definitive records for a domain and answers resolvers directly for that zone.

• Hosted by your DNS provider — Route53, Cloudflare, NS1, Google Cloud DNS — or self-hosted with something like BIND.
• Its availability bounds your availability: if it stops answering, caches save you only until TTLs expire.
• Good providers run it as a globally anycasted, DDoS-hardened service; this is why 'use two DNS providers' is a real resilience strategy.

Anycast

Announcing the same IP from multiple locations so routing directs each user to the nearest instance.

• Used by root servers, public resolvers (1.1.1.1 answers from 300+ locations), and CDNs.
• Failover is a side effect of routing: withdraw the announcement from a dead site and traffic shifts automatically.
• Works best for connectionless, stateless protocols like DNS over UDP; TCP services need more care.

DNS Propagation

The informal term for the delay between changing a DNS record and the change being visible everywhere.

• There is no push mechanism — 'propagation' is just every resolver's cached copy expiring at its own pace.
• Bound roughly by the old record's TTL: a 24-hour TTL means up to a day of mixed old/new answers.
• Standard migration practice: lower the TTL a day or two before a cutover so the change takes effect quickly.

Patterns

• Geo DNS / Latency-Based Routing — When you serve users from multiple regions and want each user directed to the nearest or fastest region.
• DNS Failover with Health Checks — When you need automatic region or endpoint failover without a global load balancer.
• CNAME to a Managed Edge — When delegating traffic steering to a CDN or cloud load balancer that changes its IPs dynamically.
• Weighted DNS for Progressive Rollouts — When you want to shift a percentage of traffic to a new deployment without client changes.

Strategies

• Pick TTLs Per Record Purpose When: Always — TTL choice is a deliberate availability/latency decision, not a default. How: Stable records (mail servers, verification TXT) get long TTLs like 86400 s. Records behind failover or frequent deploys get 30–300 s. Before a planned migration, drop the TTL a day early, wait one old-TTL period, make the change, then raise it again. Example: An e-commerce site keeps its CDN CNAME at 3600 s but its failover-managed origin A records at 60 s, so a region cutover converges within about a minute of cache expiry.
• Treat DNS as a Dependency with Its Own SLO When: When your availability target is 99.9% or higher — your DNS provider's outage becomes your outage. How: Choose a provider with anycast infrastructure and published SLAs, monitor resolution latency and failure rate from multiple regions, and for critical systems consider dual DNS providers (two authoritative providers serving the same zone) to survive a provider-wide incident. Example: After the October 2016 Dyn DDoS attack took down Twitter, Spotify, Reddit, and Netflix domains simultaneously, many large companies adopted dual-provider DNS so a single authoritative provider outage no longer makes them unresolvable.
• Layer DNS Steering Under Real Load Balancing When: When you need both coarse geographic steering and fine-grained, health-aware distribution. How: Use DNS to choose a region (coarse, slow-changing, cached), then a regional load balancer with active health checks to choose a server (fast-changing, immediate failover). DNS handles 'which continent'; the LB handles 'which machine is alive right now'. Example: A global API uses geo DNS to send users to one of three regions, and inside each region a load balancer removes unhealthy instances in seconds — combining DNS's global reach with LB-level reaction speed DNS cannot provide.
• Fail Over at the Client When DNS Can't React When: When TTL-bound DNS failover is too slow for your recovery target. How: Return multiple IPs or endpoints to clients and let the client retry alternates on connection failure. Mobile apps and service meshes commonly do this: they treat DNS answers as hints, not guarantees, and keep working through a stale answer by falling back to cached or secondary endpoints. Example: A mobile app caches the last-known-good API IPs; when the resolved IP times out, it tries the cached alternates before surfacing an error, surviving a stale-DNS window that pure DNS failover would not.

Tradeoffs

Real World

Quiz

During a cold DNS lookup, what is the correct order of servers a recursive resolver contacts?

1. Authoritative → TLD → root
2. Root → TLD → authoritative
3. TLD → root → authoritative
4. Root → authoritative → TLD

The resolver walks the hierarchy top-down: root servers point to TLD servers, TLD servers point to the domain's authoritative nameserver, which returns the final record.

Why does a DNS-based failover take minutes rather than seconds even after you update the record?

1. Authoritative servers batch updates hourly
2. BGP must re-converge across the internet first
3. Resolvers and clients keep serving the cached old IP until the TTL expires
4. TCP connections must finish before new lookups are allowed

DNS is a cached system: every resolver and client holds the old answer for up to the record's TTL, so traffic shifts gradually as caches expire — there is no push mechanism.

What does a CNAME record do?

1. Maps a name directly to an IPv4 address
2. Aliases one domain name to another name, which is then resolved separately
3. Encrypts DNS queries between client and resolver
4. Maps an IP address back to a domain name

A CNAME defers the answer: www.example.com → example.cdn.net, and the resolver then looks up example.cdn.net to find the actual A/AAAA record.

What is the fundamental limitation of DNS round-robin as a load-balancing mechanism?

1. It only supports two servers at a time
2. It has no health awareness and cannot control caching, so dead servers keep receiving traffic
3. It requires clients to support HTTP/2
4. It increases the size of DNS packets beyond the UDP limit

DNS will keep returning a dead server's IP until the record changes and caches expire; it cannot see server health, connection counts, or actual load — which is why health-checked load balancers sit beneath it.

What is the main benefit of anycast for a public DNS resolver like Cloudflare's 1.1.1.1?

1. It encrypts queries end to end
2. It guarantees every query bypasses the cache for fresh answers
3. The same IP is announced from hundreds of locations, so users are routed to the nearest one for low latency and automatic resilience
4. It lets one physical server impersonate many IP addresses

Anycast announces one IP from many PoPs; routing delivers each user to the closest instance, giving low latency worldwide and automatic failover when a location withdraws its announcement.

Building Blocks

• At-Least-Once Delivery: The delivery guarantee you actually get from networks and message queues: a message is delivered one or more times. Retrying clients, re-dispatched load balancer requests, and consumer redeliveries in Kafka/SQS all produce duplicates. Designing for duplicates is not optional — it is the default reality of distributed systems.
• Idempotency Key: A client-generated unique identifier (usually a UUID) attached to a request, typically via an Idempotency-Key header. The server treats all requests carrying the same key as one logical operation. Keys must be unique per logical operation, reused across retries of that operation, and never reused for a different operation.
• Dedup Store: The server-side record mapping idempotency keys to their outcomes: status (in-flight / completed), the stored response, and an expiry. Redis with SET NX and a TTL, or a unique constraint on a database column, are the standard implementations. The unique constraint is what makes two concurrent identical requests safe — one insert wins, the loser waits or replays.
• Safe and Idempotent HTTP Methods: Per the HTTP spec, GET, HEAD, PUT, and DELETE are idempotent — repeating them has the same effect as one call (PUT replaces a resource wholesale; DELETE of an already-deleted resource is a no-op). POST is not idempotent: each POST conventionally creates a new resource, which is why POST endpoints are the ones that need explicit idempotency keys.
• Exactly-Once Myth: No network can guarantee a message is processed exactly once — the two generals problem makes acknowledgment ambiguity fundamental. Systems advertising 'exactly-once' (Kafka, Flink) actually deliver effectively-once semantics via idempotent producers, dedup, and transactional state commits scoped to their own pipeline. The guarantee ends at their boundary; your application still needs idempotent consumers.
• Timeout and Retry Behavior: The client-side trigger for duplicates: a timeout fires, the outcome is unknown, and the client retries — possibly against a different server that never saw the first attempt. Correct behavior pairs retries with exponential backoff, jitter, a bounded retry budget, and a stable idempotency key that survives across every attempt of the same logical operation.

Definitions

Idempotency

The property that executing an operation multiple times produces the same result and side effects as executing it once.

• Idempotency is about the effect on server state and downstream side effects, not about returning identical byte-for-byte responses.
• It is what makes retries, consumer redeliveries, and duplicate webhooks safe.
• An operation that is not naturally idempotent (e.g., 'charge card') can be made idempotent with a key plus a dedup store.

Idempotency Key

A unique client-generated token that identifies one logical operation across all its retries.

• Sent as a header such as Idempotency-Key: 9f0e2... or embedded in the request body.
• Stripe popularized the header-based pattern; its API replays the original response for any repeated key within 24 hours.
• If the same key arrives with a different request payload, well-designed APIs reject it with an error rather than silently returning a mismatched result.

At-Least-Once Delivery

A delivery guarantee where messages may be delivered more than once but are never lost.

• The default for most queues (SQS standard), consumer groups, and retrying HTTP clients.
• Duplicates arise from producer retries, broker re-delivery after consumer timeouts, and network retransmission.
• The consumer-side contract is: deduplicate or make processing idempotent.

Exactly-Once Delivery

The (unattainable in general) guarantee that each message is processed precisely one time.

• Impossible to guarantee end-to-end across arbitrary systems — acknowledgments can always be lost after processing but before the ack arrives.
• What systems actually provide is effectively-once: dedup within a bounded window or transactional coupling of processing and offset commits.
• The practical recipe is always: at-least-once transport plus idempotent processing.

Dedup Store

Persistent storage that records which operation identifiers have already been processed and what their results were.

• Implemented as a Redis key with SET NX EX, or a database table with a unique constraint on the idempotency key column.
• Stores enough state to distinguish 'never seen', 'in flight', and 'completed — here is the stored response'.
• Needs a retention window longer than the maximum plausible retry span; expiry reopens the duplicate window.

Natural Idempotency

Operations that are idempotent by their semantics, without needing keys or dedup stores.

• Examples: 'set user email to X' (PUT), 'delete item 42', 'set balance to 100'.
• Counter-example: 'increment balance by 100' is not idempotent — repeating it changes the outcome.
• Designing APIs around set-to-value instead of apply-delta eliminates entire classes of duplicate bugs.

Patterns

• Idempotency-Key Header on Mutating Endpoints — Any POST endpoint whose retry would create a duplicate side effect — payments, orders, account creation, message sends.
• Unique Constraint as the Enforcer — When the dedup mechanism must survive crashes and concurrent duplicates hitting different app servers.
• Idempotent Consumer for Queues — When consuming from at-least-once systems like SQS, Pub/Sub, or Kafka.
• Set-to-Value API Design — When you control the API contract and want idempotency without any key machinery.

Strategies

• Key-Plus-Stored-Response Replay When: For payment and order APIs where a retry must return exactly what the first attempt returned. How: On first request, insert (key → in-flight) into the dedup store with a unique constraint; execute the operation; atomically update the record to (completed, response body, status code). On any retry with the same key, return the stored response verbatim. A concurrent duplicate that finds 'in-flight' waits briefly and then replays the completed result. Example: Stripe's charge API: first POST stores the key, creates the charge, and saves the full response; a retry after a client timeout returns the saved charge object with an Idempotency-Replayed-style indication, so the merchant never double-charges.
• Retry with Backoff, Jitter, and a Stable Key When: On every client that calls a fallible remote dependency. How: Generate one idempotency key per logical operation before the first attempt and reuse it across all retries. Retry only on timeouts and 5xx (never on 4xx, which will fail identically), with exponential backoff (e.g., 1 s, 2 s, 4 s) plus random jitter, capped at a small attempt budget so you don't amplify an outage. Example: A checkout client generates checkout_uuid when the user taps 'Pay', sends it on every attempt, retries up to 3 times with jittered backoff, and then surfaces 'payment status unknown — check before retrying' instead of blindly retrying forever.
• Unique Constraint + Read-Your-Write Fallback When: When writes go to a relational database and duplicates may arrive concurrently at different app servers. How: Put a unique index on the client-supplied operation ID. Attempt the insert; on constraint violation, SELECT the existing row and return it as the result. The database's uniqueness guarantee replaces any application-level locking. Example: A transfer service inserts (transfer_id, from_acct, to_acct, amount) with a unique transfer_id; a duplicate retry hits the constraint, reads back the original transfer row, and returns it — exactly one ledger entry no matter how many retries.
• Deduplication Window for Event Streams When: When consumers process high-volume event streams where storing every ID forever is too expensive. How: Keep processed message IDs in a bounded store — Redis keys with TTL, or a Bloom filter for very high volume — sized to cover the realistic redelivery window. Accept that events older than the window could theoretically reprocess, and keep downstream operations naturally idempotent as a second line of defense. Example: A notification service consuming from SQS stores each message ID in Redis with a 24-hour TTL; redeliveries within a day are dropped, and the 'send push' action itself targets a device endpoint that tolerates an occasional duplicate.
• Two-Phase Outcome Check for Non-Idempotent Third Parties When: When the downstream system (a bank, a legacy API) does not support idempotency keys. How: Assign your own operation ID, persist 'intent' before calling the third party, and after a timeout query the third party's status/query API (or reconciliation feed) using your reference ID before deciding to retry. Reconciliation jobs sweep ambiguous operations rather than guessing. Example: A payout service stores payout_intent with reference R123 before calling the bank; on timeout it calls the bank's transaction-status endpoint with R123 — retrying only if the bank confirms no such transaction exists.

Tradeoffs

Real World

Quiz

A client sends a payment request and the connection times out. What is the correct assumption?

1. The payment definitely failed and it is safe to retry blindly
2. The payment definitely succeeded and no retry is needed
3. The outcome is unknown — the request may have been processed and only the response lost
4. The server will automatically roll back the payment

A timeout means the response was lost, not that the work was. The request may have executed fully, which is why blind retries cause double charges and idempotency keys exist.

How does a server use an idempotency key to make retries safe?

1. It encrypts the request body with the key so duplicates are unreadable
2. It stores the key with the first request's result and replays that stored result when the same key arrives again
3. It forwards the key to the database so queries run faster
4. It rejects all requests that carry any key

The dedup store records the outcome of the first execution keyed by the idempotency key; retries with the same key receive the stored response instead of re-executing the operation.

Which set of HTTP methods is idempotent by specification?

1. GET, PUT, DELETE
2. GET, POST, PUT
3. POST, PATCH, DELETE
4. Only GET

GET, HEAD, PUT, and DELETE are idempotent — repeating them has the same effect as one call. POST creates a new resource per call, so POST endpoints need explicit idempotency keys.

What is the practical way to achieve 'exactly-once' behavior in a distributed system?

1. Use TCP, which guarantees exactly-once delivery
2. Set very short client timeouts so duplicates cannot happen
3. Ask the network to deduplicate packets
4. Combine at-least-once delivery with idempotent processing or deduplication

Exactly-once delivery is impossible in general because acknowledgments can always be lost; the industry recipe is at-least-once transport plus idempotent consumers or dedup stores, yielding effectively-once results.

Why is a read-then-write check insufficient for handling two concurrent identical requests?

1. Both requests can read 'not processed yet' before either writes, so both execute
2. Databases cannot store idempotency keys
3. Concurrent requests always arrive from the same client
4. Reads are slower than writes, so the second request times out

A race window between the check and the write lets both duplicates pass the check. A unique constraint or atomic SET NX makes one insert win and forces the loser to replay the stored result.