Beginner System design concept · How Systems Actually Work · 22 mins read

Latency & Performance

Break slow requests into network time, database time, and compute time so performance work targets the real bottleneck.

Open the interactive lesson See it in Design Library Management System

Latency Fundamentals

Learn what actually makes a request slow — network hops, disk reads, database queries, and serialization — and build the order-of-magnitude intuition that lets you estimate latency before you ever measure it.

Intuition

A user clicks a button and waits two seconds. Where did the time go? Without a mental breakdown of a request — client to edge, edge to server, server to database, disk reads, response serialization, and the trip back — engineers guess at fixes blindly: they tune code when the real cost is a cross-ocean network hop, or add servers when a single missing database index is the culprit. Guessing is how teams spend weeks 'optimizing' the 5% segment while the 80% segment goes untouched. Latency is the single most user-visible property of a system: Google found that an extra 500 ms of search latency dropped traffic by 20%, and Amazon famously estimated that 100 ms of added latency cost 1% of sales. It is also the property that constrains every architectural decision you will make later — caching, CDNs, and replication all exist primarily to attack latency. Interviewers expect you to decompose a request and quote rough latency numbers from memory because it proves you can reason quantitatively instead of waving your hands.

Mental Model

Every request's total latency is a sum of segments: network round trips (client-to-server and any server-to-server calls), disk or memory access time, CPU compute, database query execution, and serialization of data onto the wire. The dominant insight is that these segments differ by orders of magnitude — a memory read is ~100 nanoseconds, an SSD read ~100 microseconds, and a cross-continent network round trip ~150 milliseconds — so a single unnecessary round trip can outweigh a million memory operations. Optimizing latency means finding the biggest segment first, which is almost always network round trips and disk access, not computation. Think of it like: Think of cooking dinner. Chopping vegetables (CPU compute) takes seconds, but driving to the store for a missing ingredient (network round trip) takes 30 minutes, and ordering an ingredient from another country (cross-ocean request) takes days. A chef who sharpens their knife to chop 10% faster while making three trips to the store per meal is optimizing the wrong thing entirely.

Building Blocks

Network Round Trip Time (RTT): The time for a packet to travel from sender to receiver and back. Same-datacenter RTT is ~0.5 ms; cross-country (e.g., New York to San Francisco) is ~60-80 ms; cross-ocean is 150-300 ms. RTT is bounded by the speed of light in fiber — you cannot engineer it away, only reduce the number of trips or move the endpoints closer.
Memory vs Disk Access: Reading 1 MB sequentially from RAM takes roughly 5-10 microseconds; from an SSD, ~100 microseconds to 1 ms; from a spinning disk with a seek, several milliseconds. This ~1000x gap between RAM and disk is why 'is the data in memory?' is the first performance question for any data store.
Database Query Time: A well-indexed lookup that hits memory-cached pages takes single-digit milliseconds; an unindexed scan of a large table takes seconds to minutes. Query time is dominated by how much data must be read and whether it comes from disk or memory — which is why indexes and query shape matter more than database brand.
Serialization and Deserialization: Converting in-memory objects to bytes (JSON, Protobuf) and back costs CPU time proportional to payload size and format efficiency. Usually a small segment, but it becomes visible at high throughput or with large payloads — megabyte-sized JSON responses can add tens of milliseconds per request just in encode/decode.
CPU Compute Time: The time your application code actually runs: business logic, transformations, template rendering. For most web requests this is the smallest segment — milliseconds or less. It only dominates when you do heavy work per request (image processing, cryptography, large in-memory sorts) or when lock contention serializes threads.
Sequential vs Parallel Segment Composition: Segments that must happen one after another (query the database, then use the result to call another service) add their latencies together; segments that can run in parallel cost only the slowest of them. The number of sequential network hops in a request path is usually the single best predictor of its total latency.

Definitions

Latency

The time between initiating an operation and observing its result, measured from a specific vantage point.

Always ask 'latency as seen by whom?' — client-perceived latency includes network and rendering; server-side latency does not.
Distinct from response time only in pedantic usage; in practice the terms are used interchangeably.
Reported as a distribution (percentiles), never a single average — covered in the next subtopic.

Round Trip Time (RTT)

The time for a signal to travel from source to destination and back again.

Speed of light in fiber is roughly 200,000 km/s, so ~5 ms per 1,000 km of pure propagation — before routing, queuing, and processing overhead.
A TLS handshake alone can cost 1-2 RTTs; chatty protocols multiply RTTs quickly.
RTT is a hard physical floor: no amount of engineering beats physics, only fewer trips or shorter distances help.

Throughput

The number of operations a system completes per unit of time, e.g., requests per second.

The dual of latency: you can often trade one for the other (batching raises throughput but adds latency).
A system can have great average latency at low load and collapse at high load — the two must be evaluated together.
Little's Law connects them: concurrency = throughput x latency.

Latency Numbers Every Engineer Should Know

Jeff Dean's canonical order-of-magnitude reference table for common operations, popularized in a 2009 Google talk.

L1 cache reference ~1 ns; main memory reference ~100 ns; SSD random read ~100 microseconds; datacenter round trip ~0.5 ms; disk seek ~10 ms; cross-ocean packet round trip ~150 ms.
The value is not exact figures but the ratios: memory is ~1000x faster than SSD, which is ~100x faster than a network hop.
Memorize the orders of magnitude and you can estimate any architecture's latency floor in your head.

Chatty Protocol

A communication pattern that requires many sequential request-response exchanges to accomplish one logical operation.

Classic example: N+1 database queries — one query for a list, then one query per list item — turning 1 RTT into N+1.
Over a 60 ms RTT link, 50 sequential round trips cost 3 seconds no matter how fast the server computes.
Fixes are batching, joins, or prefetching — all reduce trip count, not per-trip time.

Cold Start

The elevated latency of the first request to a freshly started or idle component, before caches and connections are warm.

Causes: empty in-memory caches forcing disk reads, unestablished TCP/TLS connections, lazy-loaded code, JIT warmup.
Serverless functions are the famous case — an idle function may take hundreds of milliseconds to initialize.
Mitigations include warm-up traffic, connection pooling, and keeping minimum instances alive.

Co-location

Placing communicating components physically near each other — same rack, datacenter, or region — to minimize network latency between them.

A service and its database in the same availability zone see sub-millisecond RTTs; across regions they see tens of milliseconds.
The cheapest latency optimization is often deployment topology, not code changes.
The extreme form is edge computing: moving compute near the user, which Phase 3 covers via CDNs.

Patterns

Latency Budget per Request — Whenever a feature has a user-facing response-time target and multiple components contribute to it.
Collapse Sequential Hops — When a request makes multiple independent downstream calls one after another.
Eliminate Round Trips (N+1 Fix) — When logs show one page load triggering dozens of similar database queries.
Keep Hot Data in Memory — When profiling shows repeated reads of the same data hitting disk or a remote store.

Strategies

Decompose Before You Optimize When: At the start of any latency investigation. How: Break the request into segments — client network, edge, server compute, each downstream call, database time, serialization — and time each one with logs or tracing. Rank segments by contribution and only then choose where to work. Optimizing the largest segment first yields outsized returns; optimizing a 2% segment can never yield more than 2%. Example: A team about to rewrite their Python service in Go for speed instruments it first and finds 85% of request time is an unindexed database query. A one-line index migration delivers the win; the rewrite is cancelled.
Reduce Distance and Trip Count When: When the network segments dominate the latency budget. How: Attack RTT along two axes: fewer trips (batch calls, join queries, parallelize independent calls, reuse connections) and shorter trips (co-locate services with their databases, serve users from a nearby region, use persistent connections to skip TCP/TLS handshake RTTs). Example: Moving a chatty service from a cross-region database to a same-zone read replica takes 40 sequential calls from 40 ms RTT each (1.6 s total) to 0.5 ms each (20 ms total) — an 80x win from topology alone.
Estimate with Order-of-Magnitude Math When: During design interviews and architecture reviews, before any measurement exists. How: Use the canonical latency table to sanity-check designs: count sequential round trips, multiply by the appropriate RTT, add memory or disk access estimates, and compare against the latency target. If the back-of-envelope math says a design needs 30 cross-region round trips for one click, the design is dead before it is built. Example: Designing a 'like' button: 1 client-to-server RTT (60 ms) + 1 in-memory session check (~0.1 ms) + 1 indexed write (~2 ms) + response — a ~65 ms floor feels instant; no exotic technology needed.
Warm the Cold Path When: When first requests after deploys or idle periods are dramatically slower than steady-state requests. How: Pre-establish what the first request would otherwise pay for: connection pools initialized at startup, cache warm-up scripts that replay hot queries after a deploy, health-check traffic that keeps JIT-compiled code and serverless instances warm. Example: After each deploy, a warm-up script replays the top 100 product-page queries against the new servers before load balancers route users to them — eliminating the post-deploy latency spike that users had been reporting weekly.

Why Geography and Round Trips Dominate Everything Else

Light in fiber moves at about 200,000 km/s — two-thirds its vacuum speed — so a New York to Sydney round trip (~32,000 km of cable) cannot physically return in under ~160 ms, before any routing or processing overhead. This single fact explains most of what looks like 'architecture' in latency discussions. Computation keeps getting faster (memory and CPU improve year over year), but the speed of light is fixed, which means network distance is the one latency segment that never gets cheaper. The consequence shows up everywhere: a single cross-ocean round trip costs more than a thousand SSD reads or a million memory references; a TLS handshake to a faraway server costs more than generating the entire page. This is why mature systems are shaped around trip elimination — connection reuse and HTTP/2 multiplexing to avoid handshake RTTs, data denormalization to avoid sequential queries, regional deployments and (in Phase 3) CDNs and caches to shorten the distance itself. In interviews, the fastest way to demonstrate seniority is to count round trips in a proposed design out loud and multiply by RTT: 'that's 5 sequential calls at 60 ms each, so 300 ms before we compute anything — can we parallelize or batch these?' That one sentence signals quantitative reasoning better than any diagram.

Tradeoffs

Decision	Upside	Downside
Fewer, Larger Requests vs Many Small Requests	Batching collapses many RTTs into one, dramatically cutting total latency on high-latency links and reducing per-request overhead (headers, handshakes, auth checks).	Large payloads serialize slower, one failed item can fail the whole batch, and clients wait for the slowest item in the batch before seeing anything.
In-Memory Data vs On-Disk Durability	Serving from memory is ~1000x faster than disk, and pure in-memory reads make single-digit-millisecond responses routine.	Memory is volatile (lost on restart), expensive per GB, and limited in size — you must choose what earns the memory and how data survives a crash.
Co-located Monolith-Style Calls vs Distributed Services	Same-process or same-host communication is measured in nanoseconds to microseconds, eliminating network RTT, serialization, and failure modes entirely.	Tight co-location couples deployments and scaling — you give up independent scaling and team autonomy, which is why services get split despite the latency cost.
Synchronous Parallel Calls vs Asynchronous Deferred Work	Parallelizing independent calls within a request keeps the user-facing path as short as the slowest dependency, with no change to semantics.	Everything on the synchronous path still blocks the user; deferring work (queues, background jobs) shortens perceived latency further but complicates consistency and error handling — a Phase 3 tradeoff.

Real World

System	How it's used
Google (Jeff Dean's Latency Numbers)	Google's 'Numbers Every Engineer Should Know' table — memory ~100 ns, SSD ~100 us, datacenter RTT ~0.5 ms, cross-ocean RTT ~150 ms — is the industry's shared mental ruler, and Google research showed that adding just 500 ms to search results cut traffic by 20%, proving latency is a business metric, not just an engineering one.
Amazon	Amazon's often-cited internal finding that every 100 ms of added latency costs roughly 1% of sales drove a culture of latency budgets per page component, where each team owns a slice of the page-load time and must justify every millisecond.
Cloudflare	Cloudflare's entire network exists to shorten the network segment: by terminating TLS and serving content from 300+ edge locations near users, it converts 150 ms cross-ocean RTTs into sub-10 ms local ones — geography reduction as a product.
Netflix	Netflix pre-positions video content on ISP-local appliances (Open Connect) so streaming data travels a few network miles instead of crossing continents, and its API layer aggressively batches and parallelizes backend calls so one screen load doesn't fan out into hundreds of sequential round trips.
PostgreSQL (EXPLAIN and Indexes)	PostgreSQL's EXPLAIN command exposes exactly where query time goes — sequential scan vs index scan, disk pages vs cached buffers — embodying the decompose-first discipline at the database layer, where a missing index routinely turns 2 ms lookups into 2-second table scans.

Interview

Questions interviewers ask

A user reports a page takes 2 seconds to load. Walk me through how you figure out where the time is going.
Roughly how long does a main memory read take vs an SSD read vs a cross-country network round trip? Why do these numbers matter for design?
Your service makes 20 sequential database calls per request. How would you reduce latency without changing the database?
Why can't we just make the network faster to fix latency between New York and Tokyo?
Design a 'like' button endpoint and estimate its latency budget out loud.

What a strong answer covers

Candidate should decompose a request into network, compute, database, and serialization segments; quote order-of-magnitude latency numbers (memory ~100 ns, SSD ~100 us, intra-DC RTT ~0.5 ms, cross-ocean ~150 ms) and use them to estimate designs; identify round-trip count and distance as the dominant levers; and propose concrete fixes — batching, parallelization, indexing, co-location — tied to which segment they attack.

Common traps

Jumping to 'add a cache' or 'rewrite in a faster language' before decomposing where the time actually goes.
Not knowing rough latency numbers, which makes every estimate a guess — interviewers use this to test quantitative grounding.
Ignoring sequential round trips: designing flows with 10 dependent network calls without noticing they add up to seconds.
Confusing latency with throughput — saying 'the system is fast because it handles 10,000 RPS' when each request takes 5 seconds.
Treating network latency as solvable with better code; the speed of light is a hard floor, only fewer or shorter trips help.

Quiz

A request spends its time as follows: 10 ms server compute, 5 ms serialization, 1,200 ms in 40 sequential cross-region database calls. Where should you optimize first?

Rewrite the server code to cut the 10 ms compute time
Reduce the number and distance of the database round trips
Switch JSON to a faster serialization format
Add more CPU cores to the server

The database round trips contribute 1,200 ms of the 1,215 ms total — over 98%. Optimizing any other segment can save at most 15 ms; collapsing trips (batching, co-location, parallelizing) attacks the actual bottleneck.

Approximately how much slower is a cross-ocean network round trip compared to a main memory read?

About 10x slower
About 100x slower
About 1,000x slower
About 1,000,000x slower

A memory reference is ~100 nanoseconds; a cross-ocean RTT is ~150 milliseconds — a ratio of about a million to one. This is why a single unnecessary round trip outweighs enormous amounts of in-memory computation.

Why is the speed of light fundamental to latency discussions in system design?

It sets a hard physical floor on network round-trip time that no engineering can beat — only fewer or shorter trips help
It determines how fast CPUs can execute instructions
It limits how quickly SSDs can read data
It only matters for satellite internet, not fiber networks

Light in fiber travels ~200,000 km/s, so a New York–Sydney round trip cannot physically return in under ~160 ms. Since computation keeps getting faster but physics doesn't, distance is the one latency segment that never gets cheaper.

A page load triggers 1 query to fetch orders, then 1 query per order to fetch line items (50 orders). What is this pattern called and what is the fix?

Connection pooling; fix it by reusing TCP connections
Cold start; fix it with warm-up traffic
Fan-out; fix it with a load balancer
N+1 queries; fix it with a single JOIN or batched query

The N+1 pattern turns one round trip into 51 sequential ones. A JOIN (or an IN-batch query) collapses them into a single trip, often cutting latency by an order of magnitude without touching any other part of the system.

Three independent downstream calls each take 60 ms. What is the approximate downstream latency if they run sequentially vs in parallel?

180 ms sequential vs ~60 ms parallel
60 ms in both cases
180 ms in both cases
60 ms sequential vs 180 ms parallel

Sequential segments add together (3 x 60 = 180 ms); parallel segments cost only the slowest of them (~60 ms). For independent calls, parallelization is a free ~3x latency win with no changes to the services themselves.

Percentiles and Tail Latency

Learn why average latency is a dangerous illusion, what p50/p95/p99 actually measure, and how tail latency compounds across fan-out so that your slowest 1% of requests defines the user experience.

This section is part of the full PRISM roadmap, with worked examples, trade-off tables, interview questions and a quiz.

Unlock the full lesson

Finding Bottlenecks

Learn the measure-before-optimize discipline: how to decompose a request into timed segments, use logs, traces, and slow query logs to locate the real bottleneck, and avoid the false fixes that waste engineering time.