Introduction & Fundamentals

Building Blocks

• Functional Requirements (FRs): FRs define what the system must do from the user's point of view. They describe features, actions, and outcomes: users can post tweets, send messages, upload photos, search products, or process payments. A complete FR list also identifies who performs each action and any special roles such as admins, guests, or external systems. FRs bound the scope of the design and prevent endless feature creep.
• Non-Functional Requirements (NFRs): NFRs define how the system must behave while delivering features. They include latency targets (e.g., P99 < 200 ms), availability (e.g., 99.99% uptime), throughput (e.g., 100k requests/sec), durability (e.g., zero unplanned data loss), consistency guarantees, and security posture. NFRs are the primary drivers of architectural choices such as caching, replication, partitioning, and queueing.
• Constraints: Constraints are hard limits that bound the design space. Time-to-market, budget, team expertise, regulatory compliance (GDPR, PCI-DSS, HIPAA), existing tech stack, vendor relationships, and data residency laws all constrain what is practical. A design that ignores constraints is a design that will not ship.
• Components & Interfaces: Components are the building pieces — clients, APIs, load balancers, services, databases, caches, queues, and object stores. Interfaces are the contracts between them: REST or gRPC endpoints, event schemas, database schemas, and message formats. Components should be chosen after requirements and constraints are clear, not before.
• Failure Modes: Failure modes describe what happens when hardware fails, networks partition, traffic spikes, dependencies go down, or deployments go wrong. Good system design anticipates failure rather than hoping it never happens. Techniques include retries with backoff, circuit breakers, fallbacks, replication, and graceful degradation.
• Scale & Lifecycle: Systems evolve. A design that works for 1,000 users may fail at 1,000,000. Good system design plans for the next 10× and 100× milestones, chooses data models that can be partitioned, and builds observability and operability in from the start. It also plans for deprecation and migration, because no design is final.

Definitions

System

A set of software and hardware components that work together to provide value to users or other systems.

• A system has boundaries: it accepts input, processes it, stores state, and produces output.
• Systems can be monolithic (one codebase) or distributed (many services communicating over a network).
• The behavior of a system emerges from the interaction of its parts, not just the parts themselves.

Architecture

The high-level structure of a system: the major components, their responsibilities, and how they communicate.

• Architecture answers 'what are the big boxes and arrows?' before diving into code.
• Good architecture makes the system understandable, testable, and changeable.
• Architecture is a verb as much as a noun: it is the ongoing act of making structural decisions.

Component

A modular part of a system with a well-defined responsibility and interface.

• Examples: an API gateway, a payment service, a user database, a cache cluster.
• Components should be replaceable independently when their interface is stable.
• The right component boundaries reduce coupling and enable independent scaling.

Interface

The contract that defines how two components interact.

• Examples: HTTP endpoints, gRPC services, message queue topics, database tables.
• Interfaces expose capabilities and hide implementation details.
• Changing an interface often affects many consumers, so design it carefully up front.

Scalability

The ability of a system to handle growing load by adding resources without a proportional loss in performance.

• Scalability is about behavior under growth, not absolute speed.
• Horizontal scalability means adding more machines; vertical scalability means using a bigger machine.
• Not every system needs to scale infinitely; match scalability targets to realistic business growth.

Reliability

The probability that a system operates correctly over a specified period under specified conditions.

• Reliability is often expressed as availability (e.g., 99.9%, 99.99%).
• Reliable systems tolerate component failures through redundancy and automatic recovery.
• Reliability also includes data durability and correctness, not just uptime.

Fault Tolerance

The ability of a system to continue operating, possibly in a degraded mode, when parts of it fail.

• Fault tolerance requires detecting failures quickly and rerouting work around them.
• Examples: database replicas, multi-region deployments, circuit breakers, fallbacks.
• Perfect fault tolerance is expensive; decide which failures are worth surviving.

Patterns

• Requirements-First Design — Always. Before naming any component, write down FRs, NFRs, and constraints.
• Layered Architecture — When you want clear separation of concerns and independent scaling of read, write, and compute paths.
• API-First Thinking — When multiple clients or teams will consume the system.
• Graceful Degradation — When partial functionality is better than total failure.

Strategies

• Start with the User Story When: When the system has clear user-facing actions and you need to surface hidden requirements. How: Write down the sequence of actions a user takes end-to-end. Each action becomes a candidate API, service, or data store. This keeps the design grounded in real behavior rather than abstract technology. Example: For a food delivery app, trace 'customer opens app → searches restaurants → places order → tracks delivery → pays'. Each step reveals services (search, order, tracking, payment) and data flows.
• Draw the Data Flow When: When you need to communicate the design and find bottlenecks. How: Sketch how data moves from client to backend, where it is stored, where it is transformed, and where it is read. Annotate expected request rates and data sizes. Example: A chat message flows: client → gateway → message queue → persistence service → fan-out service → recipient. Drawing this exposes the queue as a critical path.
• Identify the Read/Write Split When: When workloads are unevenly split between reads and writes. How: Count reads and writes per entity. If reads dominate, add caches and replicas. If writes dominate, optimize the write path and consider partitioning. If both are high, separate read and write models. Example: A product catalog is read-heavy: cache aggressively. An order system is write-heavy: shard by customer and use a high-write database.
• Design for Failure from Day One When: When the system must be reliable enough for production use. How: For every component, ask: 'What happens if this fails?' Add retries, fallbacks, redundancy, or circuit breakers where the impact is high. Keep blast radius small. Example: If the payment provider is unreachable, queue the request and retry with idempotency keys rather than dropping the order.
• Iterate Under Constraints When: When the perfect design is impossible due to budget, time, or team size. How: Build the simplest design that meets current requirements and can evolve. Document known limitations and the trigger points that justify the next architecture change. Example: A startup may begin with a managed SQL database and a single API service, knowing it will shard once write throughput exceeds 1,000/sec.

Tradeoffs

Real World

Quiz

Which of the following best describes system design?

1. Making deliberate architectural decisions to meet requirements within constraints
2. Drawing boxes for load balancers and databases
3. Writing code before defining requirements
4. Choosing the newest technologies available

System design is about making justified architectural decisions, not just picking components or chasing trendy tech.

What should be clarified before choosing system components?

1. Functional requirements, non-functional requirements, and constraints
2. The programming language only
3. The cloud provider only
4. The color scheme of the UI

Requirements and constraints must come first because they determine which components make sense.

A Non-Functional Requirement describes:

1. How the system should behave, such as latency and availability
2. What features users can see
3. The exact database schema
4. The name of the product

NFRs describe qualities like latency, throughput, availability, and durability, not visible features.

Why is it risky to choose components before clarifying requirements?

1. The components may not solve the actual problem or fit the constraints
2. Components are always more expensive than requirements
3. Requirements are never important in interviews
4. Components cannot be changed later

Components should be selected to satisfy specific requirements and constraints. Choosing them first often leads to over-engineering or mismatched capabilities.

Fault tolerance means:

1. The system continues operating, possibly in a degraded mode, when parts fail
2. The system never has any bugs
3. All components are on a single server for simplicity
4. Developers manually fix every failure

Fault-tolerant systems detect failures and keep working, often by rerouting work or using redundant components.

Building Blocks

• Clarify Requirements: Separate what the system must do (functional), how well it must do it (non-functional), and what limits you (constraints). Ask clarifying questions, write them down, and confirm them with stakeholders. Every later decision should be traceable to a requirement or constraint.
• Back-of-the-Envelope Estimation: Convert requirements into rough numbers: daily active users, requests per second, storage growth, and bandwidth. These estimates tell you whether a single server is enough or whether you need sharding, replication, and caching. They also show the interviewer that you think quantitatively.
• API / Interface Design: Define the endpoints, request/response shapes, idempotency keys, and error codes. APIs are the contract between clients and the system and often reveal hidden data-model needs. Keep APIs minimal at first and expand them as requirements clarify.
• Data Model: Decide what entities exist, how they relate, and how they are accessed. Schema choice influences whether you use a relational database, document store, key-value store, or wide-column store. Consider read/write patterns, query shapes, and growth of data size.
• High-Level Design: Sketch the major components — clients, DNS, CDN, load balancers, services, caches, databases, queues — and the data flow between them. Keep it simple enough to explain in two minutes. Use the user journey to validate that every step is covered.
• Detailed Deep Dive: Zoom into the parts most likely to break: hot shards, single points of failure, consistency needs, or latency hotspots. Propose concrete solutions and their tradeoffs. Let the interviewer guide which area to explore.
• Bottleneck & Tradeoff Analysis: Identify what limits the system under load and describe the tradeoffs of fixing it. Show that you can reason about cost, complexity, reliability, and operational burden. End with a clear recommendation and the assumptions behind it.

Requirements

• {'text': 'Functional Requirements', 'items': ['What are the core features? (e.g., post a tweet, send a message, create a short URL)', 'Who are the users and what actions can they take?', 'Are there admin, internal, or machine-only operations?', 'What integrations are needed? (payments, identity, third-party APIs)', 'What is the expected read/write ratio for the main entities?']}
• {'text': 'Non-Functional Requirements', 'items': ['Latency: target P50 / P95 / P99 response times for reads and writes.', 'Availability: target uptime (e.g., 99.9%, 99.99%) and acceptable maintenance windows.', 'Throughput: peak requests per second, writes per second, messages per second.', 'Durability: how long must data survive, and what backup / recovery guarantees are required?', 'Consistency: is strong consistency required, or is eventual consistency acceptable for some paths?', 'Scalability: expected user growth over 6–12 months and data growth over a year.', 'Security & compliance: authentication, authorization, encryption, GDPR, PCI-DSS, HIPAA.']}
• {'text': 'Constraints', 'items': ['Budget: monthly infrastructure spend, team size, and time-to-market.', 'Technology: existing languages, frameworks, cloud provider, or on-premise hardware.', 'Operational: monitoring, on-call maturity, deployment frequency, and rollback capability.', 'Regulatory: data residency, retention policies, audit logging, and deletion requirements.']}

Capacity Estimation

Capacity Estimation

• A single relational database can absorb ~230 writes/sec but will struggle at 2,300/sec peak and 23,000 reads/sec → add read replicas and a cache layer.
• 35 TB/year of raw data exceeds what one disk can hold → partition or shard data by time or tenant.
• 90 Mbps peak fits one server but leaves no headroom → design for horizontal scaling and place static assets on a CDN.
• These numbers are order-of-magnitude; use them to choose the architecture, not to size the final invoice.

Definitions

Functional Requirement (FR)

A statement of what the system must do for its users.

• FRs describe features, actions, and outcomes, such as 'users can send messages' or 'admins can ban accounts'.
• They are usually written from the user's perspective.
• FRs define the scope of the design and help avoid scope creep.

Non-Functional Requirement (NFR)

A statement of how the system must behave while delivering functionality.

• NFRs include latency, throughput, availability, durability, consistency, and security.
• They drive architecture far more than FRs do.
• NFRs must be measurable; vague targets like 'fast' or 'reliable' are not useful.

Constraint

A limitation that restricts the set of acceptable designs.

• Constraints can be budgetary, technical, regulatory, or organizational.
• They force pragmatic choices over theoretical perfection.
• Ignoring constraints leads to designs that cannot be built or operated.

Back-of-the-Envelope Estimation

A quick calculation used to understand the rough scale of a problem.

• Estimations convert requirements into numbers such as RPS, storage, and bandwidth.
• They are intentionally imprecise but must be within an order of magnitude.
• They guide architectural decisions and expose unrealistic assumptions.

API Contract

The agreed interface between a client and a service.

• It includes endpoints, request/response formats, error codes, and versioning rules.
• A stable contract lets teams evolve implementations independently.
• API design should consider idempotency, pagination, and rate limiting.

Data Model

The structure of data entities, their relationships, and access patterns.

• The data model influences database choice, query performance, and partitioning strategy.
• A good model reflects how data is read and written, not just how it is stored.
• Data models often evolve, so migrations and backward compatibility matter.

Patterns

• Start with the User Journey — When the system has clear user-facing actions.
• Quantify Before You Architect — When scale is uncertain or a key concern.
• Fail-First Thinking — When reliability matters.

Strategies

• State the Requirements Out Loud When: At the very beginning of any design discussion. How: Write down FRs, NFRs, and constraints explicitly. Ask the interviewer or stakeholders for missing details. This demonstrates structured thinking and prevents you from solving the wrong problem. Example: 'We need a URL shortener. Let me confirm: expected 100M short links/day, read:write ratio 10:1, P99 redirect < 100ms, links never expire, and budget is moderate.'
• Estimate with Round Numbers When: Immediately after requirements are clear. How: Use simple arithmetic: DAU × actions/day ÷ seconds in a day. Round to one significant digit. The goal is an order-of-magnitude answer, not a financial forecast. Example: 10M DAU × 10 posts/day = 100M posts/day ≈ 1,200 writes/sec average, ~12,000/sec peak. Storage ≈ 100M × 500 bytes = 50 GB/day.
• Design APIs Before Storage When: When the system has external clients or multiple internal consumers. How: Define the endpoints and payload shapes first. The API reveals what data you need, how it is accessed, and where concurrency or idempotency matter. Example: For a booking system, defining POST /bookings, GET /bookings/:id, and PATCH /bookings/:id/cancel shows you need order state, inventory locks, and cancellation history.
• Follow the Interview Funnel When: During a timed system design interview. How: Spend the first 2–3 minutes on requirements, 2–3 minutes on estimation, 5 minutes on API/data model, 5–7 minutes on high-level design, and the rest on deep dives and tradeoffs. Do not get stuck in one layer. Example: If asked to design a news feed, confirm requirements, estimate posts/sec, sketch a feed service, then deep dive into fan-out vs fan-in feed generation.
• Communicate Tradeoffs Explicitly When: Whenever you make a major architectural choice. How: State what you are gaining, what you are giving up, and under what conditions you would change the decision. This shows judgment, not just knowledge. Example: 'I chose eventual consistency for the news feed because it scales reads, accepting that new posts may take a few seconds to appear. If the product requires real-time updates, we would switch to push-based fan-out.'

Tradeoffs

Real World

Quiz

What is the first step in a structured system design process?

1. Clarify requirements and constraints
2. Choose a database
3. Draw a load balancer
4. Write pseudo-code

Requirements and constraints must be clarified before any architectural decisions are made.

Which statement best describes a non-functional requirement?

1. P99 latency must be under 200ms
2. Users can post tweets
3. The system supports OAuth login
4. Admins can ban accounts

Latency targets are non-functional requirements — they describe how the system behaves, not what feature it provides.

Why are back-of-the-envelope calculations useful?

1. They give rough scale estimates that guide architecture choices
2. They replace detailed design
3. They prove the final design is correct
4. They are only useful in interviews

Rough numbers for traffic, storage, and bandwidth help you decide whether a simple or distributed design is needed.

Which part of the design should you usually deep-dive into first?

1. The components most likely to fail or bottleneck under load
2. The prettiest UI element
3. The deployment pipeline
4. The logging format

Focus on the riskiest, highest-impact parts of the system first.

In a system design interview, when should you state tradeoffs?

1. Whenever you make a major architectural choice
2. Only at the very end
3. Only if the interviewer asks
4. Never; tradeoffs show weakness

Stating tradeoffs demonstrates judgment and shows that your choices are deliberate, not accidental.

Building Blocks

• Latency: The time to complete one operation. Covered in depth in the Latency vs Throughput subtopic.
• Throughput: The number of operations completed per unit of time. Covered in depth in the Latency vs Throughput subtopic.
• Concurrency: How many operations are in flight at the same time; higher concurrency exposes contention and resource limits.
• Resource Utilization: How efficiently CPU, memory, disk, and network are used; high utilization leaves little headroom for traffic spikes.
• Vertical Scaling (Scale Up): Making a single server more powerful; simple but has hardware limits and can become a single point of failure.
• Horizontal Scaling (Scale Out): Adding more servers and distributing load; improves scalability and fault tolerance but adds coordination complexity.
• Caching: Storing frequently accessed data closer to consumers; reduces repeated expensive work at the cost of freshness.
• Load Balancing: Distributing traffic across servers to prevent bottlenecks and enable horizontal scaling.
• Elasticity: Automatically adding or removing resources based on demand, common in cloud auto-scaling groups.
• Efficiency: Useful work produced per unit of resource; better algorithms and fewer round trips improve it.

Definitions

Performance

How quickly and cheaply a system completes a single unit of work.

• Measured by latency, throughput per resource, CPU usage, and memory footprint.
• Performance optimization usually targets the hot path: the small percentage of code or data that dominates execution time.
• Improving performance does not automatically improve scalability.

Scalability

The ability of a system to maintain acceptable performance as load grows by adding resources.

• A scalable system can grow with user demand without a complete redesign.
• Scalability is usually measured by how throughput or latency changes as nodes are added.
• Perfect linear scalability is rare because coordination overhead grows with scale.

Bonus Points

• Concurrency: The number of operations active at the same time.
• Elasticity: The ability to automatically scale resources up and down based on real-time demand.
• Saturation: The point at which a resource cannot accept more work without degrading.

Patterns

• Optimize the Hot Path — When a small fraction of code or data dominates response time.
• Partition Workload — When no single server can handle all requests or data.
• Add Read Replicas — When reads dominate writes and slight replication lag is acceptable.
• Asynchronous Processing — When work does not need an immediate response and can be offloaded to background workers.
• Auto-Scaling — When load varies predictably or unpredictably and you want to match capacity to demand.

Strategies

• Measure Before Optimizing When: Before any performance or scalability work. How: Use profiling, metrics, and load tests to identify the actual bottleneck. Optimizing the wrong thing is a common and expensive mistake. Example: A team thought their database was slow, but profiling showed 80% of latency came from an unbatched loop calling a third-party API. Batching the calls fixed it with no database changes.
• Build Stateless Services When: When you need to scale compute horizontally. How: Store session and state outside the service (in a cache or database). Any server can handle any request, so load balancers can distribute traffic evenly. Example: Web servers in an e-commerce app do not store shopping carts in memory; carts live in Redis. New servers can be added during flash sales without moving state.
• Cache at Multiple Tiers When: When read latency is critical and data access patterns are predictable. How: Use browser caches, CDN edge caches, in-memory application caches, and database buffer pools. Each tier trades freshness for speed. Example: A news site caches static assets at the CDN, article HTML at the edge, and trending articles in Redis, reducing database load by 95%.
• Shard and Replicate Databases When: When a single database becomes the bottleneck. How: Partition data across multiple primary databases (sharding) and add read replicas to offload queries. Be aware of cross-shard queries and replication lag. Example: A SaaS platform shards tenant data by region. Each region has primary and replica databases, keeping query latency low and throughput high.
• Offload Work to Queues When: When tasks can be deferred without hurting the user experience. How: Accept the request, publish a message to a durable queue, and acknowledge the user immediately. Background workers process the queue at their own pace. Example: An image upload returns a pending URL immediately; a worker resizes the image and updates the URL when finished. This prevents upload spikes from overwhelming the API.

Tradeoffs

Real World

Quiz

What is the primary difference between performance and scalability?

1. Performance is about single-request speed; scalability is about handling growing load
2. Performance is for databases; scalability is for servers
3. They are measured the same way
4. Scalability always improves performance

Performance concerns how fast one request completes. Scalability concerns how the system behaves as concurrent load increases.

A single powerful database server answers queries in 50ms but maxes out at 1000 concurrent queries. This is:

1. High performance, low scalability
2. Low performance, high scalability
3. High performance, high scalability
4. Low performance, low scalability

50ms latency is high performance, but the hard limit at 1000 queries shows low scalability.

Which technique improves scalability by spreading data across multiple servers?

1. Sharding
2. Increasing CPU on one server
3. Inlining code
4. Removing logs

Sharding partitions data across servers so that no single node must store or serve everything.

Adding more servers behind a load balancer primarily improves:

1. Scalability and fault tolerance
2. Single-request latency for every user
3. Database consistency
4. Code readability

Horizontal scaling adds capacity and redundancy. It can keep latency stable under load but does not inherently make an individual request faster.

Which of these is a performance optimization rather than a scalability optimization?

1. Caching frequently accessed data in Redis
2. Sharding a database
3. Adding auto-scaling groups
4. Replicating data across regions

Caching primarily reduces latency for individual requests. Sharding, auto-scaling, and cross-region replication primarily improve scalability or availability.

Building Blocks

• P50 / P95 / P99 Latency: Percentile latencies describe the experience of most users. P50 is the median: half of requests are faster, half are slower. P95 and P99 tell you the worst-case experience for 95% or 99% of requests and are usually more meaningful than the average. A system with a low average but high P99 can still frustrate many users.
• Requests Per Second (RPS): A common throughput measure for online services. RPS depends on request complexity, resource usage, and concurrency limits. It is often paired with a latency target, such as '10,000 RPS at P99 < 200 ms'.
• Bandwidth: The maximum data rate the network can carry, usually measured in bits per second. Bandwidth is often the throughput bottleneck for large payloads such as video, bulk uploads, or big query results.
• Queuing: When requests arrive faster than they can be processed, they wait in a queue. Queue depth and queuing delay are key indicators of overload. Once a queue starts growing, latency increases nonlinearly and the system can enter a death spiral.
• Parallelism: Doing multiple things at once to reduce latency or increase throughput. Examples include multi-threading, async I/O, and distributed processing. Parallelism is powerful but adds coordination overhead and can expose race conditions.
• Batching: Grouping work together to amortize overhead such as network round trips, disk writes, or database transactions. Batching increases throughput but usually adds latency for individual items because the batch must fill before it is sent.
• Little's Law: In a stable system, average concurrency (L) equals average throughput (λ) multiplied by average latency (W): L = λ × W. This law shows why adding latency without dropping throughput increases the number of in-flight requests and can exhaust resources.
• Tail Latency: The slowest requests (P95, P99, P99.9) often dominate user experience. A single slow downstream call in a fan-out can make the overall request slow even when most calls are fast. Tail latency is especially important in microservices.
• Saturation: The point where a resource cannot process more work without degrading. Saturated resources create queues and tail latency. Monitoring utilization and queue depth helps detect saturation before users notice.

Definitions

Latency

The time delay between initiating a request and receiving a useful response.

• Latency includes network transit, processing, queueing, and disk access time.
• It is usually reported as percentiles because a single slow path can ruin an average.
• Interactive applications typically target P99 latency to bound worst-case user experience.

Throughput

The number of operations a system can complete in a given period of time.

• Throughput is often measured in requests per second, transactions per second, or bytes per second.
• It depends on both the speed of each operation and how many can run concurrently.
• Throughput optimization often sacrifices the latency of individual operations.

Bandwidth

The maximum rate at which data can be transferred over a network or bus.

• Bandwidth is a capacity limit, not a latency guarantee.
• A high-bandwidth link can still have high latency if the distance is large.
• Large payloads and streaming workloads are often bandwidth-bound.

Concurrency

The number of operations that are active at the same time.

• Higher concurrency can increase throughput up to a point.
• Beyond that point, contention for locks and resources reduces throughput and raises latency.
• Concurrency is managed with thread pools, connection pools, and async runtimes.

Saturation

The condition in which a resource is fully utilized and additional work must wait.

• Saturation is visible as rising queue depth and increasing latency.
• Common saturated resources include CPU, memory, disk I/O, network, and database connections.
• The only healthy response to saturation is to add capacity or reduce load.

Tail Latency

Latency at the high percentiles of a distribution, such as P99 or P99.9.

• Tail latency captures the experience of the unlucky few requests.
• In microservices, tail latency amplifies because a request may depend on multiple services.
• Techniques such as hedged requests, deadline propagation, and circuit breakers combat tail latency.

Little's Law

A fundamental queueing theory result: average concurrency equals average throughput times average latency.

• Formula: L = λ × W, where L is average in-flight work, λ is throughput, and W is latency.
• It applies to stable systems where arrival and completion rates are balanced.
• It explains why holding latency constant while increasing throughput requires more concurrency.

Patterns

• Caching for Latency — When a small set of data is requested frequently and staleness is acceptable.
• Batching for Throughput — When per-request overhead is high and slight delay is acceptable.
• Load Shedding — When demand exceeds capacity and latency would otherwise spiral.
• Connection Pooling & Keep-Alive — When many short-lived requests pay a high cost to open TCP/TLS connections.
• Request Coalescing — When multiple identical requests arrive at the same time.

Strategies

• Reduce Round Trips When: When latency is dominated by network calls. How: Combine requests, use GraphQL or field masks, denormalize data, and keep related data close together. Each network hop adds milliseconds and failure modes. Example: A mobile app fetches a user's profile, settings, and notifications in one request instead of three, cutting startup latency by half.
• Use Asynchronous I/O When: When a service waits on many slow downstream calls. How: Use event loops or coroutines to run many concurrent operations on a small thread pool. This keeps CPU usage low while handling high concurrency. Example: A gateway fanning out to 20 microservices uses async HTTP clients; the total latency is close to the slowest call rather than the sum of all calls.
• Partition Data and Traffic When: When a single data set or node becomes a bottleneck. How: Split data by user, region, or time range. Route traffic to the right partition so that no single node must handle everything. Example: A messaging app partitions conversations by chat ID. Each partition handles a subset of chats, allowing throughput to scale linearly with partitions.
• Compress Payloads When: When bandwidth is the bottleneck or payloads are large. How: Use gzip, zstd, or protobuf to reduce bytes on the wire. Smaller payloads reduce bandwidth usage and often improve latency. Example: An analytics API returns compressed JSON. Payload size drops by 80%, and response times improve because fewer packets are transmitted.
• Shed Non-Critical Load When: During overload to protect core functionality. How: Drop or defer low-priority requests before they saturate critical paths. Set rate limits and prioritize queues. Example: During a flash sale, an e-commerce site disables personalized recommendations but keeps search, cart, and checkout running.

Tradeoffs

Real World

Quiz

Latency measures _____, while throughput measures _____.

1. Time for one request; number of requests per unit time
2. CPU usage; memory usage
3. Database rows; cache hits
4. Read speed; write speed

Latency is the duration of a single operation. Throughput is how many operations the system handles per second.

Which technique usually reduces latency for frequently accessed data?

1. Caching
2. Batching
3. Adding more queues
4. Increasing payload size

Caching serves data from fast storage close to the user, reducing the time to fetch it.

Which technique increases throughput but may add latency to individual items?

1. Batching
2. Caching
3. Reducing payload size
4. Removing indexes

Batching groups work together to amortize overhead, which improves overall throughput but can delay each individual item.

According to Little's Law, if throughput stays the same but latency doubles, what happens to average concurrency?

1. It doubles
2. It stays the same
3. It halves
4. It becomes zero

L = λ × W. If throughput (λ) is constant and latency (W) doubles, the average number of in-flight requests (L) also doubles.

Why is P99 latency usually more important than average latency for user-facing systems?

1. It reflects the experience of the slowest common requests, which users remember
2. It is easier to calculate
3. It is always lower than the average
4. It measures server CPU usage

A low average can hide a tail of slow requests. P99 shows the worst experience for 99% of users and is a better quality target.