Scaling Systems

Building Blocks

• Interchangeable app instances behind one stable endpoint. Any instance should be able to serve the same request shape, so scaling adds capacity instead of special cases.
• A load balancer with health checks. It spreads traffic across the fleet and removes bad instances before they keep hurting users.
• Externalized state such as Redis sessions and object storage. Once state stops living on one box, users can move between instances safely.
• Autoscaling tied to demand signals like request rate, queue depth, or latency. It matters only if new instances can start fast enough to help during a spike.

Definitions

Horizontal Scaling

Adding more servers and sharing traffic across them so total capacity grows with instance count.

• It improves both throughput and fault tolerance at the app tier.
• It works best when servers are interchangeable.

Stateless Service

A service whose next request can be handled by any healthy instance because important state is stored elsewhere.

• Local memory or disk should not be required for user continuity.
• Statelessness is the main enabler for clean scale-out.

Load Balancer

The entry point that forwards requests to one backend from a pool of healthy backends.

• It hides server churn from clients behind one stable address.
• It is what makes many instances feel like one service.

Autoscaling

Automatically changing the number of instances based on measured demand or pressure.

• It absorbs normal traffic variation without manual intervention.
• It does not fix slow code or a saturated database.

Patterns

• Scale the stateless web or API tier first; it is usually the easiest place to add more capacity safely.
• Move sessions, uploads, and long-running work out of the instance before adding more servers.
• Keep one stable front door so instances can be added or removed without changing clients.

Strategies

• Externalize the blockers to scale-out first: sessions to Redis, files to object storage, and background work to queues.
• Scale on demand signals that map to user pain, such as sustained latency or queue depth, not CPU alone.
• Check downstream limits before celebrating a larger fleet, because the bottleneck often shifts to the database or another dependency.

Tradeoffs

• Compared with vertical scaling, scale-out gives elasticity and redundancy, but it adds more moving parts and network hops.
• Externalized state enables interchangeable servers, but it introduces new dependencies whose latency and availability now matter.
• Autoscaling saves cost during quiet periods, but poor thresholds can scale too late or flap during noisy traffic.

Real World

• Netflix keeps web and API tiers horizontally scalable so instance failures and traffic bursts are absorbed by fleets instead of one oversized server.
• Shopify adds application capacity during flash-sale events while keeping shared state outside the app node so any healthy instance can serve the next request.

Quiz

What is the main architectural precondition for scaling an application tier horizontally?

1. Using a larger VM type
2. Keeping the service stateless or moving state outside the instance
3. Running on bare metal only
4. Using SQL instead of NoSQL

If requests depend on local instance memory or disk, adding more servers creates inconsistent behavior. Horizontal scaling works cleanly when any instance can handle any request.

Why is a load balancer typically placed in front of horizontally scaled app servers?

1. To distribute traffic across healthy instances behind one stable endpoint
2. To make every request strongly consistent
3. To replace the database primary
4. To eliminate all network latency

The load balancer hides backend churn from clients and spreads requests across the server pool. Without it, clients would need to track instance addresses and failures themselves.

Which statement best describes vertical scaling?

1. Adding more instances behind a dispatcher
2. Splitting the database into shards
3. Moving sessions into Redis
4. Making one machine bigger by adding more CPU, RAM, or bandwidth

Vertical scaling upgrades one node instead of increasing node count. It is simple, but it has cost and hardware ceilings.

An app stores login sessions only in process memory. What is the biggest risk after adding more app instances?

1. TLS will stop working
2. The load balancer cannot perform health checks
3. Users may appear logged out when their next request lands on a different server
4. CPU usage will always decrease

Local-only session state ties a user to one machine. Once requests spread across multiple servers, follow-up requests may miss the in-memory session entirely.

What is the best way to think about autoscaling?

1. A replacement for performance tuning
2. A way to reduce the need for monitoring
3. A mechanism to adjust instance count to changing load
4. A guarantee that the database will never bottleneck

Autoscaling adds or removes capacity as demand changes. It helps with elastic load, but it does not fix inefficient code or slow downstream systems.

Building Blocks

• A reverse proxy as the public-facing hop. It can terminate TLS, normalize headers, and apply HTTP policies before traffic reaches services.
• A load-balancing function that chooses one healthy backend from a pool. That choice is about distribution and failover, not broader HTTP behavior.
• HTTP-aware features such as routing, caching, and compression. These belong to the proxy role even when the same product also balances.
• Clear tier boundaries when edge processing and backend fan-out need different ownership or scaling profiles.

Definitions

Reverse Proxy

A server that accepts client requests on behalf of internal services and can shape HTTP behavior before forwarding.

• Common jobs include TLS termination, routing, header changes, compression, and caching.
• It gives clients one controlled public entry point.

Load Balancer

A component whose core job is to distribute traffic across multiple backends.

• It combines backend selection with health awareness.
• It may exist inside a reverse proxy or as its own tier.

TLS Termination

Decrypting HTTPS at the edge so certificate handling and handshake cost are centralized.

• It simplifies cert rotation and policy enforcement.
• It also enables HTTP-aware routing and caching decisions.

Combined Tier

A deployment where one product handles both reverse-proxy features and load balancing in the same hop.

• This is common with NGINX, Envoy, and HAProxy.
• It is simple, but it concentrates more work in one place.

Patterns

• Use one combined proxy-plus-balancer tier when traffic is moderate and one layer can comfortably handle edge processing and backend distribution.
• Split the roles when TLS, caching, WAF policy, or routing logic scales differently from backend fan-out.
• Route different URL spaces to different backend pools through the reverse-proxy role, then balance within each pool.

Strategies

• Start combined for simplicity, then separate only when you have a concrete scaling, security, or ownership reason.
• Keep client-facing HTTP concerns conceptually distinct from backend-selection logic, even if one product does both jobs.
• Watch proxy CPU and memory closely: compression, cache misses, and TLS can make the edge tier the first bottleneck.

Tradeoffs

• A combined tier reduces hops and operational overhead, but it also concentrates more CPU work and blast radius in one layer.
• Separate tiers create cleaner ownership and scaling boundaries, but they add cost, latency, and more failure points.
• Centralized TLS and caching can reduce backend load, but both consume resources on the proxy tier itself.

Real World

• Cloudflare acts as a reverse proxy at the edge by terminating TLS, caching assets, and applying HTTP protections before origin systems see the traffic.
• NGINX is often deployed as a combined reverse proxy and load balancer, using one layer to route requests and spread them across app servers.

Quiz

Which responsibility is most closely associated with a reverse proxy rather than load balancing specifically?

1. Distributing requests across three app servers
2. Removing unhealthy instances from a backend pool
3. Terminating TLS and applying HTTP compression
4. Selecting a backend based on pool membership

TLS termination and compression are classic reverse-proxy jobs. Load balancing is specifically about choosing among backend servers.

Why can software like NGINX or Envoy be described as both a reverse proxy and a load balancer?

1. Because one proxy layer can both process HTTP features and choose among multiple upstream backends
2. Because every client device includes a built-in load balancer
3. Because TLS termination removes the need for backend pools
4. Because reverse proxies work only for static sites

Once a proxy accepts traffic on behalf of backends, it is already positioned to route and distribute that traffic. The same product can therefore play both roles.

When is splitting reverse-proxy and load-balancing duties into distinct tiers most justified?

1. When there is only one backend server
2. When the system has no TLS and no routing rules
3. When HTTP edge concerns and backend scaling have different operational or performance needs
4. When you want to avoid all extra network hops

Separate tiers make sense when edge processing, security policy, or caching evolve differently from backend fan-out. That lets each tier scale and be owned independently.

What is one major benefit of terminating TLS at the reverse proxy?

1. It makes databases strongly consistent
2. It removes the need for health checks
3. It allows centralized certificate management and enables HTTP-aware routing decisions
4. It guarantees lower latency than plain HTTP

Centralized TLS simplifies certificate handling and exposes decrypted HTTP information for routing, caching, or compression decisions. It does not solve unrelated consistency or health-check concerns.

What is the biggest downside of using one combined proxy-and-load-balancer tier?

1. It prevents horizontal scaling entirely
2. It concentrates more responsibilities and resource usage in one place
3. It makes backend health checks impossible
4. It only works with UDP traffic

A combined tier is simpler, but it also means the same layer handles TLS, routing, caching, and backend distribution. That concentrates both CPU work and failure impact.

Building Blocks

• An L4 path that sees transport metadata like IP, port, and protocol. It forwards connections quickly but does not understand HTTP meaning.
• An L7 path that can inspect host, path, header, method, or cookie fields. That application awareness enables richer routing decisions.
• Health checks matched to the layer, from simple port reachability at L4 to HTTP readiness endpoints at L7.
• Backend pools whose needs decide the layer: generic TCP/UDP services usually fit L4, while multi-route web apps usually need L7.

Definitions

Layer 4 Load Balancer

A load balancer that makes decisions using transport-layer data such as IP addresses, ports, and protocol.

• It is a natural fit for raw TCP or UDP forwarding.
• It cannot route by URL path, header, or cookie.

Layer 7 Load Balancer

A load balancer that understands application-layer traffic, most commonly HTTP and HTTPS.

• It can route by host, path, method, header, or cookie.
• That flexibility costs more CPU and configuration complexity.

Connection-Based Routing

Choosing a backend using connection metadata rather than request contents.

• This is the normal model for L4 devices.
• It is efficient but blind to per-request HTTP intent.

Request-Based Routing

Choosing a backend using fields from the application request itself.

• Examples include sending /api and /static traffic to different pools.
• This requires L7 visibility.

Patterns

• Use L4 for high-throughput TCP or UDP traffic when request inspection adds no value.
• Use L7 when one endpoint must route different hostnames or URL paths to different services.
• Layer them when you want an outer high-scale connection distributor and an inner HTTP-aware routing tier.

Strategies

• Choose the layer from required features first: host routing, path routing, header rules, and cookie affinity immediately point to L7.
• Choose L4 for non-HTTP workloads or pass-through cases where the extra parsing cost of L7 buys nothing.
• Treat L7 overhead as a trade for useful application awareness, not as wasted work by default.

Tradeoffs

• L4 has lower overhead and broader protocol support, but it cannot make HTTP-aware decisions.
• L7 enables rich routing and policy enforcement, but it does more per request and is more complex to operate.
• Choosing L7 can simplify multi-service web architectures, while choosing L4 keeps generic pass-through paths fast and simple.

Real World

• AWS Network Load Balancer is the standard L4 example for fast transport-layer distribution of TCP or UDP traffic.
• AWS Application Load Balancer is the standard L7 example for web systems that need host-based or path-based routing across multiple services.

Quiz

Which requirement most clearly forces a load balancer choice toward L7?

1. Handling generic UDP traffic
2. Routing /api and /images to different backend services
3. Forwarding raw TCP streams quickly
4. Passing through traffic without inspecting it

Path-based routing requires the balancer to read the HTTP request. That is an application-layer capability, so L7 is required.

What information does an L4 load balancer primarily use to make forwarding decisions?

1. IP addresses, ports, and protocol details
2. HTTP cookies and URL paths
3. JSON request bodies
4. Rendered HTML responses

L4 operates at the transport layer, so it sees connection metadata rather than application content. That is why it is fast but content-blind.

In AWS terms, which pairing is correct?

1. ALB is L4 and NLB is L7
2. Both ALB and NLB are L4 only
3. NLB is for DNS only and ALB is for databases
4. NLB is L4 and ALB is L7

AWS Network Load Balancer is the classic L4 example, while Application Load Balancer is the HTTP-aware L7 option.

Why is L7 generally heavier than L4?

1. Because it always stores more data in the database
2. Because it parses application-layer details before routing
3. Because it can only run on physical appliances
4. Because it requires clients to know all backend IPs

L7 spends extra CPU and sometimes latency budget understanding HTTP requests. That added context is what enables richer routing features.

When is L4 usually the better fit?

1. When you need cookie-based affinity and path routing
2. When each request must be routed by HTTP headers
3. When the workload is high-throughput TCP or UDP and request inspection adds no value
4. When many services share one hostname and one ALB path table

If the traffic is non-HTTP or does not need application-aware routing, L4 avoids unnecessary parsing overhead. It is optimized for fast connection handling.