Scaling Systems

Building Blocks

• Interchangeable app instances behind one stable endpoint. Any instance should be able to serve the same request shape, so scaling adds capacity instead of special cases.
• A load balancer with health checks. It spreads traffic across the fleet and removes bad instances before they keep hurting users.
• Externalized state such as Redis sessions and object storage. Once state stops living on one box, users can move between instances safely.
• Autoscaling tied to demand signals like request rate, queue depth, or latency. It matters only if new instances can start fast enough to help during a spike.

Definitions

Horizontal Scaling

Adding more servers and sharing traffic across them so total capacity grows with instance count.

• It improves both throughput and fault tolerance at the app tier.
• It works best when servers are interchangeable.

Stateless Service

A service whose next request can be handled by any healthy instance because important state is stored elsewhere.

• Local memory or disk should not be required for user continuity.
• Statelessness is the main enabler for clean scale-out.

Load Balancer

The entry point that forwards requests to one backend from a pool of healthy backends.

• It hides server churn from clients behind one stable address.
• It is what makes many instances feel like one service.

Autoscaling

Automatically changing the number of instances based on measured demand or pressure.

• It absorbs normal traffic variation without manual intervention.
• It does not fix slow code or a saturated database.

Patterns

• Scale the stateless web or API tier first; it is usually the easiest place to add more capacity safely.
• Move sessions, uploads, and long-running work out of the instance before adding more servers.
• Keep one stable front door so instances can be added or removed without changing clients.

Strategies

• Externalize the blockers to scale-out first: sessions to Redis, files to object storage, and background work to queues.
• Scale on demand signals that map to user pain, such as sustained latency or queue depth, not CPU alone.
• Check downstream limits before celebrating a larger fleet, because the bottleneck often shifts to the database or another dependency.

Tradeoffs

• Compared with vertical scaling, scale-out gives elasticity and redundancy, but it adds more moving parts and network hops.
• Externalized state enables interchangeable servers, but it introduces new dependencies whose latency and availability now matter.
• Autoscaling saves cost during quiet periods, but poor thresholds can scale too late or flap during noisy traffic.

Real World

• Netflix keeps web and API tiers horizontally scalable so instance failures and traffic bursts are absorbed by fleets instead of one oversized server.
• Shopify adds application capacity during flash-sale events while keeping shared state outside the app node so any healthy instance can serve the next request.

Quiz

What is the main architectural precondition for scaling an application tier horizontally?

1. Using a larger VM type
2. Keeping the service stateless or moving state outside the instance
3. Running on bare metal only
4. Using SQL instead of NoSQL

If requests depend on local instance memory or disk, adding more servers creates inconsistent behavior. Horizontal scaling works cleanly when any instance can handle any request.

Why is a load balancer typically placed in front of horizontally scaled app servers?

1. To distribute traffic across healthy instances behind one stable endpoint
2. To make every request strongly consistent
3. To replace the database primary
4. To eliminate all network latency

The load balancer hides backend churn from clients and spreads requests across the server pool. Without it, clients would need to track instance addresses and failures themselves.

Which statement best describes vertical scaling?

1. Adding more instances behind a dispatcher
2. Splitting the database into shards
3. Moving sessions into Redis
4. Making one machine bigger by adding more CPU, RAM, or bandwidth

Vertical scaling upgrades one node instead of increasing node count. It is simple, but it has cost and hardware ceilings.

An app stores login sessions only in process memory. What is the biggest risk after adding more app instances?

1. TLS will stop working
2. The load balancer cannot perform health checks
3. Users may appear logged out when their next request lands on a different server
4. CPU usage will always decrease

Local-only session state ties a user to one machine. Once requests spread across multiple servers, follow-up requests may miss the in-memory session entirely.

What is the best way to think about autoscaling?

1. A replacement for performance tuning
2. A way to reduce the need for monitoring
3. A mechanism to adjust instance count to changing load
4. A guarantee that the database will never bottleneck

Autoscaling adds or removes capacity as demand changes. It helps with elastic load, but it does not fix inefficient code or slow downstream systems.