Intermediate System design concept · Handling Scale & Bottlenecks · 16 mins read

Database Bottlenecks

Databases usually hit scaling pain after the app tier, especially on reads first and writes later.

Open the interactive lesson See it in Design Live Sports Score System

Read Replicas

Scale read-heavy workloads with replicas while managing lag and stale-read behavior.

Intuition

A primary database often hits read pressure before write pressure. Browsing, feeds, and dashboards can flood one node with SELECTs, leaving less headroom for commits and lock-sensitive writes. Read replicas are a common first scaling step because they add read capacity without changing the schema. The catch is lag: some reads can tolerate it, but some user flows break if they do not see the latest write.

Mental Model

One node accepts writes and streams committed changes to follower nodes. Applications send stale-tolerant reads to followers and freshness-critical reads to the primary. Replicas buy throughput by relaxing how quickly every read sees the newest state. Think of it like: It is like one master document and several copied printouts around the office. Reading a printout is convenient, but it may not include the edit made a few seconds ago.

Building Blocks

Primary Write Node: All authoritative writes commit here first, so ordering and constraints stay centralized.
Replication Stream: Replicas replay the primary’s WAL/binlog after commit. The delay between commit and replay is replication lag.
Read Routing: The app, proxy, or driver decides which queries may hit replicas and which must stay on the primary.
Lag Monitoring: Teams watch replay delay and replica health so stale followers can be removed from the read pool quickly.

Definitions

Read Replica

A follower database that replays primary changes and serves read-only traffic.

Replicas raise aggregate read throughput.
They usually do not accept ordinary writes.
They are common in PostgreSQL, MySQL, and Aurora.

Replication Lag

The time between a primary commit and that change appearing on a replica.

Lag can be tiny under normal load and large during incidents.
Slow I/O, network trouble, or big transactions widen it.
Lag is why replicas can return stale data.

Read-Your-Writes

The expectation that a client immediately sees its own successful write on the next read.

A lagging replica can violate this expectation.
Primary pinning after writes is a common fix.
Many user-facing flows feel broken without it.

Follower Promotion

Turning a replica into the new primary during failover.

Useful for availability, not just scaling.
The promoted node may still be behind at cutover time.
Failover planning is separate from read offload.

Strategies

Send Stale-Tolerant Reads to Replicas When: When many endpoints are read-heavy but do not require the newest state. How: Route catalog pages, timelines, or dashboards to replicas and keep the write path clear on the primary. Treat freshness tolerance as an explicit product choice. Example: A store serves product browsing from replicas, while checkout inventory checks stay on the primary.
Pin Post-Write Reads to the Primary When: When users expect immediate confirmation after an update. How: Keep that request flow on the primary for a short window or until a version check shows the replica has caught up. This preserves read-your-writes without disabling replicas entirely. Example: After a profile edit, the next few profile reads stay on the primary so the new value appears instantly.
Fail Closed on High Lag When: When stale data is more harmful than extra primary load. How: Drop lagging replicas from the read pool once delay crosses a threshold. A slower primary is often safer than confidently serving old account data. Example: A finance app reroutes balance reads to primary when replica lag exceeds 200 ms.

Replication lag and the read-your-writes problem

Most replica setups are asynchronous, so a write can return success before followers apply it. If the very next GET lands on a replica, the user may see the old value and think the write failed.

The usual fix is selective consistency, not “make everything synchronous.” Keep freshness-critical reads on the primary, pin users briefly after writes, or require a sufficiently caught-up replica. Strong answers connect replica lag to concrete product bugs like a missing comment or an unchanged profile.

Tradeoffs

Decision	Upside	Downside
More read capacity vs stale-read risk	Replicas absorb heavy SELECT traffic and protect primary write latency.	Not every read is guaranteed to see the latest commit.
Asynchronous replication vs tighter freshness	Async followers keep write latency lower and scale more easily.	Lag spikes can create confusing user-visible inconsistencies.
Consistency-aware routing vs simpler architecture	Smart routing captures replica benefits without breaking key flows.	The read path is no longer one uniform rule.

Real World

System	How it's used
Amazon Aurora reader endpoint	Aurora exposes reader endpoints that spread read-only traffic across replicas, while the writer instance remains the commit source.
GitHub MySQL replicas	GitHub has described using MySQL replicas for read scale while carefully handling lag-sensitive user flows.

Interview

Questions interviewers ask

When do read replicas help, and when do they not?
Why can a user fail to see their own update right after a successful write?
How would you preserve read-your-writes while still using replicas?
What would you monitor before trusting replicas in production?

What a strong answer covers

Candidates should explain primary-versus-replica roles, asynchronous lag, stale reads, and a concrete mitigation such as primary pinning. They should also say replicas scale reads, not write-heavy bottlenecks.

Common traps

Claiming replicas “solve database scaling” without limiting that claim to reads.
Forgetting that a committed write may still be invisible on a replica moments later.
Treating failover and read offload as the same design problem.

Quiz

What is the main scaling benefit of read replicas?

They offload read traffic from the primary
They make every write faster by default
They remove the need for indexes
They eliminate network latency

Replicas mainly add read capacity.

Why can read-your-writes fail with replicas?

Replicas reject all SELECT queries
The replica may not have applied the latest committed write yet
Primaries cannot return commit acknowledgments
Connection pools disable consistency

Asynchronous replication creates a lag window.

Which workload is the best fit for replica reads?

A balance check right after transfer
A write-only audit ingestor
Product browsing that tolerates slight staleness
Schema migrations

Replica reads work best for stale-tolerant traffic.

What is a common mitigation for stale reads right after a user update?

Delete the replicas
Disable retries
Increase client CPU
Temporarily route follow-up reads to the primary

Primary pinning preserves read-your-writes.

If replica lag jumps from milliseconds to several seconds, what is the safest interpretation?

Replica reads may now be dangerously stale
The system has become strongly consistent
Write throughput must be zero
Sharding is no longer needed

More lag means followers are further behind.

Database Sharding

Split one logical dataset across multiple database nodes and manage the routing cost that follows.

This section is part of the full PRISM roadmap, with worked examples, trade-off tables, interview questions and a quiz.

Unlock the full lesson

Connection Pooling

Reuse a bounded set of database sessions so the app tier does not overwhelm the database.