System Design Interview
Real-time payment fraud detection inspired by Stripe Radar and similar systems: online velocity features, rules plus a gradient-boosted model scored in milliseconds, streaming feature updates, a label feedback loop for retraining, and analyst review for borderline cases. A teaching design.
See why counting history with SQL on every transaction blows the latency budget.
What was missing: Nothing yet.
Why that's risky: Each decision runs several aggregate queries over growing tables, so it blows the 100 ms budget at peak, and fixed rules miss new fraud patterns.
What gets added: One server applying rules with SQL counts over the transactions table for each new transaction.
Trade-offs: Rules are easy to explain, but slow to compute this way and easy for fraudsters to learn.
Answer 'how much has this card done lately?' in a millisecond.
What was missing: Velocity was recomputed from history for every transaction.
Why that's risky: Rules only catch patterns someone already wrote down.
What gets added: An online feature store of sliding-window counts and amounts per card, device, IP and email; a scoring service that evaluates cached rules on those features and records each decision with its reasons.
Trade-offs: Features are approximate sliding windows (bucketed by minute) in exchange for constant-time reads.
Catch patterns no rule describes.
What was missing: Only hand-written rules.
Why that's risky: Features are updated inline and the model never learns from its mistakes.
What gets added: A model service serving a gradient-boosted model that turns the transaction and its features into a fraud probability; the scorer combines it with rules and per-merchant thresholds. Models are loaded from a registry.
Trade-offs: Models are harder to explain than rules; decisions record the top contributing features.
Keep features fresh off the hot path and retrain on real outcomes.
What was missing: Inline feature updates slowed scoring, and the model never improved.
Why that's risky: Borderline cases are forced into approve or decline, and model drift isn't watched.
What gets added: A transaction event stream feeding a streaming feature updater, a label collector that pairs transactions with their eventual outcome (chargebacks can arrive weeks later), a trainer that retrains and publishes only if the new model beats the current one, and trend analytics.
Trade-offs: Labels arrive late, so training always lags fraud by a little.
Send uncertain cases to people, and watch decision quality.
What was missing: No review path, no caller limits, no monitoring of decision quality.
Why that's risky: Remaining risk is adversarial: fraudsters adapt, so features and models need continual work.
What gets added: A review queue where borderline transactions go to analysts with the evidence they need (their verdicts become labels), caller rate limits, and monitoring of latency, decline and review rates, false positives and model drift.
Trade-offs: Review adds delay for a small share of payments in exchange for fewer wrongly declined customers.
Combine rules with ML scores and calibrate thresholds using offline evaluation data.
Use batch + streaming updates and accept eventual consistency for real-time scoring.
Rules provide immediate coverage and clear explanations while ML models mature.
Keep feature reads local and precompute heavy features to avoid synchronous lookups.
Streams power real-time monitoring and offline model retraining without blocking scoring.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
