Practice: Design an LLM API Gateway

One API in front of many model providers, inspired by OpenRouter and Portkey: API keys checked from a cache, token usage metered in the background, requests routed to a healthy provider with automatic fallback, and per-key rate limits and budgets. A teaching design.

Build this system yourself on the canvas below. Add components, connect them, and run traffic simulations to see where your design holds up and where it breaks.

Functional requirements

  • Apps send chat completion requests to one endpoint with one API key.
  • Developers create keys and set a monthly budget per key.
  • Each request is routed to a model provider; if one is down, another serves it.
  • Every request's token usage is recorded for billing.
  • Keys over their rate limit or budget are refused.

Non-functional requirements

  • The gateway adds under 20 ms before the provider is called.
  • Scale: 3K requests/s at peak across 50K API keys.
  • A single provider outage doesn't stop traffic.
  • Usage records are never lost.
  • Availability: 99.95%.
PRISM
System Design Interview
Round 1 of 4 · Architecture Design · 60:00 remaining
PRISM logo PRISM logo
Problems
AI Interview
Interview Challenges
Build & Stress Test NEW
System Architectures
Design & Fix NEW
Interactive Roadmap
System Design Guides Interview Prep Guide
Problems
Today's mission
Notifications
    Theme
    Pricing Community NEW Send feedback Report a problem NEW
    Sign in
    Theme
    Community NEW Send feedback Report a problem NEW
    All problems
    Phase
    Build & Stress Test
    Phase 01: Thinking in Systems
    Upcoming

    Components

    User
    CDN
    Load Balancer
    Server
    Cache
    Database
    Blob Storage
    Search Index
    Queue
    Worker
    Rate Limiter
    Service
    API Gateway
    Reverse Proxy
    WebSocket Server
    Third-party API

    Inspector

    Notes
    Use clear names so your design intent is easy to understand.
    Good
    Name by business meaning
    "Order API", "Restaurant Service"
    Avoid
    Generic names = zero signal
    "Server 1", "API", "Queue"
    A short description for each component makes feedback much better.
    100%
    Start by identifying:
    • Users & entry points
    • APIs & services
    • Databases & storage
    • Traffic flow & scale

    Round 1 of 4 Architecture Design

    Run a simulation to see results.

    Time Remaining
    60:00
    System Design Interview

    What are the core functional requirements?
    What are the key non-functional constraints?

    Questions

    Start Evaluation to unlock questions.

    EVALUATE MODE

    Test yourself like it's the real thing.

    A structured 4-module evaluation that mirrors how top companies assess system design candidates.

    Architecture Design
    Draw your system on the canvas. Define components, connections, and data flow.
    FR & NFR Requirements
    Answer functional and non-functional requirement questions about your design.
    MCQ Round
    Multiple choice questions testing your depth on the chosen system.
    Tradeoff Analysis
    Justify your design decisions and defend your architectural tradeoffs.
    AI Report Generated
    Choose a Problem
    No problem selected
    • 30 min
    • 45 min
    • 60 min
    Round 2 of 4
    MCQ Round
    Answer multiple-choice questions based on your design.

    Exit Interview?

    You're in the middle of an interview session. Leaving now will end your current attempt.

    Your progress will be saved.

    Open a saved design

    Select a design to load into the canvas.

    My Evaluations

    Your past evaluation sessions

    Here’s a simple request flow that follows the expected layer order.

    External User
    →
    Edge CDN → API Gateway → Load Balancer
    →
    Compute App Servers / Services
    →
    DataAccess Cache
    →
    Storage Database / Search Index
    →
    Async Queue → Worker

    Tip: keep arrows moving forward through layers (Edge → Compute → Storage). Avoid sending storage back to compute.

    Evaluation Instructions

    Read the rules carefully before starting. The test auto-submits on refresh.

    Before you start

    • Build your architecture on the canvas. The timer starts when you click Start Evaluation.
    • Don't forget to answer Functional Requirement and Non Functional Requirements.
    • When satisfied with your design, click Next to lock it and view the questions.
    • Please answer final step questions to complete the evaluation.

    Dos

    • Do read each question carefully before answering.
    • Do include required components to maximize component coverage.
    • Do save a copy of your design if you want to keep it before submission.

    Don'ts

    • Don't refresh or close the tab during an active evaluation — this will auto-submit your answers.
    • Don't switch app modes or open another tab while the evaluation is running.
    • Don't attempt to edit the design after clicking Next; the workspace will be locked.

    All the best!!

    Confirm

    Input

    Notice

    Evaluation Report
    Report a problem

    Evaluation Complete

    Generating Your Report

    Hang tight — our AI is evaluating your design…

    Did you know?

    Loading…

    Share feedback

    Tell us what worked well and what we can improve.

    Let's personalize this

    Answer a couple of quick questions so we can tailor your journey and missions.

    Your welcome gift: of Full PRISM Access

    Your Pro access starts as soon as you press Start my journey.

    SYSTEM BRIEF

    System Constraints

    What the system must do — every item is a user-facing behaviour your architecture must support.

      ↑ Engineering Constraints

      These are the failure modes you must design against — latency SLAs, durability targets, traffic ceilings.

        ⇆ Architecture Constraints

        Core Concepts to Master

        Your Journey
        PHASE – –
        0 / 0 0%
        0
        Mock Interview Checklist

        Pick a topic to start

        Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.

        Phase Mastered

        You've conquered this phase. These are the skills you now own:

          +500 XP

          Company Interview Paths

          Progress Summary