Deploying & Serving LLMs in Production
Getting a model to answer is easy; serving it reliably and affordably is the job. Learn inference, optimization, serving frameworks, cost and latency, and MLOps — remembered with spaced repetition.
- practice cards
- 75
- practice cards
- per day
- ~10 min
- per day
- level
- Intermediate
- level
- modules
- 5
- modules
What does serving a model involve?
Calling a hosted API is one thing; running a model in production is another. Serving means standing up an inference endpoint that takes requests, runs prefill and decode, streams tokens back, and stays up under real traffic — whether you rent it from a provider or self-host on your own GPUs.
The hard parts are performance and economics. Optimization techniques — the KV cache, batching, paged attention, quantization and speculative decoding — decide your throughput and latency. Serving frameworks handle OpenAI-compatible APIs, queuing, replicas and load balancing for you. And cost and latency — TTFT, tokens per second, GPU sizing and token billing — decide whether the feature is sustainable at scale.
This track grounds production serving in bite-sized, practical questions and uses spaced repetition, so the levers — and the MLOps discipline of versioning, rollouts, drift and monitoring — stay sharp when you are the one on call.
5 modules, seed to bloom
Each module is a set of practice cards — 75 in total. Answer, review, and watch your knowledge grow from seed to full bloom.
Inference Basics
Inference basics — what serving a model means, hosted vs self-hosted, prefill and decode, streaming, and statelessness.
15 cardsOptimization
Optimization — KV cache, quantization, batching, paged attention, speculative decoding, and latency versus throughput.
15 cardsServing Frameworks
Serving frameworks — what they do, OpenAI-compatible APIs, tokenizers, parallelism, queuing, replicas, and load balancing.
15 cardsCost & Latency
Cost and latency — TTFT, tokens per second, GPU versus CPU, memory sizing, token billing, utilization, and autoscaling.
15 cardsMLOps & Monitoring
MLOps and monitoring — versioning, rollouts, drift, evaluation in production, guardrails, observability, and logging.
15 cardsSample questions
A taste of the real cards. Pick an answer, then reveal the explanation.
What does it mean to "serve" a model?
- AMake a trained model available to answer live requests in production
- BTrain a model on fresh data until its accuracy stops improving more
- CCompress a model so its stored weights take far less space on disk
- DEvaluate a model on a held-out dataset to measure its final quality
What is the KV cache during LLM generation?
- AStored key and value tensors of past tokens reused across decode steps
- BA copy of the model's weights kept in CPU RAM as a backup for the GPU
- CA pool of finished responses returned directly when prompts repeat
- DA log of every request and reply kept on disk for later auditing
What does time to first token (TTFT) measure?
- AThe latency from sending a request until the first output token appears
- BThe total time taken to generate every token in a full response
- CThe time the model spends loading its weights into GPU memory
- DThe average number of tokens the model produces every second
What is model versioning in production?
- ATracking which model and weights serve traffic so you can roll back
- BRenaming the model file each time the server happens to restart
- CCounting how many tokens each version of the model has generated
- DStoring user prompts so the model can be retrained on them later
Learn it once, keep it for good
Answer a question
Each card is one practical concept with multiple options. Pick what you think is right.
Get the full answer
See the correct option plus a clear explanation, and a link to deeper docs when one is available.
Review at the right time
A spaced-repetition engine (SM-2 or FSRS) resurfaces each card just before you would forget it.
Why production serving is worth your time
Control your inference bill
TTFT, tokens per second, GPU memory and utilization decide cost — understanding them keeps an LLM feature affordable at scale.
Make it fast
The KV cache, batching and paged attention are where latency and throughput are won or lost. Know the levers, not just the buzzwords.
Use frameworks well
Serving frameworks hide real complexity — queuing, replicas, load balancing. Knowing what they do lets you tune instead of guess.
Operate it day two
Versioning, rollouts, drift and monitoring are what keep a deployed model trustworthy after launch, not just at it.
Common questions
Do I need to be an ML engineer? +
No, but some backend or infrastructure background helps — this track is aimed at people running models in production. The concepts (latency, batching, cost) are explained plainly.
Does it cover self-hosting and hosted APIs? +
Yes. The Inference Basics module covers hosted vs self-hosted serving, and the rest applies to both — optimization, cost and MLOps matter wherever the model runs.
Is it free? +
Yes, completely free. No registration or credit card is required, and all your progress is stored locally in your browser.
How long does it take? +
About 10 minutes a day. Spaced repetition means short, frequent sessions beat long cramming, so the levers and trade-offs stick.
Ready to serve models in production?
Plant your first seed today. Ten minutes a day is all it takes to run models that stay fast, affordable and reliable under real load.