Inference Engineering: Reliable AI Serving
Build and evaluate the systems that run trained AI models: latency, throughput, model selection, serving, caching, capacity, cost, and release discipline. Includes an API-first lab and an optional local-serving lab.
Your course plan
Study format
8-12 minutes per lesson, with a 60-90 minute capstone. Estimates include practice and assessment. Estimated total: 3-4 hours including the capstone and quiz.
Prerequisites and scope
AI 101 plus basic API, JSON, and spreadsheet literacy. Basic Python is helpful for the optional implementation lab; a provided measurement worksheet supports the noncoding track.
Learning outcomes
- Inference service specification
- Latency and throughput analysis
- Memory and batching tradeoffs
- Cost per accepted outcome
- Serving reliability and release controls
Four modules, three lessons each
- Specify and measure the service
- Choose and optimize a serving path
- Operate under load
- Validate a production recommendation
Completion and assessment
Complete all 12 lessons and the practice activities. Use the final-lesson rubric to review the capstone with an instructor or peer. The portal scores a separate 10-question final quiz; 80% or higher is required for the existing course-completion credential. Capstone work is not automatically graded by the portal. No academic credit or external license is implied.
Authoritative reference shelf
Technical and regulatory references reviewed October 8, 2026. Original teaching scenarios, assumptions, calculations, exercises, and rubrics are instructional examples. Verify current source requirements for an actual project or operation.
M1.1 | What inference engineering owns
Learning objective
Define the scope of an inference service and distinguish it from model training.
Core lesson
Training changes model parameters; inference uses a trained model to produce a result. Inference engineering designs the path around that result: request handling, preprocessing, context, model execution, validation, response delivery, monitoring, and recovery. A model benchmark alone does not describe the experience of a user waiting for an answer. Start with a concrete application, such as classifying service requests or drafting a grounded support response. Define acceptable quality, response time, expected traffic, data restrictions, and failure behavior. Record which parts the provider controls and which parts your team controls. A hosted API can simplify infrastructure while leaving your team responsible for integration, validation, costs, and the user-facing result.
Instructor teaching guidance
Begin by establishing what the learner already knows about m1.1 | what inference engineering owns. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: map a support request from browser to completed response. Mark queueing, retrieval, model execution, validation, and delivery. Write a service specification containing one quality threshold, one latency target, a traffic assumption, and a safe fallback.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
Success defines the complete service and accountable owners rather than treating model execution as the whole application.
Knowledge check
Which delay can your team fix if the model is fast but the customer still waits too long?
M1.2 | Measure latency and useful throughput
Learning objective
Measure user-visible inference behavior with metrics appropriate to the service.
Core lesson
Time to first token, TTFT, describes how long a streaming user waits for the first output. Inter-token latency describes the spacing between generated tokens. Total response latency includes the entire completion. Queueing, input processing, network travel, retrieval, and validation may each contribute. Report median and tail percentiles, because averages can hide a poor experience for a minority of users. Throughput measures work completed per time interval, but rejected, incorrect, or timed-out results should not count as successful work. Keep input length, output length, concurrency, region, model version, and test duration alongside measurements. Compare realistic workload shapes rather than changing several variables and attributing every improvement to the model.
Instructor teaching guidance
Begin by establishing what the learner already knows about m1.2 | measure latency and useful throughput. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Worked example: Run A averages 1 second but its 95th percentile is 8 seconds. Run B averages 1.4 seconds with a 2-second 95th percentile. Practice: choose for a time-sensitive service and justify the tradeoff. Prepare a timing table for ten synthetic requests, then explain why a larger repeated sample is needed.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
A valid comparison reports workload conditions, quality, tail latency, and completed useful work.
Knowledge check
Why can a system with higher tokens per second still produce a worse support experience?
M1.3 | Build a reproducible baseline lab
Learning objective
Construct a reproducible inference evaluation with evidence suitable for comparison.
Core lesson
Choose a small synthetic dataset with ordinary requests, long inputs, malformed inputs, missing evidence, and burst traffic. Record expected behavior before running the model. Keep a frozen evaluation set separate from examples used to tune the system. Capture model identifier, configuration, prompt version, timestamps, error category, token usage if available, and accepted-output status. API-first learners can use an approved API or exported request logs; local-serving learners can use an approved small model and the runtime documentation for their environment. No paid deployment or GPU purchase is required for the worksheet path. Treat every numerical target in this course as a teaching assumption, not a universal service standard.
Instructor teaching guidance
Begin by establishing what the learner already knows about m1.3 | build a reproducible baseline lab. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: create 20 synthetic support cases and an evaluation sheet with case ID, expected behavior, latency, cost, and pass/fail reason. Run or inspect a baseline, retain raw evidence without secrets, and rerun five cases to observe variability. Write down one potential measurement bias.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
The baseline is repeatable, uses non-sensitive cases, preserves configuration, and distinguishes observations from assumptions.
Knowledge check
What invalidates an optimization claim if the before and after runs use different input lengths?
M2.1 | Choose hosted, local, edge, or hybrid
Learning objective
Select a deployment pattern and model using application requirements.
Core lesson
Compare deployment patterns against data sensitivity, network reliability, hardware capacity, model quality, operational staffing, and total cost. Hosted inference offers managed capacity with provider dependencies. Local or private inference offers more control but requires maintenance, security, capacity planning, and hardware support. Edge inference can reduce network dependence for suitable workloads and models. Use the smallest candidate that meets measured quality and operating constraints. A routing design may send a simple request to one model and a difficult request to another, but routing itself must be evaluated. Count escalation overhead and misrouting. Verify model licenses and vendor data-handling terms. Avoid assuming that private placement by itself establishes security or regulatory compliance.
Instructor teaching guidance
Begin by establishing what the learner already knows about m2.1 | choose hosted, local, edge, or hybrid. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: compare a rural offline inspection assistant, a public marketing draft tool, and a confidential policy assistant. Build a weighted scorecard for one of them and name the unacceptable constraint. Propose a fallback when the selected service becomes unavailable.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
Selection rests on measured fitness, operating capacity, lawful use, data handling, and service continuity.
Knowledge check
When does a cheaper model increase total cost because users must repeatedly correct its output?
M2.2 | Prefill, decode, and memory budgets
Learning objective
Explain inference phases and create a preliminary memory budget.
Core lesson
For many autoregressive language models, prefill processes the input context, while decode generates output step by step. Long inputs can increase initial work; long outputs can occupy the service for longer. Key-value, or KV, caching stores intermediate attention information so the runtime can reuse prior computation. Its memory demand changes with model architecture, context length, precision, and active sequences. A memory budget must include model weights, runtime overhead, working memory, KV cache, and headroom. For a deliberately simplified teaching estimate, eight billion parameters at two bytes each imply about 16 GB of raw weights, before all other allocations. This is not a hardware sizing guarantee. Measure memory and throughput with the chosen runtime under representative concurrency.
Instructor teaching guidance
Begin by establishing what the learner already knows about m2.2 | prefill, decode, and memory budgets. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: estimate raw weight bytes at one and two bytes per parameter for the same eight-billion-parameter example. List four additional allocations. Explain why a model fitting on a GPU at one short request may fail under ten long concurrent requests.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
The estimate separates raw weights from full runtime memory and explicitly tests context and concurrency.
Knowledge check
Which memory component can grow when the weights stay unchanged but active contexts become longer?
M2.3 | Batch and quantize through controlled experiments
Learning objective
Evaluate batching and quantization as measured tradeoffs.
Core lesson
Batching combines work so hardware can be used more efficiently. Waiting to assemble a batch can also delay a request. Some language-model runtimes support continuous scheduling of active sequences; verify the runtime-specific behavior rather than copying a setting from another engine. Quantization changes numerical representations to reduce some memory or compute requirements, with effects that depend on hardware, model, and implementation. Change one optimization at a time, preserve the baseline, rerun representative quality tests, and measure tail latency and memory. Speedup without acceptable task quality is a failure. Use batch size, queue delay, and precision as controlled variables. Roll back a configuration that violates the service target, even if aggregate throughput improves.
Instructor teaching guidance
Begin by establishing what the learner already knows about m2.3 | batch and quantize through controlled experiments. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: compare three hypothetical configurations: baseline, larger batch, and lower precision. Fill in quality, median latency, 95th-percentile latency, memory, and useful requests per second. Choose only after identifying the binding service constraint and a regression test.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
An optimization is accepted only when it preserves required quality and improves the chosen service objective under comparable conditions.
Knowledge check
Why might a larger batch help offline processing but fail an interactive assistant?
M3.1 | Cache the right thing safely
Learning objective
Choose and validate caching policies for performance without losing correctness.
Core lesson
Distinguish caching model computation from caching a complete answer. Prefix caching can reuse suitable repeated input computation in supporting runtimes. Response caching can reuse a previous result only when the task, evidence, permissions, freshness, and configuration make that safe. A cache key must reflect the factors that change the answer. Use tenant boundaries, permission checks, expiration, versioned inputs, and invalidation when source documents change. Measure hit rate and latency savings, and include correctness tests for stale answers. Sensitive data must not become visible to another user through a shared cache. A cache miss should produce normal service behavior, not failure. A changed policy can invalidate an answer even if the words in the user request are identical.
Instructor teaching guidance
Begin by establishing what the learner already knows about m3.1 | cache the right thing safely. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: design cache keys for a public FAQ and a private employee-policy assistant. Include model and prompt versions, source version, and tenant or access scope where required. Create a test in which the policy changes while the question remains the same.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
Success demonstrates correct cache boundaries, freshness, invalidation, and cross-user isolation.
Knowledge check
What must invalidate a cached policy answer when an employee changes access roles?
M3.2 | Queue, scale, and protect the service
Learning objective
Design capacity and overload controls from observed workload behavior.
Core lesson
Arrival rate, service time, concurrency, and queue length interact. If incoming work persistently exceeds completion capacity, the queue grows until a limit or timeout is reached. Control request size, admission, concurrency, and deadlines. Separate interactive and batch priorities when one can starve the other. For a simplified stable-system illustration, five arrivals per second and a two-second average time in the system imply roughly ten requests in flight. Real sizing needs variability, output-length distribution, tail targets, and headroom. Scale from measured bottlenecks, not utilization alone. Define overload behavior: reject gracefully, defer eligible batch work, or route to an evaluated fallback. Cap retries with backoff and preserve idempotency for downstream actions to avoid multiplying failures.
Instructor teaching guidance
Begin by establishing what the learner already knows about m3.2 | queue, scale, and protect the service. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: design a burst plan for a service normally receiving five requests per second but temporarily receiving twenty. Set a bounded queue, user-visible timeout, retry limit, and batch priority. State what measurement would tell you to add capacity.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
A resilient plan bounds work, avoids retry storms, protects priority service, and has a measurable scaling trigger.
Knowledge check
Why can unbounded retries make an overloaded inference service less available?
M3.3 | Price accepted outcomes and energy use
Learning objective
Calculate unit economics and resource efficiency around accepted work.
Core lesson
Cost per token is only one part of service economics. Count model charges or hardware, idle capacity, networking, retrieval, storage, monitoring, human review, retries, and corrections. Compare cost per accepted task at the required quality. For an original teaching example, $20 of total processing and review cost across 80 accepted cases is $0.25 per accepted case, even if 100 requests were attempted. For local infrastructure, measure energy over the same workload and interval before deriving energy per accepted task. Include idle power and meter boundaries. A fast configuration may consume more instantaneous power yet finish sooner; another may have lower power but poor acceptance. Report both total resource use and useful work. Do not invent vendor pricing or energy savings.
Instructor teaching guidance
Begin by establishing what the learner already knows about m3.3 | price accepted outcomes and energy use. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: compare two candidates with different token costs, rejection rates, and reviewer time using a common hourly labor assumption. Calculate total cost divided by accepted outcomes. For a local lab, specify the energy measurement boundary instead of estimating from a hardware nameplate alone.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
The comparison includes quality failures, review, retries, and the full chosen resource boundary.
Knowledge check
What happens to cost per accepted task if a cheaper model doubles reviewer effort?
M4.1 | Observe and secure inference operations
Learning objective
Build observable and access-controlled inference operations.
Core lesson
Create telemetry for request rate, queue delay, total latency, token usage where available, failures, quality samples, cache behavior, resource use, and cost. Correlate a request through the application using a non-sensitive identifier. Logs should support diagnosis without becoming an uncontrolled copy of private prompts and outputs. Authenticate clients, authorize data and tools, validate input size and output shape, set rate limits, and keep credentials outside client code. Distinguish provider errors from application and validation errors. Alert on symptoms users experience, not only hardware status. A runbook names who responds, how service is contained or degraded, and how evidence is retained. Monitor model and source versions so a behavior change can be traced.
Instructor teaching guidance
Begin by establishing what the learner already knows about m4.1 | observe and secure inference operations. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: design a dashboard with six metrics and three alerts. Write a first-response runbook for rising timeouts. Include a privacy rule for logs and explain how the operator can diagnose a single request without storing its full confidential content.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
The operator can identify the affected service, likely layer, severity, evidence, and immediate containment action.
Knowledge check
Why is a healthy GPU metric insufficient when accepted-output quality is declining?
M4.2 | Release, compare, and roll back
Learning objective
Design a reproducible inference release and recovery process.
Core lesson
Version the model, prompt, retrieval configuration, serving runtime, and relevant dependencies as one identifiable release. Test the full service with representative and adversarial cases. Include outages, long inputs, stale context, malformed output, and cost spikes. Quality and latency targets must be established before comparing candidates. Use a small controlled rollout when supported, watch declared thresholds, and retain a working prior configuration. A rollback procedure must address sources, caches, and schema compatibility as well as model selection. Document why a release was accepted, the evidence, known limits, responsible owner, and a review date. Tuning against the final test set repeatedly can conceal regressions; retain independent holdout cases.
Instructor teaching guidance
Begin by establishing what the learner already knows about m4.2 | release, compare, and roll back. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Practice: write a release record for a model change with quality, latency, and cost gates. Add two rollback triggers and a test of the previous configuration. Explain how cache invalidation and source changes are included in recovery.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
A release is accepted by predetermined evidence and can be reversed without ambiguous configuration or stale data.
Knowledge check
Why does changing the model and retrieval source together complicate attribution of a quality regression?
M4.3 | Capstone: recommend a reliable inference service
Learning objective
Defend an inference architecture and configuration with comparable evidence.
Core lesson
Deliver a measured recommendation for a support-triage inference service using the synthetic baseline. Compare at least two models or serving configurations. If no API or hardware is available, use a clearly labeled supplied or hypothetical measurement table and do not present it as a live benchmark. Include architecture, traffic assumptions, quality rubric, timings, tail behavior, cost per accepted task, data controls, capacity strategy, fallback, release gates, and rollback. Explain one rejected optimization. The work should help a team choose a service under stated constraints, not claim a universal fastest model. Optional coding learners implement the request and measurement harness with their approved environment and current runtime documentation.
Instructor teaching guidance
Begin by establishing what the learner already knows about m4.3 | capstone: recommend a reliable inference service. Teach the principle in plain language, define unfamiliar terms, then connect it to the course goal. Ask the learner to explain the idea back in their own words before moving to application.
Applied exercise
Spend 60-90 minutes on a four-page recommendation plus an evaluation table. Rubric: service and architecture 20, measurement and quality 30, optimization comparison 20, economics 15, operations and rollback 15. Target 80/100 with no unexplained test conditions. Complete the separately scored portal quiz after all lessons.
Facilitation sequence
Guide the learner through the application in five steps: describe the present situation, identify the lesson principle, choose an action, name the evidence to review, and define how success will be measured. Use follow-up questions when an answer is vague or unsupported.
Mastery standard
The recommendation is reproducible, respects service constraints, and states whether evidence was measured or simulated.
Knowledge check
What additional experiment would most reduce uncertainty before production deployment?