Advanced·22 min·0/47 done

Serving models with FastAPI

Turn a model or RAG pipeline into a validated HTTP service with health checks.

Why it matters: A notebook model has no product value until it is an API. FastAPI is the usual Python choice for inference endpoints.

Basics

Why FastAPI

You describe the request with types. FastAPI validates and returns 422 on bad input. It also builds docs at /docs.

Load the model once at startup, not per request. Loading a large model inside the handler wastes seconds and memory every call.

The browser lab simulates each FastAPI step, so you can learn the full request flow on this website.

Tiny example

Validate a request dict like an API model would.

def validate(body):
    if not body.get("question"):
        raise ValueError("question required")
    return {"ok": True, "q": body["question"]}
print(validate({"question": "What is RAG?"}))

Basics

Health and predict shapes

Useful endpoints:

  • GET /health - process is up
  • GET /ready - model is loaded
  • POST /predict or /chat - real work

Return real status codes: 400 bad input, 422 validation, 503 not ready. Do not hide errors inside a 200 body.

Tiny example

Sketch health and predict responses.

health = {"status": "ok"}
predict = {"label": "pass", "score": 0.91}
print(health)
print(predict)

Basics

Deploy checklist

Run with uvicorn. For GPU inference, usually one worker per GPU because each worker loads its own model copy.

Add timeouts, body size limits, and rate limits. Log request id, model version, latency, and token counts.

Tiny example

Print a short run checklist.

checks = ["health", "ready", "timeouts", "auth"]
for c in checks:
    print("[ ]", c)

Then AI / ML

AI / ML examples

Same idea, used in real AI work. Press Try in lab to run it.

AI / ML example

AI / ML: request validation for predict

Same rules pydantic enforces, written by hand.

In AI / ML: Inference APIs reject empty text and bad thresholds early.

example.py

Python

def validate_request(payload):
    errors = []
    text = payload.get("text")
    if not isinstance(text, str) or not text.strip():
        errors.append("text required")
    elif len(text) > 5000:
        errors.append("text too long")
    thr = payload.get("threshold", 0.5)
    try:
        thr = float(thr)
        if not 0.0 <= thr <= 1.0:
            errors.append("threshold out of range")
    except (TypeError, ValueError):
        errors.append("threshold must be a number")
    if errors:
        return {"status": 422, "errors": errors}
    return {"status": 200, "text": text.strip(), "threshold": thr}

for p in [{"text": "good"}, {"text": "  "}, {"text": "ok", "threshold": 5}]:
    print(p, "->", validate_request(p))

AI / ML example 2

AI / ML: FastAPI request flow

Learn the endpoint flow without installing anything.

In AI / ML: Production inference services follow this health + predict pattern.

example.py

Python

flow = [
    "load model once",
    "check health",
    "validate request",
    "predict",
    "log latency",
]
for step in flow:
    print(step)

Takeaways

  • 1Load the model once at startup, never per request.
  • 2Ship /health and /ready separately.
  • 3Validate input and return real HTTP status codes.
Certificate