Advanced·24 min·0/47 done

Calling LLM APIs in Python

HTTP JSON payloads, structured output, and simple token cost habits for GenAI calls.

Why it matters: Every GenAI feature is HTTP plus JSON. Knowing the payload helps you debug, switch providers, and control cost.

Basics

The chat request shape

Most chat APIs take:

  • model - which model to run
  • messages - list of {role, content} with roles system, user, assistant
  • temperature - 0 for stable extraction, higher for creative text
  • max_tokens - response length cap

The response includes text plus a usage block. Log usage from day one.

Common mistake: high temperature when you need parseable JSON.

Tiny example

Build a tiny chat payload and print JSON.

import json
payload = {
    "model": "mini",
    "messages": [{"role": "user", "content": "What is RAG?"}],
}
print(json.dumps(payload, indent=2))

Basics

Parse JSON output safely

Ask the model for JSON. Do not scrape free prose with regex.

Validate required keys. Wrap json.loads in try/except. Models sometimes wrap JSON in markdown fences - strip those first.

For production, provider JSON mode plus a schema check is stronger than prompt-only JSON.

Tiny example

Load model JSON and require a key.

import json
raw = '{"label": "pass", "score": 0.91}'
data = json.loads(raw)
print(data["label"], data.get("score"))

Basics

Tokens and cost

English is roughly four characters per token. len(text) // 4 is a rough budget estimate.

Long retrieved context usually costs more than the short answer.

Production habits: set timeouts, retry 429/5xx only, cache identical prompts, and truncate context to a token budget.

Tiny example

Estimate tokens from characters.

text = "RAG uses docs as context."
tokens = max(1, len(text) // 4)
print("tokens", tokens)

Then AI / ML

AI / ML examples

Same idea, used in real AI work. Press Try in lab to run it.

AI / ML example

AI / ML: build a grounded chat payload

System + context + question is the usual RAG message shape.

In AI / ML: OpenAI-compatible chat endpoints expect this messages list.

example.py

Python

import json

def build_request(question, context):
    return {
        "model": "gpt-4.1-mini",
        "temperature": 0,
        "messages": [
            {"role": "system", "content": "Answer only from the context."},
            {"role": "user", "content": f"Context:\n{context}\n\nQ: {question}"},
        ],
    }

payload = build_request("What is MLOps?", "MLOps runs ML in production.")
print(json.dumps(payload, indent=2))
print("est tokens:", sum(len(m["content"]) for m in payload["messages"]) // 4)

AI / ML example 2

AI / ML: strip fences and validate keys

Models often wrap JSON in markdown fences.

In AI / ML: Labeling and extraction jobs need valid JSON every time.

example.py

Python

import json

def parse_model_json(raw, required):
    text = raw.strip()
    if text.startswith("```"):
        lines = [ln for ln in text.splitlines() if not ln.startswith("```")]
        text = "\n".join(lines)
    data = json.loads(text)
    missing = [k for k in required if k not in data]
    if missing:
        raise ValueError(f"missing: {missing}")
    return data

raw = '```json\n{"sentiment": "positive", "confidence": 0.92}\n```'
print(parse_model_json(raw, ["sentiment", "confidence"]))

Takeaways

  • 1Chat APIs share model, messages, temperature, and max_tokens.
  • 2Use temperature=0 and validate JSON when you parse output.
  • 3Budget tokens, set timeouts, and retry only transient errors.
Certificate