Intermediate·24 min·0/47 done

pandas for datasets

Build, filter, group, and clean tables with pandas before you train.

Why it matters: Eval results, feature tables, and cost logs are DataFrames. Bad rows beat a good model.

Basics

DataFrame and Series

A DataFrame is a table. A Series is one column.

First checks on new data:

df.head()
df.shape
df.dtypes
df.describe()

Common mistake: a numeric column loaded as object because of stray text.

Tiny example

Build a tiny table and print shape and mean.

import pandas as pd
df = pd.DataFrame({"score": [0.9, 0.7], "model": ["mini", "large"]})
print(df.shape)
print(df["score"].mean())

Basics

Filter and add columns

Filter with a boolean mask: df[df["score"] >= 0.7].

Combine conditions with & and |, and wrap each part in parentheses.

Add a column with assignment: df["passed"] = df["score"] >= 0.7.

For assignment to filtered rows, prefer .loc.

Tiny example

Keep rows with score >= 0.7.

import pandas as pd
df = pd.DataFrame({"score": [0.9, 0.4, 0.8]})
print(df[df["score"] >= 0.7])

Basics

groupby and missing data

groupby splits rows, applies a function, and combines results.

Count nulls with df.isna().sum(). Then choose dropna() or fillna(...) on purpose.

Also check df.duplicated().sum() before training.

Tiny example

Group by model and print mean score.

import pandas as pd
df = pd.DataFrame({"model": ["a", "a", "b"], "score": [0.9, 0.7, 0.8]})
print(df.groupby("model")["score"].mean())

Then AI / ML

AI / ML examples

Same idea, used in real AI work. Press Try in lab to run it.

AI / ML example

AI / ML: filter eval rows

Keep fast, high-score calls and sort them.

In AI / ML: LLM eval logs are DataFrames of model, latency, and score.

example.py

Python

import pandas as pd

df = pd.DataFrame({
    "model": ["a", "a", "b", "b"],
    "latency_ms": [400, 1200, 300, 900],
    "score": [0.9, 0.6, 0.85, 0.45],
})
df["passed"] = df["score"] >= 0.7
kept = df[(df["latency_ms"] < 1000) & (df["score"] >= 0.7)]
print(kept.sort_values("score", ascending=False))

AI / ML example 2

AI / ML: per-model quality report

groupby().agg() builds a small metrics table.

In AI / ML: Cost and quality dashboards start as groupby aggregates.

example.py

Python

import pandas as pd

df = pd.DataFrame({
    "model": ["mini", "mini", "large", "large"],
    "latency_ms": [820, 640, 1500, 1320],
    "score": [0.91, 0.72, 0.65, 0.80],
})
report = df.groupby("model").agg(
    n=("model", "size"),
    avg_latency=("latency_ms", "mean"),
    avg_score=("score", "mean"),
).round(2)
print(report)

Takeaways

1Run head, shape, dtypes, and describe before anything else.
2Filter with masks and parentheses; assign with .loc when needed.
3groupby().agg() builds the per-segment metrics reports need.

Certificate