Project: ML Experiment Tracker
Build a mini MLflow: save every training run's settings and scores to a JSON file, then find the best run. Uses classes, files, JSON and errors.
Why it matters: Every ML team tracks experiments. MLflow and Weights & Biases do this at scale, and interviewers like it when you can explain how it works inside.
Step 1
What you will build
An ExperimentTracker class that an ML engineer could really use:
log_run(name, params, metrics)saves one training run toruns.jsonbest_run(metric)finds the run with the highest score- Bad input raises a clear
ValueErrorinstead of saving junk
Because the runs live in a file, they survive after the program ends. Run it twice on your laptop and the history grows, just like a real tracker.
Step 2
A run is a dict
Keep what you chose (params, like the learning rate) apart from what you measured (metrics, like accuracy). That split is exactly how MLflow stores runs.
Tiny example
Settings in params, results in metrics.
run = {
"name": "baseline",
"params": {"lr": 0.01, "epochs": 5},
"metrics": {"accuracy": 0.81},
}
print(run["metrics"]["accuracy"])Step 3
Save after every run
Write the whole list to the file each time you log a run. When the program starts again, __init__ reads the file back, so nothing is lost.
Tiny example
json.dumps turns a list of dicts into text.
import json
from pathlib import Path
runs = [{"name": "baseline", "metrics": {"accuracy": 0.81}}]
Path("runs.json").write_text(json.dumps(runs, indent=2))
print(json.loads(Path("runs.json").read_text())[0]["name"])Step 4
Find the best run
Skip runs that do not have the metric you asked for, and return None if no run has it. Code that other people call should never crash on a fair question.
Tiny example
max() with a key picks the highest score.
runs = [
{"name": "a", "metrics": {"accuracy": 0.81}},
{"name": "b", "metrics": {"accuracy": 0.86}},
]
best = max(runs, key=lambda run: run["metrics"]["accuracy"])
print(best["name"])Build it here · answer checked automatically
Build the project
Finish log_run and best_run. log_run raises ValueError when metrics is empty, appends the run as a dict with name, params and metrics, and saves all runs to the JSON file. best_run(metric) returns the run with the highest value for that metric, skips runs without it, and returns None when no run has it.
Edit the code, then press Run & check. Passing it marks this project done.
python console
Project
~/lesson.py
Ctrl+Enter
console output
ready
$ waiting for Run...On your laptop
Run the real thing on your computer
Run the full tracker on your laptop. It only uses the standard library, so there is nothing to install.
- 1
Make a folder called
experiment-tracker, save the files below into it, and open a terminal there. On Mac and Linux, typepython3instead ofpython. - 2
Run it twice. The second time it prints 6 runs, because the first 3 were saved in
runs.json. Open that file to see your data.
tracker.py
"""ML Experiment Tracker: a mini MLflow that saves training runs to a JSON file."""
import json
from pathlib import Path
class ExperimentTracker:
def __init__(self, path="runs.json"):
self.path = Path(path)
if self.path.exists():
self.runs = json.loads(self.path.read_text(encoding="utf-8"))
else:
self.runs = []
def log_run(self, name, params, metrics):
if not metrics:
raise ValueError("metrics must not be empty")
for key, value in metrics.items():
if not isinstance(value, (int, float)):
raise ValueError(f"metric {key} must be a number, got {value!r}")
self.runs.append({"name": name, "params": params, "metrics": metrics})
self.path.write_text(json.dumps(self.runs, indent=2), encoding="utf-8")
def best_run(self, metric):
candidates = [run for run in self.runs if metric in run["metrics"]]
if not candidates:
return None
return max(candidates, key=lambda run: run["metrics"][metric])
def leaderboard(self, metric):
ranked = sorted(
(run for run in self.runs if metric in run["metrics"]),
key=lambda run: run["metrics"][metric],
reverse=True,
)
for rank, run in enumerate(ranked, start=1):
print(f"{rank}. {run['name']:<12} {metric}={run['metrics'][metric]:.3f} params={run['params']}")
def main():
tracker = ExperimentTracker("runs.json")
tracker.log_run("baseline", {"lr": 0.01, "epochs": 5}, {"accuracy": 0.81})
tracker.log_run("bigger-lr", {"lr": 0.1, "epochs": 5}, {"accuracy": 0.86})
tracker.log_run("more-epochs", {"lr": 0.01, "epochs": 20}, {"accuracy": 0.84})
print(f"{len(tracker.runs)} runs saved in {tracker.path}")
tracker.leaderboard("accuracy")
print("Best run:", tracker.best_run("accuracy")["name"])
if __name__ == "__main__":
main()README.md
# ML Experiment Tracker
A mini MLflow in plain Python. It logs each training run's parameters and metrics
to a JSON file and finds the best run.
## How to run
python tracker.py
## Example output
3 runs saved in runs.json
1. bigger-lr accuracy=0.860 params={'lr': 0.1, 'epochs': 5}
Best run: bigger-lr
## What I learned
- Designing a small class with clear methods
- Saving and loading data with JSON so it survives restarts
- Validating input and raising clear errors
## Next ideas
- Add a delete_run(name) method
- Plot accuracy per run with matplotlib
- Compare this design with how MLflow stores runs.gitignore
__pycache__/ .venv/ runs.json
Run it from the same folder:
python tracker.py
You should see something like:
3 runs saved in runs.json
1. bigger-lr accuracy=0.860 params={'lr': 0.1, 'epochs': 5}
2. more-epochs accuracy=0.840 params={'lr': 0.01, 'epochs': 20}
3. baseline accuracy=0.810 params={'lr': 0.01, 'epochs': 5}
Best run: bigger-lrShow it to recruiters
Put it on GitHub
- 1
Create a free account at github.com if you do not have one yet.
- 2
Click + (top right), then New repository. Name it
experiment-tracker, choose Public, and click Create repository. - 3
Easiest way: on the new repository page, click uploading an existing file, drag in
tracker.py,README.md,.gitignore, and click Commit changes. Done. - 4
The way companies work: install Git from git-scm.com, open a terminal in your project folder, and run these. Replace
YOUR-USERNAMEwith your GitHub username. The twogit configlines are needed only once per computer. The first push asks you to sign in: on Windows a browser window opens. On Mac or Linux, GitHub does not accept your account password here, so use a personal access token as the password, or publish with GitHub Desktop instead.git config --global user.name "Your Name" git config --global user.email "you@example.com" git init git add . git commit -m "First version of experiment-tracker" git branch -M main git remote add origin https://github.com/YOUR-USERNAME/experiment-tracker.git git push -u origin main
- 5
Put the repository link on your resume and LinkedIn under Projects, with one line on what it does and one on what you learned. Recruiters open these.
Takeaways
- 1A class keeps the data (runs) and the actions (log, best) together.
- 2Saving to JSON makes data survive after the program stops.
- 3Raise clear errors on bad input instead of saving junk.