scikit-learn: classification
Train classifiers, read a confusion matrix, and tune the decision threshold.
Why it matters: Spam, sentiment, intent, and fraud are classification. Accuracy alone lies on imbalanced data.
Basics
The classifiers
Common starters:
- Logistic regression - fast baseline
- Decision tree - easy to read, overfits if deep
- Random forest - many trees voting
- KNN - needs scaled features
Start with logistic regression on tabular data.
Scale features for KNN and regularised linear models. Trees do not need scaling.
Tiny example
Score accuracy on a tiny label list.
y_true = ["spam", "ham", "spam", "ham"]
y_pred = ["spam", "ham", "ham", "ham"]
acc = sum(a == b for a, b in zip(y_true, y_pred)) / len(y_true)
print("accuracy", acc)Basics
Confusion matrix metrics
From true/false positives and negatives:
- Accuracy = correct / total (weak on imbalance)
- Precision = TP / (TP + FP)
- Recall = TP / (TP + FN)
- F1 = balance of precision and recall
If 99% are negative, always predicting negative scores 99% accuracy and catches nothing.
Tiny example
Count TP and FP for binary labels.
y_true = [1, 0, 1, 0, 1]
y_pred = [1, 0, 0, 0, 1]
tp = sum(t == 1 and p == 1 for t, p in zip(y_true, y_pred))
fp = sum(t == 0 and p == 1 for t, p in zip(y_true, y_pred))
print("tp", tp, "fp", fp)Basics
Thresholds
predict_proba returns class probabilities. The 0.5 cutoff is a choice.
Lower threshold -> more positives (higher recall, lower precision).
Raise threshold -> fewer, more certain positives.
Pick the threshold from the cost of each error type.
Tiny example
Apply a decision threshold to probability scores.
probs = [0.9, 0.2, 0.6, 0.4]
threshold = 0.5
preds = [1 if p >= threshold else 0 for p in probs]
print(preds)Then AI / ML
AI / ML examples
Same idea, used in real AI work. Press Try in lab to run it.
AI / ML example
AI / ML: logistic spam baseline
Print accuracy and a confusion matrix.
In AI / ML: Intent, spam, and triage routers often start as logistic regression.
example.py
Python
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
rng = np.random.default_rng(42)
n = 400
length = rng.normal(50, 20, n)
links = rng.integers(0, 6, n).astype(float)
y = ((-4 + 0.05 * length + 0.9 * links + rng.normal(0, 1, n)) > 0).astype(int)
X = np.column_stack([length, links])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
pred = LogisticRegression().fit(X_tr, y_tr).predict(X_te)
print("accuracy", round(accuracy_score(y_te, pred), 3))
print(confusion_matrix(y_te, pred))AI / ML example 2
AI / ML: accuracy trap on rare fraud
A dummy model can look great and catch zero fraud.
In AI / ML: Fraud and rare-event models are judged on recall of the positive class.
example.py
Python
import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, recall_score
rng = np.random.default_rng(0)
n = 1000
X = rng.normal(0, 1, (n, 3))
y = np.zeros(n, dtype=int)
fraud = rng.choice(n, size=20, replace=False)
y[fraud] = 1
X[fraud] += 2.0
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=1, stratify=y)
for name, clf in [("dummy", DummyClassifier(strategy="most_frequent")), ("logreg", LogisticRegression())]:
pred = clf.fit(X_tr, y_tr).predict(X_te)
print(name, "acc", round(accuracy_score(y_te, pred), 3),
"recall", round(recall_score(y_te, pred, zero_division=0), 3))Takeaways
- 1Accuracy hides failure on imbalanced data - read the confusion matrix.
- 2Precision matters when false positives cost; recall when false negatives cost.
- 30.5 is a choice: use predict_proba and set the threshold from real cost.