Machine Learning

Run a complete example · 01

Build a majority-label baseline

Use training labels to choose a rule, then separate test labels to score it.

from collections import Counter
train_y = [0, 0, 0, 1]
test_y = [0, 1, 1, 0]
counts = Counter(train_y)
baseline = min(counts, key=lambda label: (-counts[label], label))
predictions = [baseline] * len(test_y)
correct = sum(pred == true for pred, true in zip(predictions, test_y))
print('baseline class:', baseline)
print('predictions:', predictions)
print(f'test accuracy: {correct / len(test_y):.2f}')

python features-labels-baselines-ml-majority-baseline.py

baseline class: 0
predictions: [0, 0, 0, 0]
test accuracy: 0.50

Read the code step by step

Counter counts training labels. min orders by negative count, so the largest count comes first; the class ID resolves a tie. List repetition makes one prediction per test row. zip pairs predictions with labels, and sum counts True comparisons as one. The output is a baseline, not evidence that word count or links are useful.

Run a complete example · 01

Keep complete devices together

Build the 120-row split without a random library call.

groups = [device for device in range(12) for _ in range(10)]
train = [i for i, device in enumerate(groups) if device < 8]
valid = [i for i, device in enumerate(groups) if 8 <= device < 10]
test = [i for i, device in enumerate(groups) if device >= 10]
train_devices = {groups[i] for i in train}
test_devices = {groups[i] for i in test}
print('rows:', len(train), len(valid), len(test))
print('shared devices:', sorted(train_devices & test_devices))
print('test error:', 2 / len(test))

python framing-splits-leakage-ml-group-split.py

rows: 80 20 20
shared devices: []
test error: 0.1

Read the code step by step

The first list repeats each device ID ten times. enumerate supplies row positions. The three conditions assign complete groups. A set removes repeated IDs; & finds shared IDs. The empty list confirms no training/test device overlap. The final line uses an assumed two errors; it does not train or evaluate a real sensor model.

Run a complete example · 02

Expose leakage with a deliberately simple memorizer

Mina records two readings from each of four devices. She trains on one reading per device and tests on the other. A model that only remembers device IDs scores perfectly. Then she tests on entirely new devices and the score falls to one half. The model did not suddenly get worse. The first split answered whether it could recognise familiar devices, while the second asked about new ones.

rows = [(device, label) for device, label in
        [("A", 0), ("B", 1), ("C", 0), ("D", 1)] for _ in range(2)]

def evaluate(train, test):
    memory = {device: label for device, label in train}
    fallback = int(sum(label for _, label in train) > len(train) / 2)
    predictions = [memory.get(device, fallback) for device, _ in test]
    accuracy = sum(p == label for p, (_, label) in zip(predictions, test)) / len(test)
    overlap = {d for d, _ in train} & {d for d, _ in test}
    return accuracy, sorted(overlap)

row_train, row_test = rows[::2], rows[1::2]
group_train = [row for row in rows if row[0] in {"A", "B"}]
group_test = [row for row in rows if row[0] in {"C", "D"}]
row_score, overlap = evaluate(row_train, row_test)
group_score, clean_overlap = evaluate(group_train, group_test)
assert row_score == 1.0 and group_score == 0.5 and not clean_overlap
print(f"row split: accuracy={row_score:.2f}; shared devices={overlap}")
print(f"group split: accuracy={group_score:.2f}; shared devices={clean_overlap}")

python framing-splits-leakage-memorizer-leakage-audit.py

row split: accuracy=1.00; shared devices=['A', 'B', 'C', 'D']
group split: accuracy=0.50; shared devices=[]

Follow the execution

1. Construct repeated measurements with a constant label per device. These synthetic rows make the shortcut explicit.

2. Fit the memory and majority fallback only on training rows. The tie rule chooses label zero.

3. Compare a row split with a disjoint-device split. Count shared IDs as well as accuracy.

Read the code step by step

Accuracy=(number of correct test predictions)/(number of test rows). The row test has four correct predictions out of four. The grouped test has two correct out of four because both unseen devices receive the training-only fallback zero, and only device C has label zero.

The useful unit of separation depends on deployment. For a new patient, device or video, keep that complete group out of training. For future events on known devices, use a suitable time split instead. Group separation alone does not stop future information leaking through preprocessing. Fit imputers, scalers and feature selection only on the training portion of each fold.

This is a diagnostic toy, not evidence that a particular real model has 50% accuracy. A deliberately weak baseline can reveal a shortcut more clearly than a complex model. The official scikit-learn cross-validation guide provides group-aware splitters; choose the grouping variable from the question you want to answer.

Run a complete example · 01

Fit a median, then reuse it

NaN represents unknown numeric entries in this example.

import numpy as np
train = np.array([2., np.nan, 6., 4.])
test = np.array([np.nan, 10.])
missing = np.isnan(train)
median = np.median(train[~missing])
filled_train = np.where(missing, median, train)
filled_test = np.where(np.isnan(test), median, test)
print(f'training missing fraction: {missing.mean():.2f}')
print('training flag:', missing.astype(int).tolist())
print('filled training:', filled_train.tolist())
print('filled test:', filled_test.tolist())

python missing-data-eda-ml-median-imputation.py

training missing fraction: 0.25
training flag: [0, 1, 0, 0]
filled training: [2.0, 4.0, 6.0, 4.0]
filled test: [4.0, 10.0]

Read the code step by step

isnan marks missing entries. ~ reverses the Boolean mask so indexing selects observed training values. median finds their middle value. where selects the replacement only at missing positions. astype(int) makes flags readable as zero and one. The test uses the same saved median. This prepares numeric data; it does not show that filling improves prediction.

Run a complete example · 01

Encode known and unseen categories

An explicit last column records unknown categories.

vocabulary = ['paper', 'wood']
def encode(value):
    return [int(value == name) for name in vocabulary] + [int(value not in vocabulary)]
print('columns:', vocabulary + ['unknown'])
for value in ['wood', 'paper', 'metal']:
    row = encode(value)
    print(value, row, 'sum:', sum(row))

python categorical-preprocessing-ml-category-columns.py

columns: ['paper', 'wood', 'unknown']
wood [0, 1, 0] sum: 1
paper [1, 0, 0] sum: 1
metal [0, 0, 1] sum: 1

Read the code step by step

The list comprehension compares a value with each saved name. int converts each comparison to zero or one. The final entry tests membership in the whole vocabulary. Printing the sum checks that exactly one indicator is active. This encoder prepares features only. It neither trains a classifier nor learns how to handle metal correctly.

Run a complete example · 01

Compare a line and a constant

Use the same three rows for an arithmetic demonstration.

import numpy as np
x = np.array([1., 2., 3.])
y = np.array([3., 5., 8.])
prediction = 1 + 2 * x
residual = prediction - y
mse = np.mean(residual ** 2)
baseline_mse = np.mean((y.mean() - y) ** 2)
print('prediction:', prediction.tolist())
print(f'MSE: {mse:.6f}; RMSE: {np.sqrt(mse):.6f}')
print(f'constant MSE: {baseline_mse:.6f}')

python linear-regression-mse-ml-line-loss.py

prediction: [3.0, 5.0, 7.0]
MSE: 0.333333; RMSE: 0.577350
constant MSE: 4.222222

Read the code step by step

np.array stores ordered numbers. Multiplication and subtraction act on matching entries. **2 squares each residual; mean adds and divides by three. sqrt restores the original unit. y.mean supplies the constant baseline. These rows are used only to compare arithmetic. The chosen line is not a fitted least-squares optimum or an independent test result.

Run a complete example · 01

Standardize and take one step

All gradients below use the old parameter values.

import numpy as np
x = np.array([2., 6.]); y = np.array([1., 3.])
z = (x - x.mean()) / x.std()
w, b, rate = 0., 0., 0.1
error = w * z + b - y
gw = 2 * np.mean(error * z)
gb = 2 * np.mean(error)
w, b = w - rate * gw, b - rate * gb
print('scaled inputs:', z.tolist())
print(f'gradients: {gw:.1f}, {gb:.1f}')
print(f'weights: {w:.1f}, {b:.1f}')
print(f'loss: {np.mean((w * z + b - y) ** 2):.3f}')

python gradient-descent-scaling-ml-gradient-step.py

scaled inputs: [-1.0, 1.0]
gradients: -2.0, -4.0
weights: 0.2, 0.4
loss: 3.200

Read the code step by step

mean and std compute training statistics; NumPy std uses the population convention here. error stores old residuals. Multiplying by z gives the weight contribution to the derivative. Multiplying means by two comes from differentiating a square. The tuple assignment updates both parameters from old gradients. The last line recomputes the loss. This is one exact step for a tiny linear model, not a full training workflow.

Run a complete example · 01

One probability, two decisions

Use illustrative fixed weights, not a trained sky classifier.

from math import exp
x, w, b = 2., 0.8, -1.2
score = b + w * x
probability = 1 / (1 + exp(-score))
print(f'score: {score:.4f}; probability: {probability:.4f}')
for threshold in [0.50, 0.65]:
    label = int(probability >= threshold)
    print(f'threshold {threshold:.2f}: class {label}')

python logistic-thresholds-ml-sigmoid-cutoffs.py

score: 0.4000; probability: 0.5987
threshold 0.50: class 1
threshold 0.65: class 0

Read the code step by step

The first calculation creates a linear score. exp implements the exponential in sigmoid. The loop reuses one probability. >= compares it with each cutoff; int turns True or False into one or zero. The output changes class while keeping the score fixed. No data were used to fit these weights or to justify either threshold.

Run a complete example · 02

Choose a decision threshold with an explicit error cost

A school machine flags damaged practice parts for inspection. Missing a damaged part costs four effort units; inspecting a good part costs one. Mina first uses the usual 0.5 threshold. She then writes out the two types of mistakes on a small validation set. Lowering the threshold catches a missed damaged part while adding only one unnecessary inspection. She freezes the choice before looking at the final test set.

scores = [0.10, 0.35, 0.45, 0.60, 0.80]
truth = [0, 1, 0, 1, 1]

def cost_at(threshold):
    predicted = [int(score >= threshold) for score in scores]
    fp = sum(p == 1 and y == 0 for p, y in zip(predicted, truth))
    fn = sum(p == 0 and y == 1 for p, y in zip(predicted, truth))
    return fp + 4 * fn, fp, fn

candidates = [0.30, 0.50, 0.70, 1.10]
for threshold in candidates:
    cost, fp, fn = cost_at(threshold)
    print(f"threshold={threshold:.2f} FP={fp} FN={fn} cost={cost}")
best = min(candidates, key=lambda t: (cost_at(t)[0], t))
assert best == 0.30 and cost_at(best) == (1, 1, 0)
print(f"selected on validation: {best:.2f}")

python logistic-thresholds-validation-cost-threshold.py

threshold=0.30 FP=1 FN=0 cost=1
threshold=0.50 FP=0 FN=1 cost=4
threshold=0.70 FP=0 FN=2 cost=8
threshold=1.10 FP=0 FN=3 cost=12
selected on validation: 0.30

Follow the execution

1. State costs before comparing thresholds. Here the objective is FP+4*FN, not accuracy.

2. For each candidate, form predictions with score >= threshold and count both error types.

3. Minimize validation cost and state the tie rule. Keep the final test set separate from this choice.

Read the code step by step

At threshold 0.30, the prediction vector is [0,1,1,1,1]. There is one false positive and no false negative, so total cost is 1. At 0.50 there are no false positives but one false negative, so cost is 4. The all-negative candidate 1.10 costs 12. Report cost per case by dividing by five if different validation-set sizes must be compared.

If p is a calibrated probability, false-positive cost is C_FP, false-negative cost is C_FN and correct decisions cost zero, predict positive when C_FP*(1-p) <= C_FN*p. Rearranging gives p >= C_FP/(C_FP+C_FN), which is 0.2 here. That population decision rule is not the same as selecting among a finite grid using five observed labels. Scores need not be calibrated probabilities.

Costs, prevalence and calibration can change after deployment. A threshold does not repair a poor ranking or an invalid test split. This is a tiny illustration; use enough representative validation data and inspect uncertainty before drawing practical conclusions.

Run a complete example · 01

Compute four metrics from counts

These counts describe one threshold on an invented evaluation set.

tp, tn, fp, fn = 8, 78, 12, 2
total = tp + tn + fp + fn
accuracy = (tp + tn) / total
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * tp / (2 * tp + fp + fn)
for name, value in [('accuracy', accuracy), ('precision', precision),
                    ('recall', recall), ('F1', f1)]:
    print(f'{name}: {value:.3f}')
print('clips to review:', tp + fp)

python metrics-imbalance-ml-confusion-counts.py

accuracy: 0.860
precision: 0.400
recall: 0.800
F1: 0.533
clips to review: 20

Read the code step by step

The four counts cover all hundred clips once. Each division matches a different question. The F1 expression uses counts directly and equals the harmonic mean when precision and recall are defined. The loop prints each ratio to three decimal places. The code assumes nonzero denominators in this example. It computes metrics; it does not train an audio detector.

Run a complete example · 01

Inspect every nearest-neighbour vote

The table is synthetic and already uses comparable feature scales.

import numpy as np
x = np.array([[0.,0.], [1.,0.], [2.,0.], [4.,0.]])
y = np.array([0,0,1,1])
query = np.array([1.6,0.])
squared = np.sum((x - query) ** 2, axis=1)
order = np.argsort(squared, kind='stable')
print('squared distances:', np.round(squared, 2).tolist())
for k in [1,3]:
    labels = y[order[:k]]
    print('k:', k, 'labels:', labels.tolist(), 'prediction:', int(labels.mean() > 0.5))

python nearest-neighbours-ml-knn-vote.py

squared distances: [2.56, 0.36, 0.16, 5.76]
k: 1 labels: [1] prediction: 1
k: 3 labels: [1, 0, 0] prediction: 0

Read the code step by step

Subtracting query uses it for every row. sum(axis=1) adds feature contributions within each row. argsort returns row indices from smallest distance to largest; stable ordering resolves equal distances by row order. Slicing [:k] selects neighbours. Their mean counts the fraction of ones. This is a genuine tiny neighbour classifier, with no evidence about real audio accuracy.

Run a complete example · 01

Compute a two-word classifier

Use supplied training count totals to show every probability.

import numpy as np
counts = np.array([[3.,1.], [1.,3.]])
word_prob = (counts + 1) / (counts.sum(axis=1, keepdims=True) + 2)
query = np.array([1.,0.])
log_scores = np.log([0.5,0.5]) + np.log(word_prob) @ query
weights = np.exp(log_scores - log_scores.max())
probabilities = weights / weights.sum()
print('word probabilities:', np.round(word_prob, 3).tolist())
print('class probabilities:', np.round(probabilities, 3).tolist())
print('class:', ['spam','ordinary'][int(np.argmax(probabilities))])

python text-naive-bayes-ml-naive-bayes-counts.py

word probabilities: [[0.667, 0.333], [0.333, 0.667]]
class probabilities: [0.667, 0.333]
class: spam

Read the code step by step

Rows represent classes and columns represent words. sum with keepdims preserves a column-shaped denominator for each row. Adding one implements smoothing. @ adds query-count-weighted log probabilities. Subtracting the largest log score keeps exponentials numerically manageable without changing the normalized result. argmax selects the higher score. The classifier is based only on these counts, not a trained language model.

Run a complete example · 01

Score a split and average trees

Compute the local criterion without building a whole tree.

def gini(positive, total):
    p = positive / total
    return 2 * p * (1 - p)
parent = gini(6, 10)
children = 0.5 * gini(4, 5) + 0.5 * gini(2, 5)
probabilities = [0.2, 0.7, 0.9]
print(f'parent: {parent:.2f}; children: {children:.2f}')
print(f'gain: {parent - children:.2f}')
print(f'forest probability: {sum(probabilities) / len(probabilities):.2f}')

python trees-ensembles-ml-gini-forest.py

parent: 0.48; children: 0.40
gain: 0.08
forest probability: 0.60

Read the code step by step

The function converts counts into a positive fraction, then computes binary Gini impurity. Equal child sizes give weights of 0.5. Subtracting child impurity measures the local gain. sum divided by len averages tree probabilities. The numbers demonstrate one candidate split and one ensemble rule; no tree search or forest training occurs.

Run a complete example · 01

Fit one residual stump

The split at x<2 is fixed so we can inspect both leaf means.

import numpy as np
x = np.array([0.,1.,2.,3.]); y = np.array([1.,1.,3.,3.])
prediction = np.full_like(y, y.mean())
residual = y - prediction
left = x < 2
correction = np.where(left, residual[left].mean(), residual[~left].mean())
updated = prediction + 0.5 * correction
print('residual:', residual.tolist())
print('updated:', updated.tolist())
print(f'MSE before: {np.mean((prediction-y)**2):.2f}')
print(f'MSE after: {np.mean((updated-y)**2):.2f}')

python boosting-residuals-ml-residual-stump.py

residual: [-1.0, -1.0, 1.0, 1.0]
updated: [1.5, 1.5, 2.5, 2.5]
MSE before: 1.00
MSE after: 0.25

Read the code step by step

full_like makes a constant starting prediction with one entry per target. The Boolean left mask defines two leaves. Each leaf uses the mean residual, which minimizes its squared residual error. where sends each row its leaf correction. Adding half the correction performs one boosting step. No split search, multiple rounds, or independent validation are included.

Run a complete example · 01

Measure groups and PCA variance

A rectangle makes both calculations visible.

import numpy as np
x = np.array([[0., 0.], [0., 2.], [4., 0.], [4., 2.]])
centers = np.array([[0., 1.], [4., 1.]])
assignment = np.array([0, 0, 1, 1])
loss = np.sum((x - centers[assignment]) ** 2)
centered = x - x.mean(axis=0)
covariance = centered.T @ centered / len(x)
values = np.linalg.eigvalsh(covariance)
print(f'cluster loss: {loss:.1f}')
print('covariance:', covariance.tolist())
print(f'first variance share: {values[-1] / values.sum():.2f}')

python clustering-pca-ml-cluster-pca-geometry.py

cluster loss: 4.0
covariance: [[4.0, 0.0], [0.0, 1.0]]
first variance share: 0.80

Read the code step by step

centers[assignment] selects one center per row. Squaring and summing yields the grouping loss. mean(axis=0) computes each column mean. .T swaps axes; @ multiplies matrices to form population covariance. eigvalsh returns its eigenvalues in ascending order, so the final one is largest. This code measures chosen groups and variance. It does not discover rock types or run an iterative k-means fit.

Run a complete example · 01

Rank three tagged videos

The tags are hand chosen; this example learns no embeddings.

import numpy as np
names = ['astronomy', 'robotics', 'mixed']
items = np.array([[1.,0.], [0.,1.], [1.,1.]])
profile = np.array([1.,1.])
scores = (items @ profile) / (np.linalg.norm(items, axis=1) * np.linalg.norm(profile))
order = np.argsort(-scores, kind='stable')
for index in order:
    print(f'{names[index]}: {scores[index]:.3f}')

python content-recommendations-ml-cosine-recommendations.py

mixed: 1.000
astronomy: 0.707
robotics: 0.707

Read the code step by step

@ forms one dot product per item. norm computes vector length; axis=1 keeps item lengths separate. Division removes length effects. Sorting negative scores puts larger values first, with input order resolving ties. The mixed item ranks first under these tags. This is an inspectable scoring rule, not a validated viewer-preference model.

Run a complete example · 01

Separate fitting objective and validation

Candidate summaries illustrate two different comparisons.

candidates = {'A': (1.0, 9), 'B': (1.4, 1)}
strength = 0.1
for name, (mse, squared_weights) in candidates.items():
    print(f'{name} objective: {mse + strength * squared_weights:.1f}')
folds = {0.0: [1.0, 1.4, 1.2], 0.1: [0.9, 1.0, 1.1]}
means = {value: sum(errors) / len(errors) for value, errors in folds.items()}
for value, score in means.items():
    print(f'lambda {value:.1f}: validation MSE {score:.1f}')
print('chosen lambda:', min(means, key=means.get))

python validation-regularization-shift-ml-regularization-validation.py

A objective: 1.9
B objective: 1.5
lambda 0.0: validation MSE 1.2
lambda 0.1: validation MSE 1.0
chosen lambda: 0.1

Read the code step by step

Each candidate stores its training MSE and squared-weight sum. The first loop adds the penalty. The folds dictionary stores independent illustrative validation summaries. The next calculation averages equal-sized folds. min with key=means.get selects the strength with the smallest validation error. It does not select by penalized training loss. These are supplied summaries, not results from a fitted model.

Run a complete example · 01

Inspect a probability bin

All four examples belong to one bin near probability 0.8.

import numpy as np
probability = np.array([0.8,0.8,0.8,0.8])
label = np.array([1.,1.,0.,0.])
brier = np.mean((probability - label) ** 2)
print('bin count:', len(label))
print(f'mean probability: {probability.mean():.2f}')
print(f'observed positive fraction: {label.mean():.2f}')
print(f'Brier score: {brier:.2f}')

python calibration-monitoring-ml-calibration-bin.py

bin count: 4
mean probability: 0.80
observed positive fraction: 0.50
Brier score: 0.34

Read the code step by step

The arrays pair each probability with its true label. Subtraction, squaring, and mean implement the Brier score. The label mean is the positive fraction because labels are zero or one. The printed count reminds us how little evidence four cases provide. The script diagnoses supplied predictions; it does not fit a calibrator or monitor a live system.