Supervised learning, gently

What a classifier is doing, and how to tell whether it worked

This primer supports the supervised coding exercise in Chapter 6 (§6.1.4) and the prediction exercises in Chapter 8. It assumes you have never trained a model. By the end you will know what the training/test split is for, what the model actually sees, and, most importantly, how to read the output honestly.

Everything below runs on synthetic data, so you can execute it without collecting anything first. The complete script is here: supervised_learning_demo.py. All numbers on this page are the real output of that script.

The problem in one sentence

You have hand-coded (or LLM-coded) labels for a few hundred documents and 20,000 more you have not coded. A supervised classifier learns the pattern that separates your labels and applies it to the rest.

That is the entire idea. The difficulty is not the fitting; it is knowing whether the result is good enough to use as a measure.

The three ideas that unlock everything else

1. The split

You divide your labeled data into a portion the model learns from and a portion it never sees until you evaluate it.

X_train, X_test, y_train, y_test = train_test_split(
    df["text"], df["label"], test_size=0.3, stratify=df["label"], random_state=42)

The split exists because the question is never “can the model reproduce labels it has already memorized?” but “will it generalize to documents it has not seen?” Evaluating a model on its training data is like grading students on the answer key they studied from.

Two details that matter in practice:

stratify keeps the class proportions the same in both halves. Without it, a rare category can end up almost absent from one side, and your evaluation becomes noise.
random_state fixes the random draw so the split is reproducible. Record it (Chapter 9).

2. The feature matrix

Models do not read text. Before training, every document becomes a row of numbers. With TF-IDF features, each column is a term and each cell says how characteristic that term is of that document, relative to the corpus:

	access	optimization	students
d0	0.108	0.200	0.113
d1	0.227	0.000	0.472
d2	0.223	0.206	0.116

Two things follow from this representation, and Chapter 6 (§6.4) develops both. Word order is gone: “reduced inequality” and “increased inequality” produce identical rows. And a term that appears in nearly every document gets down-weighted toward zero, even if it is conceptually central to your construct.

3. The output is a probability

Most classifiers return not just a label but a probability: ("equity", 0.83). You can threshold it, rank cases by it, or carry it into a later analysis. Chapter 9 (§9.4) warns that these probabilities are often poorly calibrated, so treat them as scores rather than as literal chances until you have checked.

A worked example

The demo builds 500 abstracts about education policy. 110 are equity-framed and 390 are efficiency-framed, so the category of interest is about a fifth of the corpus. The two kinds share most of their vocabulary (access, community, cost, and program appear in both senses), and 8% of the labels are deliberately flipped to imitate inconsistent upstream coding. That is a deliberately realistic setting, not a demo designed to succeed.

vec = TfidfVectorizer(min_df=2)
Xtr, Xte = vec.fit_transform(X_train), vec.transform(X_test)
clf = LogisticRegression(max_iter=1000, random_state=42).fit(Xtr, y_train)
pred = clf.predict(Xte)

The single most common beginner error

Notice fit_transform on the training set but transform on the test set. The vocabulary and term weights are learned from training data only. Calling fit_transform on everything at once lets information from the test set leak into training, and your reported performance becomes fiction. If you have an AI assistant write this code for you, this is the first thing to check (Chapter 6).

Reading the results honestly

Accuracy on the held-out set is 0.873. Good?

No. Start with the baseline. A model that ignores the text and always guesses the majority class gets 0.780. So all that machinery bought about nine points over a rule you could write on a napkin. Always compute this comparison; a headline accuracy without it is uninterpretable.

Now the confusion matrix, which sorts the 150 test documents by what they actually were (rows) versus what the model said (columns):

	predicted efficiency	predicted equity
actually efficiency	114	3
actually equity	16	17

And the class-level report:

              precision    recall  f1-score   support
  efficiency      0.877     0.974     0.923       117
      equity      0.850     0.515     0.642        33
    accuracy                          0.873       150

The story the aggregate number hid: the model finds barely half the equity documents (recall 0.515). Sixteen of the 33 equity abstracts were filed as efficiency. If your research question is about equity framing, the model is silently discarding half your evidence.

The three numbers, in plain language:

Precision (0.850): when the model says “equity,” how often is it right? Of 20 documents flagged equity, 17 genuinely were.
Recall (0.515): of the equity documents that exist, how many did the model find? 17 of 33.
F1 (0.642): the harmonic mean of the two, useful as a single summary but only after you have looked at both parts.

This is the marginal-cases problem from Chapter 5 (§5.7) in numerical form. The errors are not symmetric: 16 false negatives against 3 false positives. The minority category loses genuine members and gains a few impostors, so it ends up looking both smaller and noisier than it really is.

Does one split tell the truth?

A single train/test split is one draw. Cross-validation repeats the exercise across different partitions:

5-fold macro F1: [0.698 0.702 0.787 0.787 0.884]   mean 0.772, sd 0.068

The spread from 0.70 to 0.88 is the honest uncertainty in that mean. Reporting only “F1 = 0.77” would imply a precision the evidence does not support (Chapter 9, §9.4.4).

Can we fix the recall problem?

The standard first move is to tell the model to weight the rare class more heavily:

LogisticRegression(max_iter=1000, class_weight="balanced")

Result:

	precision	recall
equity, default	0.850	0.515
equity, balanced	0.720	0.545

Recall rises slightly; precision falls more. This is the honest outcome and worth sitting with: class weighting redistributes errors, it does not create information that is not in the data. When the minority signal is genuinely weak (shared vocabulary, noisy labels), better labels and better features help; a hyperparameter does not.

The audit checklist

Before treating classifier output as a variable in your analysis:

Report precision and recall per class, never accuracy alone.
Compare against the majority-class baseline.
Read the errors. Pull 10 false negatives and 10 false positives and read them. This is where you learn whether the model is confused or your coding rules are.
Check whether errors concentrate in a time period, venue, or group. Non-random error biases downstream estimates (Chapter 3, §3.5.6).
Report the spread across folds or seeds, not just the mean.
If your labels came from an LLM, remember the classifier inherits their bias (Chapter 6, Exercise 6.1.4, step 6).

Where to go next

The exercise this supports: Chapter 6, §6.1.4
Where the error asymmetry is theorized: Chapter 5, §5.7
The audit protocol in full: Chapter 6, §6.6.3
Prompting and its failure modes, if your training labels will come from an LLM