Supervised learning, gently

What a classifier is doing, and how to tell whether it worked

What a classifier is doing and how to tell whether it worked: a supervised learning primer for the coding and prediction exercises, assuming no prior training.

This primer introduces supervised learning for Chapter 6 (§6.1.4) and Chapter 8, assuming you have never trained a model. The complete script generates synthetic data and the evaluation results shown below, so you can run it without collecting data first.

The problem in one sentence

You have hand-coded (or LLM-coded) labels for a few hundred documents and 20,000 more you have not coded. A supervised classifier learns the pattern that separates your labels and applies it to the rest.

Before using those predictions as a measure, you need to know which documents the model gets wrong and what those errors mean for your research question.

The three ideas that do most of the work

1. The split

You divide your labeled data into a portion the model learns from and a portion it never sees until you evaluate it.

X_train, X_test, y_train, y_test = train_test_split(
    df["text"], df["label"], test_size=0.3, stratify=df["label"], random_state=42)

The split tests whether the model generalizes to unseen documents. Evaluating it on its training data is like grading students on the answer key they studied from.

Two details that matter in practice:

stratify keeps class proportions approximately similar in both sets. This helps retain examples of a rare category for evaluation, provided you have enough labeled cases.
random_state fixes the random draw so the split is reproducible. Record it (Chapter 9).

2. The feature matrix

Models do not read text. Before training, every document becomes a row of numbers. With TF-IDF features, each column is a term and each cell says how characteristic that term is of that document, relative to the corpus:

	access	optimization	students
d0	0.108	0.200	0.113
d1	0.227	0.000	0.472
d2	0.223	0.206	0.116

This representation loses word order: “policy shapes research” and “research shapes policy” have the same unigram counts. Frequent terms receive less inverse-document-frequency weight than rare ones. That weighting does not tell you which terms are theoretically important (Chapter 6, §6.4).

3. Scores and classification thresholds

The logistic regression used here can return a probability estimate as well as a label, for example an estimated probability of 0.83 for “equity.” You can use a threshold to choose a label or rank cases for inspection. Check calibration before treating the score as a reliable estimate of how often a prediction is correct (Chapter 9, §9.4).

A worked example

The demo generates 500 short texts from education-policy vocabularies. It starts with 80 equity and 420 efficiency texts, then flips 8% of the labels to simulate coding errors. The resulting labels are 110 equity and 390 efficiency. The categories share words such as access, community, cost, and program, making the classification task less clear-cut.

The evaluation below compares predictions with these noisy labels. Agreement with them is not the same as recovering the categories used to generate the texts.

vec = TfidfVectorizer(min_df=2)
Xtr, Xte = vec.fit_transform(X_train), vec.transform(X_test)
clf = LogisticRegression(max_iter=1000, random_state=42).fit(Xtr, y_train)
pred = clf.predict(Xte)

Keep test data out of feature fitting

Notice fit_transform on the training set but transform on the test set. The vocabulary and term weights are learned from training data only. Calling fit_transform on everything at once lets information from the test set leak into training, which can make performance appear better than it would be on new data. Check this when reviewing AI-generated code (Chapter 6).

Reading the errors behind accuracy

Accuracy on the held-out set is 0.873. To interpret it, start with the baseline: always predicting the majority class gives 0.780. The fitted model improves on that by about nine percentage points, but you still need to see which errors remain.

The confusion matrix sorts the 150 test documents by their test-set labels (rows) and predicted labels (columns):

	predicted efficiency	predicted equity
labeled efficiency	114	3
labeled equity	16	17

And the class-level report:

              precision    recall  f1-score   support
  efficiency      0.877     0.974     0.923       117
      equity      0.850     0.515     0.642        33
    accuracy                          0.873       150

Read the three metrics for equity:

Precision (0.850): when the model says “equity,” how often is it right? Of 20 documents predicted as equity, 17 carried that label in the test set.
Recall (0.515): of the documents labeled equity, how many did the model find? 17 of 33.
F1 (0.642): the harmonic mean of the two, useful as a single summary but only after you have looked at both parts.

If you selected equity documents using these predictions, you would miss sixteen labeled cases and include three labeled efficiency. Overall accuracy hides that asymmetry, the marginal-cases problem from Chapter 5 (§5.7). Here the predicted set is smaller; other error balances could make it larger.

Variation across data splits

A single train/test split is one draw. Cross-validation repeats the exercise across different partitions. The demo uses a pipeline that refits text features within each training fold, following scikit-learn’s leakage guidance. Macro F1 averages the two classes’ F1 scores with equal weight:

5-fold macro F1: [0.698 0.702 0.787 0.787 0.884]   mean 0.772, sd 0.068

Scores range from about 0.70 to 0.88 across these folds. Report that variation alongside the mean: performance depends partly on which documents were held out. The fold standard deviation is not a confidence interval for performance on a new population.

Can we fix the recall problem?

One option is to give the rare class more weight during fitting:

LogisticRegression(max_iter=1000, class_weight="balanced")

Result:

	precision	recall
equity, default	0.850	0.515
equity, balanced	0.720	0.545

In this run, weighting raises recall slightly and lowers precision. Whether that tradeoff is useful depends on the cost of missed cases and incorrect inclusions in your study. Inspect the coding rules and text features as well as model settings; the errors may begin before the classifier is fitted.

The audit checklist

Before treating classifier output as a variable in your analysis:

Report precision and recall per class, never accuracy alone.
Compare against the majority-class baseline.
Read the errors. Inspect ten false negatives and ten false positives, or all available cases if there are fewer. Compare each with the text and the coding rule.
Check whether errors concentrate in a time period, venue, or group, and assess how that pattern could affect the comparison you plan to make (Chapter 3, §3.5.6).
Report the spread across folds or seeds, not just the mean.
If your labels came from an LLM, check whether the classifier reproduces their errors (Chapter 6, Exercise 6.1.4, step 6).

Where to go next

The exercise this supports: Chapter 6, §6.1.4
Where the error asymmetry is theorized: Chapter 5, §5.7
The audit protocol in full: Chapter 6, §6.6.3
Prompting and its failure modes, if your training labels will come from an LLM