Chapter 6 — Automated coding

Corpus preparation, coding prompts, and classification materials

Chapter 6 turns your OpenAlex corpus into coded data using three instruments in sequence: a keyword dictionary, prompt-based LLM coding, and a supervised classifier. This page carries the technical support for each step.

What you will produce. One coded dataset, three ways, plus the validity audit that compares them. The comparison is the deliverable: three instruments disagreeing on the same documents is the evidence you reason from.

Materials for this chapter

Text representation notebook

Preprocessing, NER, embeddings, and document similarity, with committed output from a real run.

Topic modeling notebook

One corpus through three topic models, then a text classifier. Read the notebook notes first, because these carry heavy dependencies.

Automated coding tools

Current libraries, models, and coding agents, kept up to date as software changes.

Studies on concept representation

Forty-eight published studies annotated for how they built their measures, and what evidence they offered that the measures worked.

The conceptual argument these instruments implement is in Chapter 5, which filters the studies database to concept representation.

Before you start

You need the corpus from Chapter 4 and a working environment (Getting started). If any of the following are unfamiliar, take the relevant primer first; the chapter assumes all three.

If you have never… Read
worked with JSON, tokenization, or parquet Working with text data
written a coding prompt Prompting and its failure modes
trained a classifier Supervised learning, gently

Step 1 — The coding corpus (§6.1.1)

The deliverable is one document-level table that every later exercise reads from: doc_id, text, year, venue, plus whatever metadata may become controls. The inverted-index reconstruction, duplicate handling, and short-document flagging are walked through in Working with text data.

Save it once, as parquet, and treat it as frozen for the rest of the chapter. Every instrument you build is then comparable, because they all coded the same documents.

Step 2 — Dictionary coding (§6.1.2)

import re, pandas as pd

DICT = {
    "equity":     ["equity", "inequality", "disparity", "justice",
                   "marginalized", "underrepresented", "barriers"],
    "efficiency": ["efficiency", "cost-effective", "optimization",
                   "scalable", "productivity", "throughput"],
}

def score(text, terms):
    t = text.lower()
    hits = sum(len(re.findall(rf"\b{re.escape(w)}\w*\b", t)) for w in terms)
    return hits, 1000 * hits / max(len(t.split()), 1)   # count and rate per 1k words

for cat, terms in DICT.items():
    df[[f"{cat}_n", f"{cat}_per1k"]] = df["text"].apply(
        lambda s: pd.Series(score(s, terms)))

Keep both the raw count and the length-normalized rate: the pair is easier to interpret than either alone. Then read the top- and bottom-scoring documents for each category. A first draft dictionary is always wrong in instructive ways, and the fastest way to find out how is to read the documents it ranks most confidently.

Step 3 — LLM coding and the stress test (§6.1.3)

The prompt template, the six-part promptbook anatomy, the four induced failure modes, and the agreement-check protocol are all on Prompting and its failure modes.

Exercise 6.1.3 asks you to keep two sets of labels: those from your careful prompt and those from the deliberately biased variant. Save both. Exercise 6.1.4 uses the biased set.

Step 4 — Supervised classification (§6.1.4)

The full walkthrough, with a runnable script and real output, is at Supervised learning, gently. The short version:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

X_tr, X_te, y_tr, y_te = train_test_split(
    df["text"], df["llm_label"], test_size=0.3, stratify=df["llm_label"], random_state=42)

vec = TfidfVectorizer(min_df=2)
Xtr, Xte = vec.fit_transform(X_tr), vec.transform(X_te)   # fit on TRAIN only
clf = LogisticRegression(max_iter=1000, random_state=42).fit(Xtr, y_tr)

print(confusion_matrix(y_te, clf.predict(Xte)))
print(classification_report(y_te, clf.predict(Xte), digits=3))

Then repeat with the biased labels and compare the two classifiers’ predicted class distributions. That comparison is the point of the exercise: label bias does not wash out downstream, it propagates.

Step 5 — Topic modeling (§6.2)

Library choices and their trade-offs are in the automated coding tool guide. Two practices matter more than the library:

  • Label topics with representative documents, not top terms alone. Top terms are ambiguous: network, diffusion, influence, spread, cascade could be epidemiology or information spread. Reading three representative documents settles it.
  • Change one thing per run and log settings, seed, and outputs in memory/runs/. Topic models are sensitive to preprocessing and to the number of topics; without a run log you cannot tell which change produced which result.

For the LLM-assisted validation step, ask a model to label each topic from its top terms and representative documents without showing it your labels, then compare. Agreement is weak evidence of interpretability; disagreement is a flag worth close reading.

Step 6 — The validity audit (§6.6)

The audit checklist and the marginal-cases worked example are in the chapter itself. The computational piece is straightforward and non-negotiable:

print(classification_report(y_true, y_pred, digits=3))   # per class, always

Report precision and recall for the minority class, compare against a majority-class baseline, read ten false negatives and ten false positives, and check whether errors concentrate in any subgroup of theoretical interest.

Where to go next