Chapter 6 — Automated coding

Corpus preparation, coding prompts, and classification materials

Corpus preparation, coding prompts, and classification materials for Chapter 6: dictionary, LLM, and classifier coding of your corpus, closed by a validity audit.

Chapter 6 turns your OpenAlex corpus into coded data using three instruments in sequence: a keyword dictionary, prompt-based LLM coding, and a supervised classifier. Apply them to the same documents so you can investigate how their measurement choices affect the results.

What you will produce. Three sets of labels for one corpus and a validity audit comparing them. Read the disagreements against your construct definition and source documents, then explain what each instrument captures or misses.

Materials for this chapter

Text representation notebook

Preprocessing, named-entity recognition, embeddings, and document similarity, with saved outputs you can inspect.

Topic modeling notebook

Three topic-modeling approaches, followed by links for reviewing a text classifier. Read the notebook notes first, because these carry heavy dependencies.

Automated coding tools

Libraries, models, and coding agents, reviewed as software changes.

Studies on concept representation

Forty-eight published studies annotated for how they built their measures, and what evidence they offered that the measures worked.

Chapter 5 develops the measurement reasoning behind these instruments.

Before you start

You need the corpus from Chapter 4 and a working environment (Getting started). If any of the following are unfamiliar, take the relevant primer first; the chapter assumes all three.

If you have never… Read
worked with JSON, tokenization, or parquet Working with text data
written a coding prompt Prompting and its failure modes
trained a classifier Supervised learning, gently

Step 1 — The coding corpus (§6.1.1)

The deliverable is one document-level table that every later exercise reads from: doc_id, text, year, venue, plus whatever metadata may become controls. The inverted-index reconstruction, duplicate handling, and short-document flagging are walked through in Working with text data.

Save it once, as parquet, and treat it as frozen for the rest of the chapter. Every instrument you build is then comparable, because they all coded the same documents.

Step 2 — Dictionary coding (§6.1.2)

import re, pandas as pd

DICT = {
    "equity":     ["equity", "inequality", "disparity", "justice",
                   "marginalized", "underrepresented", "barriers"],
    "efficiency": ["efficiency", "cost-effective", "optimization",
                   "scalable", "productivity", "throughput"],
}

def score(text, terms):
    t = text.lower()
    hits = sum(len(re.findall(rf"\b{re.escape(w)}\w*\b", t)) for w in terms)
    return hits, 1000 * hits / max(len(t.split()), 1)   # count and rate per 1k words

for cat, terms in DICT.items():
    df[[f"{cat}_n", f"{cat}_per1k"]] = df["text"].apply(
        lambda s: pd.Series(score(s, terms)))

Keep both the raw count and the length-normalized rate: the pair is easier to interpret than either alone. Then read the top- and bottom-scoring documents for each category. A first draft dictionary may count a term in the wrong sense or miss wording you did not anticipate. Inspect these cases and use them to revise the coding rules.

Step 3 — LLM coding and the stress test (§6.1.3)

The prompt template, the six-part promptbook anatomy, the four induced failure modes, and the agreement-check protocol are all on Prompting and its failure modes.

Save labels from both your careful prompt and the deliberately biased variant in Exercise 6.1.3; Exercise 6.1.4 uses the biased set.

Step 4 — Supervised classification (§6.1.4)

For a full walkthrough with a runnable script and its output, see Supervised learning, gently. Start with this example:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

X_tr, X_te, y_tr, y_te = train_test_split(
    df["text"], df["llm_label"], test_size=0.3, stratify=df["llm_label"], random_state=42)

vec = TfidfVectorizer(min_df=2)
Xtr, Xte = vec.fit_transform(X_tr), vec.transform(X_te)   # fit on TRAIN only
clf = LogisticRegression(max_iter=1000, random_state=42).fit(Xtr, y_tr)

print(confusion_matrix(y_te, clf.predict(Xte)))
print(classification_report(y_te, clf.predict(Xte), digits=3))

Repeat with the deliberately biased labels and compare the predicted class distributions. Inspect documents whose labels change. The exercise lets you trace whether errors in the training labels carry into the classifier’s predictions.

Step 5 — Topic modeling (§6.2)

Whichever library you choose from the automated coding tool guide:

  • Label topics with representative documents, not top terms alone. Top terms are ambiguous: network, diffusion, influence, spread, cascade could be epidemiology or information spread. Read representative documents to assess the theme, and inspect more cases if the interpretation remains ambiguous.
  • Change one thing per run and log settings, seed, and outputs in memory/runs/. Topic models are sensitive to preprocessing and to the number of topics; without a run log you cannot tell which change produced which result.

For the LLM-assisted validation step, ask a model to label each topic from its top terms and representative documents without showing it your labels, then compare. Agreement is weak evidence of interpretability; disagreement is a flag worth close reading.

Step 6 — The validity audit (§6.6)

The audit checklist and the marginal-cases worked example are in the chapter itself. The code below reports one part of that audit:

print(classification_report(y_true, y_pred, digits=3))   # per class, always

Report precision and recall for each class and compare against a majority-class baseline. Read a sample of false negatives and false positives, starting with ten of each if available, and check whether errors concentrate in a subgroup of theoretical interest.

Where to go next