Tools: automated coding

Dictionaries, classifiers, topic models, and LLM coding

Tools for Chapter 6: dictionary coding, supervised classification, topic models, and LLM coding, with checks for the labels each produces.

Chapter 6 compares ways to turn text into measures. Choose tools that let you inspect how those measures were produced and test them against your coding rules. Tool selection reviewed August 2026; explanatory notes revised September 2026.

Dictionary coding

  • quanteda (R): dictionary lookup, keyword-in-context, and document-feature matrices.
  • LIWC: psycholinguistic dictionaries; commercial license.
  • Moral Foundations Dictionary: freely available.
  • Regular expressions in Python or R can implement a project-specific dictionary. Save the patterns and test how they handle negation, spelling variants, and context.

Supervised classification

  • scikit-learn: logistic regression, SVM, random forests, cross-validation, and the metrics functions used in Chapter 6.
  • Hugging Face Transformers: pretrained models for fine-tuning when a bag-of-words baseline is not enough.
  • setfit: few-shot classification when labels are scarce, a common situation in social science.

Topic modeling

  • BERTopic (Python): modular embedding-based pipeline for grouping texts and describing their topics.
  • Top2Vec (Python): learns topic groupings from document embeddings; inspect representative texts before assigning substantive labels.
  • stm (R): models topic prevalence and content in relation to covariates.
  • gensim (Python): LDA and Word2Vec.

LLM-based coding

Access. OpenAI, Anthropic, Google AI Studio, OpenRouter (one interface across many models), Ollama (runs models locally or through cloud options; check the local-only settings when data must stay on your machine).

Structuring calls. instructor and Pydantic help validate response structure; instructor can retry when validation fails. A response can pass a schema check and still misclassify the text, so inspect its evidence and label as well. LangChain and LlamaIndex offer fuller pipeline frameworks; for many coding tasks, a plain loop and schema validator are easier to audit.

Batch costs. If your provider offers batch processing, compare its price, turnaround time, and data-handling terms with ordinary requests. Estimate cost from a small pilot before submitting the full corpus.

Agentic coding assistants (“vibe coding”)

Claude Code, OpenAI Codex, Cursor, GitHub Copilot, Google Antigravity, and similar assistants can help write analysis code from a natural-language description. Their execution and review features vary. Chapter 7 (§7.1.4) discusses the researcher’s responsibility for checking the resulting analysis.

Audit generated code before trusting it

Check that the script splits training and test data before fitting features, reports class-level metrics, records a random seed, and reads the stable corpus file (Chapter 6). Then spot-check predictions by hand: code can run while applying the wrong coding rule or leaking test information.

Validation and audit

  • sklearn.metrics.classification_report and confusion_matrix: class-level precision, recall, F1.
  • seaborn: error-pattern plots across subgroups.
  • irr (R) / sklearn.metrics.cohen_kappa_score: intercoder agreement for the human-agreement checks in Chapters 4 and 6.

Documentation

Jupyter, Quarto, and Git. Promptbooks and promptlogs are plain Markdown; templates are on the Chapter 2 page.