Prompting and its failure modes

Writing a coding prompt, and finding out how it is wrong

This primer supports the LLM coding exercise in Chapter 6 (§6.1.3) and the screening exercise in Chapter 4 (§4.3.1). It covers what belongs in a coding prompt, gives you templates you can adapt, and then does the part most guides skip: it shows how a reasonable-looking prompt produces biased labels, and how to catch that.

The core idea: a prompt is a codebook

When you hand a coding task to a trained human assistant, you give them a codebook: the construct definition, the decision rules, worked examples, and instructions for ambiguous cases. A coding prompt is the same document, written for a machine coder. Chapter 6 calls the collection of these a promptbook.

This framing does real work. It tells you what belongs in the prompt (everything you would tell a human), how to evaluate it (agreement against human coding, not vibes), and why it must be archived with your materials (it is your measurement instrument).

Anatomy of a coding prompt

A complete prompt has six parts. Omitting any one of them is the most common cause of the failures in the next section.

[1. ROLE AND TASK]
You are coding abstracts from academic articles on education policy.
Assign exactly one label to the abstract provided.

[2. CONSTRUCT DEFINITION]
"Equity-oriented" means the abstract frames its problem in terms of unequal
outcomes, access, or treatment across social groups, OR proposes remedies aimed
at reducing such inequality.
"Efficiency-oriented" means the abstract frames its problem in terms of cost,
throughput, optimization, or resource allocation, without reference to
distribution across groups.

[3. DECISION RULES]
- If both framings are present, choose the one that motivates the study's
  stated purpose.
- The word "access" alone is NOT sufficient evidence of equity framing;
  "data access" and "access to computing resources" are not about social groups.
- Judge only the text provided. Do not use knowledge about the authors,
  the journal, or the field.

[4. OUTPUT SCHEMA]
Return JSON only:
{"label": "equity" | "efficiency" | "insufficient_evidence",
 "evidence": "<verbatim quote from the abstract, max 25 words>",
 "confidence": "high" | "low"}

[5. WORKED EXAMPLES]
Abstract: "We evaluate a scalable optimization framework for improving student
throughput in admissions pipelines."
-> {"label": "efficiency", "evidence": "scalable optimization framework for
improving student throughput", "confidence": "high"}

Abstract: "This study examines community-based strategies for reducing barriers
to college access among first-generation students."
-> {"label": "equity", "evidence": "reducing barriers to college access among
first-generation students", "confidence": "high"}

[6. GUARDRAILS]
If the abstract does not contain enough information to decide, return
"insufficient_evidence". Do not guess. The "evidence" field must be a verbatim
substring of the abstract; never paraphrase or invent it.

Two of these parts do most of the protective work:

  • The evidence field. Requiring a verbatim quote makes fabricated reasoning visible. If the model inferred a label from background knowledge rather than the text, the evidence field will be empty, paraphrased, or not actually present in the abstract, and you can check that automatically.
  • The insufficient_evidence option. Without an explicit way to abstain, a model will guess. Every guess enters your dataset looking exactly like a real measurement.

How a reasonable prompt goes wrong

Chapter 6’s exercise asks you to write a deliberately flawed prompt, observe the damage, and repair it. Here are the four failure modes worth inducing on purpose, because each produces a distinct and recognizable signature in your labels.

1. Presupposition

❌ “Identify how this abstract promotes equity.”

The instruction presupposes that it does. Models are strongly disposed to comply with the framing they are given, so this prompt finds equity framing nearly everywhere. Signature: the minority class balloons; the evidence quotes become strained.

✅ “Determine whether this abstract is equity-oriented, efficiency-oriented, or neither, according to the definitions above.”

2. Asymmetric description

❌ A prompt where the equity definition runs 80 words with three examples, and the efficiency definition runs 12 words with none.

Detail is a thumb on the scale: the richly specified category is easier to match. Signature: labels drift toward whichever class you described more fully. This one is insidious because the prompt looks careful, and it is the most common accidental version.

✅ Give each category comparable definitional depth and an equal number of examples.

3. No abstention option

❌ Removing insufficient_evidence from the schema.

Signature: zero missing labels, which looks like clean data but is actually a suppressed uncertainty distribution. Truly ambiguous documents get sorted, usually toward the majority class.

4. Inviting outside knowledge

❌ Omitting “judge only the text provided.”

Stuhler and colleagues document a vivid case: a model asked to extract religion from obituaries repeatedly inferred it from names and nationalities, even when instructed not to. Signature: high apparent accuracy in aggregate, with errors concentrated in exactly the subgroups where the stereotype is wrong (Chapter 6, §6.1.3 validity note).

The stress-test protocol

This is the workflow for Chapter 6, Exercise 6.1.3, steps 3–4:

  1. Run your careful prompt on a subsample of 200–300 documents. Record the label distribution.
  2. Run a deliberately flawed variant on the same documents. Change one thing (add a presupposition, or drop the abstention option), not several.
  3. Compare distributions. How far did the equity share move? Which documents flipped?
  4. Read ten flipped documents. This is where the learning happens: you see the specific wording that pushed the model.
  5. Repair and re-run. Confirm the fix moves the distribution back.
  6. Keep the biased labels. Chapter 6’s Exercise 6.1.4 (step 6) has you train a classifier on them, to see that label bias does not average out downstream. It propagates.
Cost control

Develop against 20 documents, stress-test against 200, and only then run the full corpus. Most of a prompting budget gets wasted running unstable prompts at scale.

Checking agreement, not accuracy

Once the prompt is stable, hand-code a random sample yourself before looking at the model’s labels for those documents, then compare.

  • Agreement above roughly 80% on a clearly defined construct is a reasonable working threshold, but the number matters less than the pattern of disagreement.
  • Read every disagreement. Three outcomes: the model was wrong (revise the prompt), you were wrong (revise your understanding), or the case is genuinely ambiguous (revise the definition, and note that the construct has fuzzy edges).
  • Check whether errors cluster by period, venue, or group. Non-random error biases any estimate that conditions on the affected variable (Chapter 5, §5.7).
Agreement is not validity for open-ended tasks

For closed classification with a real right answer, agreement measures accuracy. For open-ended work (discovering themes, judging a whole document), competent human coders legitimately disagree, so there is no single key to score against, and high consistency across prompts or models may reflect shared training bias rather than correctness. Chapter 7 (§7.4) takes this up: when the task is interpretive, validation shifts toward structured expert judgment of output quality.

A second template: relevance screening

For Chapter 4’s screening exercise, the same anatomy applies with a narrower task:

You are screening search results for a systematic literature review.

INCLUSION: empirical studies of civic participation in which participation is
measured at the individual level.
EXCLUSION: theoretical/conceptual pieces with no data; studies where
"participation" refers to patient participation in clinical care or user
participation in sensing systems; non-English abstracts.

Judge only the title and abstract provided.

Return JSON only:
{"decision": "include" | "exclude" | "uncertain",
 "reason": "<one clause, citing the criterion applied>",
 "evidence": "<verbatim phrase from the abstract>"}

If the abstract is too vague to apply the criteria, return "uncertain".
Do not guess.

Note the explicit exclusions naming the homonyms that a keyword query sweeps in. That is usually where the screening gains come from.

Recording what you did

The promptbook entry that travels with your project (Chapter 2’s kb/codebooks/) should contain: the prompt text verbatim, the output schema, the model name and version, the settings (temperature and any others), the date of the run, the agreement rate against your hand-coded sample, and known failure modes from the audit. Revisions go in the promptlog (memory/PROMPTLOG.md), not on top of the promptbook: the promptbook is the current instrument, the promptlog is its history.

Chapter 9 (§9.3.3) adds one more requirement. Because you cannot pin a commercial model the way you pin a library version, archive the prompt–output pairs themselves. Cached outputs are the reproducibility boundary for everything downstream.

Where to go next