Working with text data

From an API response to an analysis-ready corpus

This primer supports the corpus preparation step in Chapter 6 (§6.1.1). It covers the vocabulary the exercises assume (JSON, tokenization, parquet), and it walks through the one genuinely fiddly step in the pipeline: reconstructing OpenAlex abstracts from their inverted index.

If you have only ever worked with rectangular survey data, this is the page that gets you from “the API returned something” to “I have a table I can code.”

JSON: nested data, not a spreadsheet

Almost every research API returns JSON. Where a spreadsheet is a grid, JSON is a set of nested labeled boxes: a record can contain a list of authors, each of which contains a list of affiliations, each with its own fields.

{
  "id": "https://openalex.org/W2741809807",
  "title": "Community-based strategies for college access",
  "publication_year": 2021,
  "authorships": [
    {"author": {"display_name": "A. Rivera"},
     "institutions": [{"display_name": "State University"}]},
    {"author": {"display_name": "B. Chen"}, "institutions": []}
  ]
}

In Python this becomes nested dictionaries and lists, and you reach into it by key:

record["title"]                                        # a string
record["authorships"][0]["author"]["display_name"]     # 'A. Rivera'
len(record["authorships"])                             # 2 authors

The practical task in Chapter 6 is flattening: deciding which nested fields become columns in a flat table, and what to do when a record has two authors and another has forty. That decision is a research decision (what is your unit of analysis?), not a technical one, which is why Chapter 3 treats it as data structure rather than plumbing.

Reading JSON without going cross-eyed

Paste a single record into any browser-based JSON viewer, or in a notebook use print(json.dumps(record, indent=2)). Never try to read raw minified JSON.

Reconstructing OpenAlex abstracts

OpenAlex does not ship abstracts as text. For legal reasons it ships an inverted index: a dictionary mapping each word to the positions where it occurs.

inverted_index = {
    "Community-based": [0], "strategies": [1], "reduce": [2],
    "barriers": [3, 11], "to": [4], "college": [5], "access": [6, 12],
    "for": [7], "first-generation": [8], "students": [9, 15],
    "despite": [10], "faced": [13], "by": [14],
}

To recover the abstract you invert the inversion: expand every (word, position) pair, sort by position, and join.

def reconstruct_abstract(inv):
    """Rebuild abstract text from an OpenAlex inverted_index. Returns None if absent."""
    if not inv:
        return None
    positions = [(pos, word) for word, ps in inv.items() for pos in ps]
    positions.sort()
    return " ".join(word for _, word in positions)

Running it on the example above returns:

'Community-based strategies reduce barriers to college access for
 first-generation students despite barriers access faced by students'

Three warnings from experience:

  • Handle the missing case. A large share of OpenAlex records have no abstract at all. if not inv: return None covers both None and {}. Count how many you lose and report it, because missingness is rarely random across venues or years (Chapter 3).
  • Do this once and save. Debugging a coding error against an inverted index is miserable. Reconstruct, save, and work from the saved file thereafter.
  • The text is not pristine. Punctuation attaches oddly and truncated abstracts are common. Read twenty of them before you trust the corpus.

Building the analysis table

The goal of Chapter 6 §6.1.1 is one stable, document-level table that every later exercise reads from.

import pandas as pd

df = pd.DataFrame({
    "doc_id": ["W1", "W2"],
    "text":   [reconstruct_abstract(inverted_index), "short"],
    "year":   [2021, 2019],
})

df["n_words"]   = df["text"].str.split().str.len()
df["too_short"] = df["n_words"] < 20        # flag, do not silently delete
df = df.drop_duplicates(subset="doc_id")

Flag rather than delete. A too_short column lets you exclude documents in one analysis and include them in a robustness check; deleting rows makes that choice invisible and irreversible (Chapter 9’s analytic transparency).

Saving: why parquet

df.to_parquet("coding_corpus.parquet")     # requires: pip install pyarrow
back = pd.read_parquet("coding_corpus.parquet")

CSV is fine and universal, but it stores everything as text, so a column of years can come back as strings and a list column comes back as a mangled string. Parquet preserves column types and is much smaller for text corpora. Use parquet for intermediate files you will reload; use CSV when a human or another program needs to read it.

Tokenization and normalization

Before dictionary matching or feature building, text is split into units and normalized. Chapter 6’s hands-on tip covers the choice between stemming and lemmatization; the terms themselves are simple:

  • Tokenization splits a string into words (or subword pieces): "college access" → ["college", "access"].
  • Stemming chops endings by rule: organizing, organization, organizational → all organ. Fast, sometimes too aggressive.
  • Lemmatization uses a dictionary to return real base forms: organizing → organize, ran → run. Slower, and usually the safer choice for social-science coding, because it keeps the process (organize) distinct from the entity (organization).
  • Stop words are high-frequency words carrying little discriminating information. Standard lists cover the, of, and; the useful ones are corpus-specific (in a nonprofit corpus, nonprofit is a stop word).
Preprocessing is a measurement decision

Every step here changes what your instrument can see. Lowercasing merges US (the country) with us (the pronoun). Aggressive stemming can merge two concepts your theory distinguishes. Record these choices in your codebook; they belong in the methods section, not in an undocumented cell at the top of a notebook.

Checklist before you start coding

  1. One row per document, with a stable doc_id that survives every later merge.
  2. Text reconstructed, missing abstracts counted and reported.
  3. Duplicates removed; short/empty documents flagged rather than dropped.
  4. Corpus boundaries (years, venues, language) written down as inclusion rules.
  5. Saved as a single file that every subsequent exercise reads from.
  6. A provenance note recording how the raw records became this table.

Where to go next