Working with text data

From an API response to an analysis-ready corpus

From an API response to an analysis-ready corpus: the JSON, tokenization, and parquet vocabulary the exercises assume, walked through end to end.

If you are used to rectangular survey data, an API response may look unfamiliar. This primer takes you from nested JSON records to a table you can code in Chapter 6 (§6.1.1), reconstructing OpenAlex abstracts and examining the choices that change your text.

JSON: nested data, not a spreadsheet

Many research APIs return JSON. Where a spreadsheet is a grid, JSON is a set of nested labeled boxes: a record can contain a list of authors, each of which contains a list of affiliations, each with its own fields.

{
  "id": "https://openalex.org/W_EXAMPLE",
  "title": "Community-based strategies for college access",
  "publication_year": 2021,
  "authorships": [
    {"author": {"display_name": "A. Rivera"},
     "institutions": [{"display_name": "State University"}]},
    {"author": {"display_name": "B. Chen"}, "institutions": []}
  ]
}

The record above is illustrative. In Python, JSON objects and arrays become dictionaries and lists, which you can inspect by key and position:

record["title"]                                        # a string
record["authorships"][0]["author"]["display_name"]     # 'A. Rivera'
len(record["authorships"])                             # 2 authors

The practical task in Chapter 6 is flattening: deciding which nested fields become columns in a flat table, and what to do when a record has two authors and another has forty. If one row represents an article, duplicating that row for every author changes what a row count means. Chapter 3 treats these choices as part of defining the unit of analysis.

Inspecting a JSON record

In a notebook, import json and use print(json.dumps(record, indent=2)) to indent the nested fields. Start with one record so you can see how authors, affiliations, and other lists fit together.

Reconstructing OpenAlex abstracts

OpenAlex work records represent available abstracts as an inverted index: a dictionary mapping each word to the positions where it occurs.

inverted_index = {
    "Community-based": [0], "strategies": [1], "reduce": [2],
    "barriers": [3, 11], "to": [4], "college": [5], "access": [6],
    "for": [7], "first-generation": [8], "students": [9],
    "despite": [10], "in": [12], "school": [13], "funding": [14],
}

To recover the abstract, expand every (word, position) pair, sort by position, and join.

def reconstruct_abstract(inv):
    """Rebuild abstract text from an OpenAlex inverted_index. Returns None if absent."""
    if not inv:
        return None
    positions = [(pos, word) for word, ps in inv.items() for pos in ps]
    positions.sort()
    return " ".join(word for _, word in positions)

Running it on the example above returns:

'Community-based strategies reduce barriers to college access for
 first-generation students despite barriers in school funding'

As you reconstruct the corpus, check three things:

  • Handle missing abstracts. if not inv: return None covers both None and {}. Count records without abstracts and compare their venues, languages, and years with those you retain. This helps you assess what an abstract-based analysis leaves out.
  • Save the reconstructed text. A coding error is easier to investigate when you can read the text that the instrument received. Keep it alongside the document identifier.
  • Inspect the text. Read a sample for truncation, missing words, or unexpected formatting. Compare doubtful records with their source before using them.

Building the analysis table

The goal of Chapter 6 §6.1.1 is one stable, document-level table that every later exercise reads from.

import pandas as pd

df = pd.DataFrame({
    "doc_id": ["W1", "W2"],
    "text":   [reconstruct_abstract(inverted_index), "short"],
    "year":   [2021, 2019],
})

df["n_words"]   = df["text"].str.split().str.len()
df["missing_text"] = df["text"].isna()
df["too_short"] = df["n_words"].fillna(0) < 20  # flag, do not silently delete
df = df.drop_duplicates(subset="doc_id")

Keep the flags with the table. You can then compare results with and without short records and report the exclusion rule. The twenty-word threshold here is illustrative; choose one suited to your documents and coding task.

Saving: why parquet

df.to_parquet("coding_corpus.parquet")     # requires: pip install pyarrow
back = pd.read_parquet("coding_corpus.parquet")

CSV stores values as text, so the program reading it must infer or be told each column’s type. Parquet stores column types and supports compression, which helps when you reload an analysis table. Check the types after saving and reading either format; use CSV when compatibility with another tool is the priority.

Tokenization and normalization

Preparing text for dictionary matching or feature building involves choices about which units and distinctions to keep:

  • Tokenization splits a string into words (or subword pieces): "college access" → ["college", "access"].
  • Stemming chops endings by rule: organizing, organization, organizational → all organ. Fast, sometimes too aggressive.
  • Lemmatization maps inflected words to base forms, often using vocabulary and part-of-speech information: organizing → organize, ran → run. It can preserve a distinction between a process (organize) and an entity (organization) that stemming removes. Check how it handles the distinctions your coding task needs.
  • Stop words are words selected for removal because they are expected to add little useful information to a particular representation. Standard lists cover the, of, and; the useful ones are corpus-specific (in a nonprofit corpus, consider whether nonprofit helps distinguish documents before removing it).
Preprocessing is a measurement decision

Every step here changes what your instrument can see. Lowercasing merges US (the country) with us (the pronoun). Aggressive stemming can merge two concepts your theory distinguishes. Record these choices in your codebook; they belong in the methods section, not in an undocumented cell at the top of a notebook.

Checklist before you start coding

  1. One row per document, with a stable doc_id that survives every later merge.
  2. Text reconstructed, missing abstracts counted and reported.
  3. Duplicates removed; short/empty documents flagged rather than dropped.
  4. Corpus boundaries (years, venues, language) written down as inclusion rules.
  5. Saved as a single file that every subsequent exercise reads from.
  6. A provenance note recording how the raw records became this table.

Where to go next