Chapter 4 — Gathering literature

Retrieving a personal corpus from OpenAlex

Chapter 4 builds the dataset that every later exercise uses: a literature collection from your own field, retrieved from OpenAlex. Everything the chapter needs is on this page.

What you will produce. One documented corpus from your own field, with a codebook recording every field you kept and every screening decision you made. Chapters 6, 8, and 10 all read from it, so it is worth building carefully once.

Materials for this chapter

Technical walkthrough

The OpenAlex retrieval, paging, filtering, and flattening code, below on this page.

Working with text data

Reconstruct abstracts from inverted indices and build the analysis table. Take this first if JSON and parquet are unfamiliar.

Data management tools

The maintained repository list, plus cleaning and versioning tools.

Studies on data management

Ninety-one published studies annotated for how they sourced and structured their data, which gives you models for the boundary argument you have to make.

The design-stage thinking behind this exercise is in Chapter 3, which is also where the studies database is filtered to data-management practice.

Why OpenAlex

It is free, requires no API key, covers roughly 250 million scholarly works, and returns structured metadata (authors, institutions, venues, citations, topic tags). Its coverage is not uniform, which is the point of Chapter 4’s data-source evaluation: check what it misses in your field before you build on it.

Polite pool

Include your email in requests. It costs nothing, routes you to a faster and more reliable service tier, and lets OpenAlex contact you if a query is causing problems.

import requests

BASE = "https://api.openalex.org/works"
MAILTO = "you@university.edu"          # use your real address

A first query

params = {
    "search": "civic participation",
    "filter": "from_publication_date:2015-01-01,type:article,language:en",
    "per-page": 25,
    "mailto": MAILTO,
}
r = requests.get(BASE, params=params, timeout=30)
r.raise_for_status()
data = r.json()

print(data["meta"]["count"], "works match")
print(data["results"][0]["title"])

meta.count is the total number of matches, not the number returned. Check it before paging: a query returning 400,000 works needs narrowing, not patience.

Paging through results

For anything beyond one page, use cursor paging. Offset paging breaks past 10,000 records.

import time

def fetch_works(search, filter_str, mailto, max_records=2000, per_page=200):
    """Page through OpenAlex results with a cursor. Returns a list of raw records."""
    out, cursor = [], "*"
    while cursor and len(out) < max_records:
        params = {"search": search, "filter": filter_str, "per-page": per_page,
                  "cursor": cursor, "mailto": mailto}
        r = requests.get(BASE, params=params, timeout=60)
        r.raise_for_status()
        payload = r.json()
        out.extend(payload["results"])
        cursor = payload["meta"].get("next_cursor")
        time.sleep(0.2)               # be a good citizen
    return out[:max_records]

Two habits worth building now, both from Chapter 3:

  • Save the raw response before touching it. Write the untouched JSON to data/raw/ and never modify it. Everything downstream is rebuilt from that file, so a preprocessing mistake costs a rerun of your script rather than a re-download.
  • Record the query. The search string, filters, date of retrieval, and record count are part of your method. OpenAlex is updated continuously, so the same query run next month returns a different corpus.
import json, datetime, pathlib

records = fetch_works("civic participation",
                      "from_publication_date:2015-01-01,type:article,language:en",
                      MAILTO)

pathlib.Path("data/raw").mkdir(parents=True, exist_ok=True)
stamp = datetime.date.today().isoformat()
with open(f"data/raw/openalex_{stamp}.json", "w") as f:
    json.dump({"query": {"search": "civic participation",
                         "filter": "from_publication_date:2015-01-01,type:article,language:en",
                         "retrieved": stamp, "n": len(records)},
               "results": records}, f)

Useful filters

Goal Filter
Journal articles only type:article
Date range from_publication_date:2015-01-01,to_publication_date:2024-12-31
Has an abstract has_abstract:true
Open access only is_oa:true
By topic/concept concepts.id:C17744445 (look up the concept ID first)
By venue primary_location.source.id:S137773608
Minimum citations cited_by_count:>10

Filters combine with commas (AND). The full grammar is in the OpenAlex documentation.

Search versus filter

search does relevance ranking over text and is fuzzy; filter is exact. A corpus built purely on search includes a long tail of weak matches, which is precisely why Chapter 4 adds an LLM screening pass with a human agreement check (Exercise 4.3.1). See Prompting and its failure modes for the screening prompt template.

Flattening to a table

def flatten(rec):
    return {
        "doc_id": rec["id"].rsplit("/", 1)[-1],
        "title": rec.get("title"),
        "year": rec.get("publication_year"),
        "venue": (rec.get("primary_location") or {}).get("source", {}).get("display_name"),
        "n_authors": len(rec.get("authorships", [])),
        "cited_by": rec.get("cited_by_count"),
        "doi": rec.get("doi"),
        "inverted_index": rec.get("abstract_inverted_index"),
    }

import pandas as pd
df = pd.DataFrame([flatten(r) for r in records])

Abstracts arrive as an inverted index and must be reconstructed. That step, plus saving a stable analysis file, is covered in Working with text data and is where Chapter 6 picks up.

Checklist for the chapter’s deliverable

  1. Data-source review written (raw, preprocessing, content, retrieval levels).
  2. Query recorded with filters, date, and resulting count.
  3. Raw JSON archived unmodified in data/raw/.
  4. Flattened table with a stable doc_id.
  5. Screening pass documented with its agreement rate (Exercise 4.3.1).
  6. ERD and codebook drafted for the tables you keep.

Where to go next