Chapter 4 — Gathering literature

Retrieving a personal corpus from OpenAlex

Materials for building the literature dataset every later exercise uses: a corpus from your own field, retrieved from OpenAlex, screened and documented.

Use the OpenAlex examples below to retrieve literature from your field, inspect its coverage, and record the decisions that define your corpus.

What you will produce. One documented corpus from your own field, with a codebook recording every field you kept and every screening decision you made. Chapters 6, 8, and 10 all use it, so clear identifiers and documented boundaries will help you compare results.

Materials for this chapter

Technical walkthrough

The OpenAlex retrieval, paging, filtering, and flattening code, below on this page.

Working with text data

Reconstruct abstracts from inverted indices and build the analysis table. Take this first if JSON and parquet are unfamiliar.

Data management tools

The maintained repository list, plus cleaning and versioning tools.

Studies on data management

Ninety-one published studies annotated for how they sourced and structured their data, which gives you models for the boundary argument you have to make.

Chapter 3 develops the data-management reasoning behind this exercise.

Why OpenAlex

OpenAlex returns structured records for scholarly works, including authors, institutions, publication venues, references, and topic tags. Check a sample against literature you know: which journals, languages, years, or kinds of work are missing from your field? That review helps you define what a corpus built from these records can represent.

API access and limits

For corpus retrieval, create a free OpenAlex API key. The access documentation describes the free usage allowance and rate limits; small queries also work without a key. Store your key in OPENALEX_API_KEY, or enter it privately when the example prompts you. Access guidance checked September 2026.

import os
from getpass import getpass
import requests

BASE = "https://api.openalex.org/works"
API_KEY = os.environ.get("OPENALEX_API_KEY") or getpass("OpenAlex API key: ")
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

A first query

params = {
    "search": "civic participation",
    "filter": "from_publication_date:2015-01-01,type:article,language:en",
    "per_page": 25,
}
r = requests.get(BASE, params=params, headers=HEADERS, timeout=30)
r.raise_for_status()
data = r.json()

print(data["meta"]["count"], "works match")
if data["results"]:
    print(data["results"][0]["title"])

meta.count gives the total matches; this response contains only the requested page. Inspect some records before continuing. If many fall outside your question, refine the search and filters before spending your retrieval allowance on more pages.

Paging through results

Use cursor paging to retrieve successive pages. The example stops at 2,000 records and requests at most 100 per page.

import time

def fetch_works(search, filter_str, max_records=2000, per_page=100):
    """Page through OpenAlex results with a cursor. Returns a list of raw records."""
    if not 1 <= per_page <= 100 or max_records < 0:
        raise ValueError("Use 1–100 records per page and a non-negative limit.")
    out, cursor = [], "*"
    while cursor and len(out) < max_records:
        params = {"search": search, "filter": filter_str, "per_page": per_page,
                  "cursor": cursor}
        r = requests.get(BASE, params=params, headers=HEADERS, timeout=60)
        r.raise_for_status()
        payload = r.json()
        if not payload["results"]:
            break
        out.extend(payload["results"])
        cursor = payload["meta"].get("next_cursor")
        time.sleep(0.2)               # space requests apart
    return out[:max_records]

If OpenAlex returns 429, stop and check your remaining allowance and request rate. The code stops on HTTP errors so that an interrupted retrieval is not mistaken for a complete corpus.

Two habits from Chapter 3 help preserve the retrieval:

  • Save the raw response before touching it. Write the untouched JSON to data/raw/ and never modify it. Everything downstream is rebuilt from that file, so a preprocessing mistake costs a rerun of your script rather than a re-download.
  • Record the query. The search string, filters, date of retrieval, and record count are part of your method. OpenAlex is updated continuously, so the same query run next month may return a different corpus.
import json, datetime, pathlib

records = fetch_works("civic participation",
                      "from_publication_date:2015-01-01,type:article,language:en")

pathlib.Path("data/raw").mkdir(parents=True, exist_ok=True)
stamp = datetime.date.today().isoformat()
with open(f"data/raw/openalex_{stamp}.json", "w") as f:
    json.dump({"query": {"search": "civic participation",
                         "filter": "from_publication_date:2015-01-01,type:article,language:en",
                         "retrieved": stamp, "n": len(records)},
               "results": records}, f)

Useful filters

Goal Filter
Journal articles only type:article
Date range from_publication_date:2015-01-01,to_publication_date:2024-12-31
Has an abstract has_abstract:true
Open access only is_oa:true
By topic topics.id:T12345 (replace with the topic ID you have checked)
By venue primary_location.source.id:S137773608
Minimum citations cited_by_count:>10

Filters combine with commas (AND). The full grammar is in the OpenAlex documentation.

Search versus filter

search finds text matches and ranks their relevance; filter constrains fields such as publication year or language. Neither establishes whether a study meets your inclusion criteria. Inspect retrieved records and document the screening decision. Chapter 4’s LLM-assisted screening exercise includes a human agreement check (Exercise 4.3.1). See Prompting and its failure modes for the screening prompt template.

Flattening to a table

def flatten(rec):
    return {
        "doc_id": rec["id"].rsplit("/", 1)[-1],
        "title": rec.get("title"),
        "year": rec.get("publication_year"),
        "venue": ((rec.get("primary_location") or {}).get("source") or {}).get("display_name"),
        "n_authors": len(rec.get("authorships", [])),
        "cited_by": rec.get("cited_by_count"),
        "doi": rec.get("doi"),
        "inverted_index": rec.get("abstract_inverted_index"),
    }

import pandas as pd
df = pd.DataFrame([flatten(r) for r in records])

Abstracts arrive as an inverted index and must be reconstructed. That step, plus saving a stable analysis file, is covered in Working with text data and is where Chapter 6 picks up.

Checklist for the chapter’s deliverable

  1. Data-source review written (raw, preprocessing, content, retrieval levels).
  2. Query recorded with filters, date, and resulting count.
  3. Raw JSON archived unmodified in data/raw/.
  4. Flattened table with a stable doc_id.
  5. Screening pass documented with its agreement rate (Exercise 4.3.1).
  6. ERD and codebook drafted for the tables you keep.

Where to go next