Tools: data management
Repositories, retrieval, cleaning, and documentation
The maintained version of Chapter 4’s tool list: repositories, retrieval, cleaning, and documentation, plus the data repository catalog.
Use these tools to find a source, retrieve its records, and document how you prepared them. The list accompanies Chapter 4. Tool selection reviewed August 2026; OpenAlex access notes updated September 2026.
Data repositories and catalogs
| Repository | Scope | Notes |
|---|---|---|
| ICPSR | Social science | Social-science data and codebooks; access conditions vary by collection |
| Dataverse | General, federated | University and journal repositories using a shared platform |
| OSF | General | Preregistration, storage, and project pages in one place |
| Zenodo | General, CERN-hosted | Archives research outputs and issues DOIs |
| re3data | Registry of repositories | Use it to find the right domain repository |
| Google Dataset Search | Index | Broad but uneven; verify provenance at the source |
| Data Commons | Harmonized public statistics | Useful for demographic and economic covariates |
| Harvard Dataverse | General | Research data, including replication materials |
| UK Data Service | UK social/economic | Registration required; strong survey holdings |
| GESIS | European social science | ALLBUS, Eurobarometer, ISSP |
Bibliographic and scholarly data
- OpenAlex: scholarly records and citation links. The book’s running example uses a free API key; see Chapter 4 resources for access and retrieval.
- Crossref: DOI registration metadata supplied by publishers and other members.
- Semantic Scholar API: scholarly records, including abstracts and citation information where available.
- Zotero: reference management; the author’s
zotero-files2mdconverts a Zotero library’s attachments into Markdown for use as an AI knowledge base (Chapter 1, Chapter 2).
Retrieval and cleaning
- requests / httpx: HTTP clients for API work.
- pandas: tabular data preparation in Python; polars provides another dataframe interface.
- OpenRefine: interactive cleaning with an operation history you can save and replay. Record why you made each consequential change as well.
- pyarrow: parquet support; see Working with text data.
Documentation and structure
- ERD tools: dbdiagram.io, draw.io, MySQL Workbench.
- Version control: Git with GitHub or institutional GitLab.
- Archiving with a DOI: Zenodo, OSF.
Suggesting a change
Found a dead link or a repository that belongs here? Corrections go through the errata page.