Tools: data management
Repositories, retrieval, cleaning, and documentation
The maintained version of the “Useful Tools” list closing Chapter 4, plus the data repository catalog the chapter refers to. Last reviewed: August 2026.
Data repositories and catalogs
| Repository | Scope | Notes |
|---|---|---|
| ICPSR | Social science | Deep archive with codebooks; institutional membership often required for some holdings |
| Dataverse | General, federated | Many university and journal instances; Harvard Dataverse is the largest |
| OSF | General | Preregistration, storage, and project pages in one place |
| Zenodo | General, CERN-hosted | Issues DOIs; good default for archiving your own data |
| re3data | Registry of repositories | Use it to find the right domain repository |
| Google Dataset Search | Index | Broad but uneven; verify provenance at the source |
| Data Commons | Harmonized public statistics | Useful for demographic and economic covariates |
| Harvard Dataverse | General | Where much replication data for political science lands |
| UK Data Service | UK social/economic | Registration required; strong survey holdings |
| GESIS | European social science | ALLBUS, Eurobarometer, ISSP |
Bibliographic and scholarly data
- OpenAlex: free, no key, ~250M works. The book’s running example; see Chapter 4 resources.
- Crossref: DOI registration metadata, authoritative for publication records.
- Semantic Scholar API: includes abstracts and citation contexts.
- Zotero: reference management; the author’s
zotero-files2mdconverts a Zotero library’s attachments into Markdown for use as an AI knowledge base (Chapter 1, Chapter 2).
Retrieval and cleaning
- requests / httpx: HTTP clients for API work.
- pandas: the workhorse for tabular data in Python; polars is a faster alternative for large tables.
- OpenRefine: interactive cleaning with a replayable operation history, which is what makes it defensible rather than merely convenient.
- pyarrow: parquet support; see Working with text data.
Documentation and structure
- ERD tools: dbdiagram.io, draw.io, MySQL Workbench.
- Version control: Git with GitHub or institutional GitLab.
- Archiving with a DOI: Zenodo, OSF.
Suggesting a change
Found a dead link or a repository that belongs here? Corrections go through the errata page.