Global event data,
forged.
Raw GDELT in. Clean, reproducibly-sampled, cross-referenced Parquet out. One CLI, one machine, the whole archive.
The pipeline Five stages. Each one a separate command, re-runnable in isolation.
scrape
Checksum-verified, concurrent download of the raw archive.
→ CSV
convert
CSV → Parquet, with optional Hive partitioning for historical data.
→ Parquet
filter
Drop rows missing your configured columns.
→ Cleaned
sample
Seeded reservoir sampling in a single streaming pass.
→ Sample
crossref
Join a sampled Events output back onto GKG.
→ Sample + GKGWhy it exists The full archive is public. Getting it is the hard part.
The GDELT API
Built for small, recent queries: tight time windows, a ~250-row cap per query, and rate limits. The full history is out of reach.
The BigQuery mirror
Complete, but free-tier quotas (1TB query, 10GB storage, 1GB egress) don't cover the hundreds of GB involved, and full scans get expensive.
The raw bulk archive
Complete and free, but thousands of individual files with no tooling. That's the path GdeltForge automates, end to end, locally.
What you get
The whole archive
Events back to 1979, GKG 1.0 since 2013, GKG 2.1 and Mentions since 2015 — the full history, not the last three months the API allows.
Columnar storage
Parquet throughout, with optional Hive partitioning for yearly and monthly historical dumps.
Transparent lineage
Every stage explicit and independently testable. Nothing runs automagically.
Seven datasets
Events (daily, native 15-minute granularity, and the 1979-2013 historical dump), GKG 2.1, legacy GKG 1.0 (plus its separate Counts file), and Mentions, each through the same stages.
Bundled CAMEO codes
gdeltforge codes looks up valid values across seven column families, offline.
Honest scope
No orchestration, no VGKG, no hosted infrastructure. The comparison page says when to use something else.
Sampling modes Reproducible sampling is a first-class stage, not an afterthought.
Why it matters
Every mode is seeded, so the same command reproduces the same sample against a fixed file layout; only indexed mode is guaranteed to stay reproducible if the archive later gets re-chunked into a different number of files (see Limitations). Reproducible isn't the same guarantee as representative: indexed is a true random sample of the archive, while calendar and stratified filtered trade population-level representativeness for even coverage on purpose (see Representativeness). All modes stream over an archive far larger than RAM in a single pass, on one machine, with no cluster or warehouse. See Filtered Sampling for the full syntax, or Sampling Theory for which classical algorithm each mode actually runs.
Crossref Events enriched with GKG, without collapsing the join.
When to reach for what¶
| Need | Reach for |
|---|---|
| Reproducible, seeded, class-balanced samples of the full Events archive, offline | GdeltForge |
| Events enriched with GKG, many-to-many preserved | GdeltForge (crossref) |
| Recent article/tone queries over a small time window | A DOC 2.0 API client |
| Whole-archive SQL analytics, nothing to install | BigQuery's public dataset |
| An existing Spark or DuckDB pipeline | Stay on it |
See Comparison to Other Tools for the full breakdown.
Quickstart¶
One week of data, end to end. No config file needed: GdeltForge writes a conservative default on first run.
pip install gdeltforge
gdeltforge scrape --dataset events --start-date 2024-01-01 --end-date 2024-01-07
gdeltforge convert --dataset events
gdeltforge filter --dataset events
gdeltforge sample --dataset events --mode indexed -n 1000 --out sample.parquet
Where to go next
Getting Started →
Install it and run your first pipeline.
CLI Reference →
Every command, every flag, with real examples.
Recipes →
Runnable, end-to-end workflows.
Configuration →
The full settings.yaml reference.
Architecture →
How the pipeline is put together, and why.
Limitations & Roadmap →
What's out of scope, and what's next.
GdeltForge processes data published by the GDELT Project, which makes it available for unlimited and unrestricted use, provided any use or redistribution includes a citation and a link to their site. GdeltForge itself is an independent, unofficial tool, not affiliated with, endorsed by, or sponsored by the GDELT Project.