Skip to content

GdeltForge

Forges the raw GDELT 2.0 Events archive into clean, reproducibly-sampled Parquet.

GdeltForge is a lightweight but scalable data pipeline to extract, transform, and load the entire GDELT 2.0 Events Database. It's designed for research workflows that need:

  • Large-scale event data, from the 1979 historical backfill through today
  • Efficient columnar storage (Parquet), with optional Hive partitioning for historical data
  • Reproducible sampling (indexed, daily, filtered, stratified)
  • Transparent, modular data lineage: every stage is explicit, nothing runs "automagically"
git clone https://github.com/Vinicius-Teixeirac/GdeltForge.git
cd GdeltForge
uv sync
gdeltforge scrape --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge convert
gdeltforge filter
gdeltforge sample --mode indexed -n 10000

Note

GdeltForge isn't published to PyPI yet, so installation is from source for now (see Getting Started). A pip install gdeltforge release is planned.

About GDELT

GDELT (the Global Database of Events, Language, and Tone) monitors broadcast, print, and web news from nearly every country, in over 100 languages (translating 65 of them into English in realtime), processing it continuously with new records published every 15 minutes. It's one of the largest open datasets of global news activity available.

GDELT actually publishes several distinct tables, not just Events:

  • Events (what GdeltForge processes): structured, CAMEO-coded records of who-did-what-to-whom-where. Each row is a single event extracted from a news article: two actors, an action, a date, and a location.
  • Global Knowledge Graph (GKG): themes, emotions (2,300+ dimensions via GDELT's GCAM sentiment engine), people, organizations, and imagery/video, extracted from the same articles.
  • Mentions: every re-report of an event by a different outlet over time, not just the first.

GdeltForge is scoped to the Events table specifically, since it's the table best suited to the structured, tabular, seeded-sampling workflow GdeltForge is built around. GKG and Mentions support is on the roadmap but not built yet; see Comparison to Other Tools for what to reach for if you need them today.

Why this exists

The GDELT Events Database is extremely rich, but getting the full archive through official channels is genuinely difficult:

  • The GDELT API is built for small, recent queries: tight time windows, a ~250-row cap per query, and rate limits that make pulling the full historical dataset impractical.
  • The BigQuery mirror has full data, but free-tier quotas (1TB/month query, 10GB storage, 1GB egress) are far too small for the hundreds of GB involved, and full-table scans get expensive fast.
  • The raw bulk archives are available and complete, but they're thousands of individual ZIP files that need automated downloading, streaming/chunked processing, columnar storage, and memory-safe filtering and sampling before they're actually usable.

GdeltForge automates that last path end-to-end: scrape -> convert -> filter -> sample, each stage independent and re-runnable.

Where to go next