About

Background

GdeltForge was written for research on representation learning for structured events. That work compares tabular, graph and text representations of GDELT events with a multi-domain geometric model, and it needed samples of GDELT's whole Events archive that could be described in a methods section and regenerated by anyone reading it. Neither GDELT's APIs nor BigQuery's free tier could provide that, so GdeltForge works from the raw bulk archive instead, on a single machine.

Its first use was the chapter "Multi-domain Geometric Representation Learning for Structured Events" (Freitas, Sakiyama and Marcacini, BRACIS 2026, Springer LNAI 17107, doi:10.1007/978-3-032-39895-6_13).

What the research needed, and the decisions that followed

GDELT as published, kept apart from any correction. The research treats the archive as GDELT published it as the primary data, and a corrected version as a robustness check. So convert only changes the format, and every correction lives in clean, which writes to its own directory, keeps GDELT's own value beside each repair (the *_original columns), and marks every file it writes. --source picks the copy a sample or join reads: cleaned (the default) or converted, the archive as published. sample, aggregate and crossref warn when a directory read as converted holds cleaned files, so a cleaned file can't pass for GDELT's own data. Data Cleaning describes each correction and why it is made.

Sample designs you can state. Each sampling mode implements one classical design (simple random sampling, or stratified sampling with equal allocation), seeded, so the same command over the same files draws the same rows. A calendar or stratified sample writes each stratum's true size beside it (<out>.strata.json), the N_h needed to reweight the sample toward population estimates. Sampling Theory gives the algorithm and source behind each mode.

Periods sampled across files. The first daily sampler capped its quota per file and then concatenated the files, so wherever a period spans several files (the yearly and monthly historical Events files, or GKG 2.1 and Mentions with about 96 files a day) it drew a multiple of the quota. --mode calendar replaced it: it samples each period's rows across every file that holds them.

Every column, at full precision. The pipeline keeps every column by default and stores coordinates and tone as Float64: downstream place matching relies on exact coordinate values, and narrowing to Float32 changes them (see Configuration).

The Events to GKG join, many-to-many. One event is reported by many articles, and one article reports many events. crossref keeps every combination, so an analysis can decide how to aggregate them. Crossref Join Semantics covers the details.

The whole archive, on one machine. Every stage streams or reads in bounded batches, so the full archive can be processed on a workstation, without a cluster or a warehouse.

Author

Vinícius Teixeira Carvalho Freitas, Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo, São Carlos, Brazil.

Citing

If GdeltForge contributes to published work, please cite it. The repository's CITATION.cff holds the details (GitHub shows them as "Cite this repository"); in BibTeX:

@software{freitas_gdeltforge,
  author  = {Freitas, Vinícius Teixeira Carvalho},
  title   = {GdeltForge: reproducible sampling and cross-referencing of the full GDELT archive},
  url     = {https://github.com/Vinicius-Teixeirac/GdeltForge},
  license = {Apache-2.0},
  year    = {2026}
}

Please also cite the GDELT Project, as its terms of use require, with a link to gdeltproject.org. Its usual reference is: Kalev Leetaru and Philip A. Schrodt, "GDELT: Global Data on Events, Location and Tone, 1979–2012", paper presented at the International Studies Association Annual Convention, 2013.

GdeltForge is an independent, unofficial tool, not affiliated with, endorsed by, or sponsored by the GDELT Project.