Configuration

GdeltForge reads a single YAML file, resolved in this order:

1. --config PATH passed to the CLI
2. the GDELTFORGE_CONFIG environment variable
3. ./config/settings.yaml, relative to the current working directory (the default)
4. GdeltForge's own built-in default, bundled inside the installed package, but only when neither 1 nor 2 was given at all and nothing exists at 3 either

For a real, ongoing project, start from the template:

cp config/settings.example.yaml config/settings.yaml

and customize it, especially paths and filter.columns_to_check.

Tier 4: the built-in default

You don't have to do that first, though. If none of 1-3 resolve to a real file, GdeltForge falls back to a config bundled inside the package itself rather than failing outright, and writes it out to ./config/settings.yaml so it becomes a normal, editable file for the rest of that session (logged clearly when this happens; the write is best-effort, so a read-only working directory still gets a working, in-memory-only config rather than an error).

This exists for exactly the case where copying a template first isn't practical: a pip install gdeltforge (in a fresh Google Colab session, for instance) drops nothing into the working directory the way a git clone's config/settings.example.yaml does, and an ephemeral environment that wipes its filesystem on every session reset means you'd otherwise be reconstructing that file by hand every single time. --config/GDELTFORGE_CONFIG still work exactly as before for anything you want to survive past the current session, e.g. a file saved on a mounted Google Drive.

It is deliberately a different, more conservative file than settings.example.yaml, not the same content with the paths changed: every row and every column survive by default (filter.columns_to_check is present but empty for every dataset, and there's no output_columns/float32_columns pruning anywhere), so a first run's output is never silently shaped by choices you didn't make. settings.example.yaml remains the place to look for the storage/row-filtering wins (GKG column pruning, zstd compression, geocoding-required filtering) that are worth opting into deliberately once you know your data's shape; see the source of src/gdeltforge/config/default_settings.yaml for its own paths (real ./data/... locations, not settings.example.yaml's ./path_example/... placeholders) and every other default it sets.

An explicit --config/GDELTFORGE_CONFIG pointing at a path that turns out to be missing still raises FileNotFoundError rather than silently falling back: that's almost always a typo, not a request to use the built-in default instead.

Datasets and --dataset

convert, filter, sample, and scrape all accept --dataset {events,events-15min,events-reduced,gkg-v1,gkg-v1-counts,gkg-v2,mentions}, and require it: there is no default. This selects which set of columns/columns_numeric/filter.columns_to_check/paths.* keys a command reads; see below for exactly how each section is namespaced per dataset.

--dataset	Config key	Status
events	gdelt_event	Full support
events-15min	gdelt_event_15min	Full support (Events at native GDELT 2.0 granularity; genuinely a different, 61-column schema from events' 58, not just finer granularity, see below)
events-reduced	gdelt_event_reduced	Full support (a single static 1979-2013 historical dump, pre-aggregated and structurally unlike every other dataset here, see below)
gkg-v2	gdelt_gkg_v2	Full support (GKG 2.1, the current format, live since Feb 2015)
mentions	gdelt_mentions	Full support (the bridge table between Events and GKG; see Comparison for why)
gkg-v1	gdelt_gkg_v1	Full support (legacy format, April 2013 through February 2015 as the primary feed, still published daily since)
gkg-v1-counts	gdelt_gkg_v1_counts	Full support (GKG 1.0's separate, narrower "Counts" file, one row per count mention rather than per document)

events and events-15min cover the same underlying events at different granularity and schema richness, not the same file at two speeds: events is the daily/monthly/yearly archive, still served in the older, GDELT-1.0-compatible 58-column format for backward compatibility; events-15min is discovered from the same 15-minute master file list gkg-v2/mentions use, in native GDELT 2.0's 61-column format, with an ADM2Code field added to each of the three geo blocks (Actor1Geo/Actor2Geo/ActionGeo) that the daily format doesn't carry. They deliberately get separate paths.* directories (see below) so pulling both for an overlapping date range can never double-count the same events into one pipeline's output.

events-reduced is GDELT.MASTERREDUCEDV2.1979-2013.zip, served from the same /events/ listing events scrapes but never fetched by events' own scrape: opening it directly showed it's a genuinely different product, not a narrower copy of what events already covers. It's a single static file (no per-day/month/year archive, so --start-date/--end-date don't apply to scrape --dataset events-reduced, see CLI Reference), roughly 1.08 GB zipped and 6.58 GB / ~87.3M rows uncompressed, collapsed on DATE+ACTOR1+ACTOR2+EVENTCODE rather than one row per event: its 17 columns (Date, Source, Target, CAMEOCode, NumEvents, NumArts, QuadClass, Goldstein, and the Source/Target/Action geo triples) carry no GlobalEventID, SOURCEURL, or DATEADDED at all. That makes crossref categorically impossible against it (there's no per-event or per-article identity left to bridge through Mentions/GKG), and rules out narrowing the scrape itself by date; row-level narrowing after convert still works normally through its own Date column, e.g. --filter '{"Date": {"op": "between", "min": 19900101, "max": 19991231}}'. Because the raw file is too large to read into memory whole, convert processes it in chunks and always writes it Hive-partitioned by Year (computed from Date, since the filename carries none), regardless of converter.partitioning.enabled, a toggle that otherwise only ever governs Events' own opt-in yearly/monthly split; see below.

columns

columns.<dataset> lists every column in that dataset's schema, in file order. It's used to name the otherwise-headerless columns when reading the raw CSVs, so it should match the official field list for that dataset unless you know you're working with a modified schema: Events (the daily archive's 58-column schema; events-15min's 61-column schema was instead verified directly against a real downloaded file, byte for byte, not this reference), GKG 2.1, GKG 1.0 (covers both gkg-v1 and gkg-v1-counts).

columns_numeric.<dataset> lists which of those columns should be coerced to numeric types (via polars' cast(strict=False), to Int64 or Float64 depending on the column; a value that doesn't fit the target type becomes null) rather than kept as strings.

GKG's own repeated/structured sub-fields (themes, persons, GCAM scores, EventIds, and similar list-valued columns, in both GKG 2.1 and GKG 1.0) are stored as their raw, still-delimited strings in the Parquet output. Parsing those into their own structured columns is a separate concern this version doesn't attempt.

paths

All directories the pipeline reads from or writes to. Absolute or relative paths both work. Events keeps its original, unprefixed keys; every other dataset uses a prefixed sibling key for the same four stages, since mixing different datasets' files in one directory would be a real correctness hazard, not just an organizational one. The actual config key is <prefix><base key>, e.g. gkg_v1_counts_ + downloaded_data_directory = gkg_v1_counts_downloaded_data_directory.

The key names don't nest, but the example paths do: settings.example.yaml points every stage at data/<dataset>/<stage> (data/events/raw, data/gkg_v2/parquet, data/mentions/filtered, ...) rather than a flat data/<dataset>_<stage>, so the seven datasets stay easy to tell apart on disk even though nothing requires following that convention if you'd rather lay it out differently.

--dataset	Path prefix
events	(none, unprefixed)
events-15min	event_15min_
events-reduced	event_reduced_
gkg-v2	gkg_v2_
mentions	mentions_
gkg-v1	gkg_v1_
gkg-v1-counts	gkg_v1_counts_

Base key	Used by	Purpose
downloaded_data_directory	scrape	Where ZIP files land
unzipped_data_directory	convert	Scratch space for extracted CSVs (cleaned up automatically unless converter.keep_unzipped is true)
parquet_data_directory	convert, filter, sample	Flat Parquet output
filtered_data_directory	filter, sample	Flat filtered Parquet output

Two further keys exist for Events and events-reduced: parquet_historical_directory and filtered_historical_directory (Hive-partitioned Parquet, one directory per Year). For Events these are only used when converter.partitioning.enabled is true, gating its opt-in yearly/monthly split; events-15min, GKG 2.1, Mentions, and GKG 1.0/Counts have no pre-2013 yearly/monthly archive to partition at all, so they have no historical variant. events-reduced is the one exception to that toggle rather than a third case of "no historical variant": it has no flat output mode whatsoever, so its two historical keys are required and always used regardless of converter.partitioning.enabled, since every row it ever writes is Hive-partitioned by Year.

scraping

Key	Default	Description
retries	3	Retry attempts per file before giving up
timeout	30	Per-request timeout, in seconds
method	requests	Link-collection method: requests or selenium (see below)
chromedriver_path	null	Only used when method: selenium. Absolute path to chromedriver.exe, for when automatic download is blocked by a firewall
max_workers	8	Concurrent download workers

GKG 2.1 / Mentions discovery is a different mechanism entirely

method/chromedriver_path below only apply to --dataset events (and, per below, gkg-v1/gkg-v1-counts). GKG 2.1 and Mentions publish at 15-minute granularity (not daily) under data.gdeltproject.org/gdeltv2/, discovered via a single master file list rather than an HTML page to scrape: there's no requests-vs-selenium choice for them. Because of that granularity, a multi-year --dataset gkg-v2/mentions scrape can easily imply hundreds of thousands of small files; GdeltForge logs a warning (not a hard stop) before starting a scrape that large, and max_workers below is worth raising for that many-small-files workload.

GKG 1.0 uses Events' HTML-listing mechanism, at a different URL

gkg-v1 and gkg-v1-counts are daily files, like Events, not the 15-minute batches GKG 2.1/Mentions publish. Discovery scrapes an HTML directory listing at data.gdeltproject.org/gkg/ the same way Events scrapes data.gdeltproject.org/events/, just with a different base URL and filename shape (YYYYMMDD.gkg.csv.zip / YYYYMMDD.gkgcounts.csv.zip). That reuses the same parsing logic as the requests method above, but the exact markup of GKG 1.0's index page hasn't been directly confirmed against a live response (this environment can't reach data.gdeltproject.org at all); it's inferred from both paths sharing the same TLS certificate mismatch, i.e. the same underlying GCS bucket. Worth a small-scale test scrape before relying on it for a large historical pull.

requests vs selenium (Events only)

The scraper needs to read the file index at data.gdeltproject.org/events/ before it can download anything. That index is a plain, server-rendered HTML directory listing, not JS-rendered, so a headless browser is unnecessary overhead:

	requests (default)	selenium
How it works	Plain HTTP GET + regex over the HTML	Launches headless Chrome, waits for the DOM, reads <a> tags
Dependencies	Just requests (already required)	Chrome install + a version-matched ChromeDriver + selenium/webdriver-manager (uv pip install '.[selenium]')
Measured speed	~0.4s	~16s (~40x slower)
Failure modes	None specific to this site	Breaks whenever Chrome auto-updates past the pinned ChromeDriver version
When to use	Always, unless the page ever stops being static	Fallback only, in case GDELT ever switches this index page to a JS-rendered listing

Both methods return an identical set of URLs. selenium is kept purely as a fallback; installing it is optional (selenium/webdriver-manager are not in the default dependency set).

Concurrency and checksum verification

Downloads run through a bounded thread pool (max_workers) since they're I/O-bound. Each download is also checksum-verified: the GDELT index publishes an MD5 per file, which the scraper captures and checks after each download completes. A mismatch is treated the same as a network failure: the file is discarded and retried up to retries times before being reported as failed, so a corrupted or truncated download never silently ends up in the dataset.

converter

Key	Default	Description
keep_unzipped	false	Keep extracted CSVs after conversion instead of deleting them
file_pattern	"*.zip"	Glob pattern for which files in downloaded_data_directory to convert. A bare .csv matched here (e.g. "*.csv") is read directly, no extraction step, for a CSV that didn't come from a fresh scrape; see below
max_workers	null	Worker processes for conversion. null uses os.cpu_count()
max_workers_by_dataset.<dataset>	none	Overrides max_workers for one dataset. See "Capacity planning" below: a worker count safe for one dataset isn't necessarily safe for another, since it depends on peak per-worker memory
output_columns.<dataset>	none	Restricts CSV parsing to just these columns instead of every column columns.<dataset> defines. columns.<dataset> is still needed in full (it's what maps each raw position to a name on files with no header row), but the pruned subset is passed to polars.read_csv as integer positions, not names: this is what makes polars skip allocating/decoding whatever isn't in output_columns, the same optimization pandas' own usecols gave under the previous implementation. See "output_columns and crossref" below before pruning a dataset you plan to crossref later
compression.<dataset>	zstd	Parquet codec for converter's own output (parquet_data_directory), independent of filter.compression below for the filtered output that follows it. polars' own writer already supports zstd, gzip, brotli, lz4, and snappy natively, so this needs no new dependency
partitioning	see below	Optional Hive partitioning for historical (pre-daily) files

Conversion is CPU-bound (CSV parsing + Parquet writing), and each ZIP is independent, so it runs across a ProcessPoolExecutor.

output_columns is worth setting for GKG 2.1 in particular: most of its columns are free-text fields (quotations, all-names, GCAM, extras XML, image/video embeds) that a themes/tone/persons/orgs crossref never reads. See "Capacity planning" below for what dropping them and raising max_workers_by_dataset measurably bought on real data.

A bare .csv input (not inside a ZIP) is accepted alongside the archives real GDELT scrapes distribute, matched the same way, via file_pattern. It's read as-is with no extraction step; --delete-source/keep_unzipped still apply the same way (deleting or keeping the source once its parquet output is confirmed written), except keep_unzipped has nothing to keep, since nothing was ever extracted. Its filename plays no part in file-type detection (that requires a literal .zip suffix), so it always flat-writes to parquet_data_directory, regardless of converter.partitioning.enabled.

If a CSV extracted from a ZIP fails its own parquet write, or keep_unzipped kept it around after a config change invalidated its .done marker, it's left sitting in unzipped_data_directory. As long as the source ZIP is still in downloaded_data_directory, a plain rerun of convert picks it up again from scratch. If the ZIP is gone by then, see --recover-unzipped in the CLI reference.

Hive partitioning for historical data

The GDELT archive distributes pre-2013 data in yearly and monthly ZIPs (e.g. 1979.zip, 200601.zip) rather than daily files. Keeping those as flat Parquet files means every query scans thousands of files. Enabling partitioning routes them into a structured directory tree instead, so filters on Year or MonthYear skip irrelevant files entirely.

Off by default. To enable it:

paths:
  # existing paths ...
  parquet_historical_directory: "./data/events/historical"
  filtered_historical_directory: "./data/events/filtered_historical"

converter:
  partitioning:
    enabled: true
    rules:
      - file_type: yearly    # e.g. 1979.zip
        by: ["Year"]
      - file_type: monthly   # e.g. 200601.zip
        by: ["Year", "MonthYear"]

With partitioning enabled, gdeltforge convert produces two separate output areas:

data/
├── parquet/                        # daily files (2013-present), unchanged
│   ├── 20130401.export.parquet
│   └── ...
└── parquet_historical/             # yearly/monthly files, Hive-organized
    ├── Year=1979/
    │   └── 1979.parquet
    ├── Year=2006/
    │   └── MonthYear=200601/
    │       └── 200601.parquet
    └── ...

Daily ZIPs (2013-present) always go to parquet_data_directory as flat files, unaffected by this setting. Historical ZIPs that have already been converted are tracked with .done marker files, so re-running convert skips them safely. filter and sample detect the historical directory automatically from the config and include its data without any extra flags.

events-reduced is a related but separate case, not governed by converter.partitioning.enabled at all: it always writes Hive-partitioned by Year (never MonthYear, since its one file carries no month-level structure of its own), because it has no flat output mode to fall back to in the first place. Its parquet_historical_directory/filtered_historical_directory are required as soon as --dataset events-reduced is used, whether or not partitioning.enabled is set for Events.

Resumability

Flat output (Events daily, GKG 1.0, GKG 2.1, Mentions) is tracked with the same kind of .done marker historical output gets above: an interrupted convert run resumes from wherever it stopped instead of reprocessing every file from the start, the same way scrape skips already-downloaded files.

The marker records the run's own output_columns and compression settings, not just that a file was processed. Changing either between runs invalidates markers left by the old configuration, so a rerun reprocesses affected files instead of silently serving output shaped by settings that no longer match.

Markers are written as a dot-prefixed sibling of the data (.<name>.done), the standard convention for "not a data file" that pandas.read_parquet/pyarrow.dataset already skip on their own. A directory convert/filter has written into is safe to point any tool at directly, no special handling needed, even though filter's own source directory is convert's output directory and therefore always ends up holding both. Markers from before this existed aren't dot-prefixed; they're recognized and quietly migrated to the current naming the next time that file is checked, without reprocessing it.

filter

Key	Description
max_workers	Worker processes for filtering, same tradeoffs as converter.max_workers. null (default) uses os.cpu_count()
columns_to_check.<dataset>	Rows with a NaN/null value in any of these columns are dropped. Nested under the dataset name (mirroring columns/columns_numeric), one list per dataset
output_columns.<dataset>	Projects the filtered output down to this column subset, independent of columns_to_check (row-filtering still runs against the full row first). Unset keeps every column, same as before this existed. See "output_columns and crossref" below before pruning a dataset you plan to crossref later
compression.<dataset>	Parquet codec for the filtered output. Unset defaults to zstd. polars' own writer already supports zstd, gzip, brotli, lz4, and snappy natively, so this needs no new dependency
float32_columns.<dataset>	Narrows these float64 columns to float32 on write. Unset keeps every float column at full float64 precision. See "Capacity planning" below before using this: it's a real precision change, not free compression

This is the one section you should always customize: the example values are illustrative, not a recommendation. Pick the columns that matter for your analysis, e.g. if you don't need geocoding, don't require Actor1Geo_Lat/Actor1Geo_Long to be non-null, since that drops any event GDELT couldn't geolocate. See Column Redundancy for which columns are safe to prune from output_columns because they duplicate another column, measured against real GDELT data, and which look redundant but aren't.

Filtered output is resumable the same way convert's is (see "Resumability" above): a .done marker per file records columns_to_check, output_columns, float32_columns, and compression, so an interrupted filter run resumes instead of restarting from the first file, and changing any of those settings invalidates old markers rather than silently skipping files that need to be reprocessed under the new configuration.

output_columns and crossref: four columns you can't prune away

Both converter.output_columns and filter.output_columns share this same hazard: if you plan to run gdeltforge crossref on a dataset later, whichever stage you prune it in must keep the column the join actually runs on, no matter how aggressively you trim everything else:

Dataset	Required column	Used by
gdelt_event	GlobalEventID	Both join paths
gdelt_gkg_v1 / gdelt_gkg_v1_counts	EventIds	Direct join (crossref --gkg-version v1 / v1-counts)
gdelt_gkg_v2	V2DOCUMENTIDENTIFIER	Two-hop join (crossref --gkg-version v2)1
gdelt_mentions	GLOBALEVENTID, MentionIdentifier	The bridge hop itself, needed only for the v2 path1

Note that SOURCEURL is not on this list: the two-hop join to GKG 2.1 goes through Mentions' MentionIdentifier (which captures every article that mentioned an event), not through Events' own SOURCEURL (which only ever holds one representative article). Pruning SOURCEURL doesn't affect crossref at all.

gdelt_event_reduced has no entry in this table at all, and can't gain one: crossref needs a per-event or per-article identity to bridge through, and its pre-aggregated DATE+ACTOR1+ACTOR2+EVENTCODE rows carry none, no GlobalEventID, SOURCEURL, or DATEADDED. Sampling it and passing the result to crossref raises the same clear "must include a 'GlobalEventID' column" error as any other dataset missing its required column, not a special case.

See Crossref Join Semantics for how often an event ends up joined to more than one article (and vice versa) on real data, and for on_duplicate_document/dedupe_mentions, the two knobs controlling what happens when GKG 2.1 or Mentions themselves carry more than one record for what the join treats as a single (event, article) pair.

Dropping one of the required columns above doesn't corrupt anything: crossref checks for it explicitly and raises a clear error ("... must include a 'GlobalEventID' column" or similar) rather than silently returning wrong or empty results. The problem is when that error shows up: potentially after convert, filter, and a sample run have already completed on the pruned data, discovering the missing column only once you actually try to enrich it. Both run_converter and run_filter warn proactively instead, at the point output_columns is configured for either stage, against a single REQUIRED_JOIN_COLUMNS mapping shared with crossref.py itself so the two can't drift apart.

Every other column, the ones not on the table above, is handled more leniently: sample --mode filtered/--stratify and every crossref --gkg-version default their own output projection to a dataset's full columns.<dataset> schema unless you pass --columns yourself, and a real file pruned by output_columns isn't guaranteed to still have all of it. Rather than failing outright, whichever of those columns is actually missing is dropped, logged as a warning naming exactly what and why, and the run proceeds with what's left; only a column something can't function without at all (the join key above, --stratify's own column, a --filter condition's own column) still raises a clear error if genuinely missing, since silently dropping one of those would trade a loud failure for a quiet, misleading one instead.

Scraping has no equivalent warning, and can't: scrape downloads whole files, it never parses or selects individual columns, so there's no column-level decision to warn about at that stage. The closest real analog at the scrape stage is a coarser, dataset-level one, not choosing a column: crossref --gkg-version v2 needs Mentions data to exist locally at all, so scraping GKG 2.1 without ever also scraping Mentions produces the same downstream failure for a different reason. Nothing currently warns about that either.

The other way crossref can find nothing: sampling from before a GKG generation existed

Even with every required column intact, crossref finds nothing for an event dated before the target GKG generation's coverage actually starts¹, since there are no rows to match against, not a configuration problem. Both crossref_events_gkg_v1 and crossref_events_gkg_v2 now warn about this too, checked against each sampled event's DATEADDED (not Day, deliberately: Day is when an event is reported to have occurred, which can be far in the past for retrospective reporting, e.g. a 2003 event appearing in a 2013 daily file, while DATEADDED is when GDELT actually processed the record and matches the daily file's own date by construction, which is what actually determines whether a corresponding GKG/Mentions record could exist). The warning is a diagnostic, not a filter: events within coverage in the same sample still join normally, and the check silently skips if DATEADDED isn't in the sample at all (e.g. pruned out via --columns).

For a sample that genuinely spans both eras, --gkg-version auto (crossref_events_gkg_auto) is worth reaching for instead of picking one version and accepting the gap: rather than routing each event to exactly one generation by its own DATEADDED, it attempts every eligible event against both. DATEADDED only decides whether an event is within either generation's coverage window at all, not which single path is allowed to match it: a Mentions row is timestamped by when it was created, not by its event's DATEADDED, so an event from before 2015-02-18 can still have a real GKG 2.1 match created much later, and GKG 1.0 remains live and daily-published today, so a recent event isn't guaranteed to be GKG-2.1-only either. See the "GKG-Enriched Events Across the 2013-2015 Boundary" recipe in Recipes for a full worked example, including why the two generations' output columns are concatenated rather than unified (they don't share a single field name in common).

Dropping one of the columns above doesn't corrupt anything: crossref checks for it explicitly and raises a clear error ("... must include a 'GlobalEventID' column" or similar) rather than silently returning wrong or empty results. The problem is when that error shows up: potentially after filter and a sample run have already completed on the pruned data, discovering the missing column only once you actually try to enrich it. filter now warns proactively instead, at the point where output_columns is configured, if it detects a dataset's join key isn't in the kept column list, so you find out before those later steps run rather than after.

Capacity planning: real measured numbers

Everything below was measured against real GDELT data (not synthetic benchmarks), on GKG 2.1, since it's the dataset these knobs matter most for: mostly free-text fields, and 15-minute-interval files means a multi-year pull is hundreds of thousands of files. Treat these as a starting point for sizing your own pull, not a guarantee: your mix of news volume, disk, and CPU will shift the numbers.

Compression codec, one real day (120,728 rows, all 27 columns, previously snappy-only):

Codec	Size	Write time	vs. snappy
snappy (previous default)	744.7 MB	11.0s	baseline
gzip	422.3 MB	514.0s	1.8x smaller, ~47x slower to write
zstd (recommended)	410.8 MB	14.3s	1.8x smaller, same order of write time
brotli	284.2 MB	87.7s	2.6x smaller, ~8x slower to write
zstd, level 19	232.6 MB	slow	3.2x smaller, not worth it once columns are pruned (below)

gzip and zstd level 19 both cost far more write time than they're worth here; plain zstd is the pick, which is what filter.compression defaults gdelt_gkg_v2 to.

Column pruning, same day, output_columns set to the join key plus themes/tone/persons/orgs (10 of 27 columns):

Variant	Size	vs. snappy/all-columns
10 columns, snappy	97.1 MB	7.0x smaller
10 columns, zstd	54.7 MB	12.4x smaller

Column pruning did most of the work; the codec switch on top was a smaller, roughly 1.8x bonus. A second real day (2023-06-01, a heavier news day at 200,740 rows) landed at 94 MB pruned+zstd, consistent with the same ratio, so expect day-to-day variance of roughly 55-95 MB/day for GKG 2.1 rather than a single fixed number.

Conversion worker count, same 96-file day, output_columns active:

max_workers	Time	Rate	Notes
4, no pruning	30.2s	3.1 files/s	previous default, previous behavior
4, pruned	21.4s	4.5 files/s	pruning alone, same worker count
8, pruned	10.7s	9.0 files/s	new max_workers_by_dataset default for gdelt_gkg_v2
12, pruned	10.1s	9.5 files/s	marginal gain over 8, less headroom
20, pruned	11.4s	8.5 files/s	previously crashed unpruned at this count; completed cleanly pruned, but no faster

8 workers was picked over 12: nearly identical throughput with more memory headroom, since the original crash at os.cpu_count() was never root-caused, only reproduced and then avoided by pruning.

Putting it together: projecting to the full ~385,728-file GKG 2.1 archive (15-minute files, 2015-present) at these measured rates, with mentions (needed for the crossref join) excluded since it is small enough not to move these numbers much:

Scope	Files	Wall-clock (scrape + convert + filter)	Disk
Previous approach (no pruning, 4 workers, snappy)	385,728	~103 hours (~4.3 days)	~2.9 TB
Pruned + zstd + 8 workers	385,728	~46 hours (~1.9 days)	~220-380 GB

Scrape throughput (~4.2 files/s) is network-bound against data.gdeltproject.org and unaffected by any of the above; convert is where pruning and worker count actually move the number, from the previous bottleneck (~71 hours) down to roughly 12 hours.

Raw scrape footprint, separate from the Disk figures above: those are convert/filter's Parquet output, after column pruning and codec choice both apply. scrape downloads GDELT's files whole, unconditionally; output_columns can't reduce what lands on disk at this stage, since it only takes effect once convert parses a file. So the raw archive is close to a fixed cost, not a tunable one.

Measured two full real days directly (96 files each, GKG 2.1's 15-minute cadence):

Day	Total	Avg/file
2020-01-01	405.3 MB	4.22 MB
2023-06-01 (the same "heavier news day" referenced above)	822.5 MB	8.57 MB

Roughly a 2x day-to-day spread, GKG 2.1's raw size tracks news volume as much as its converted output does. A live master-file-list check counted 395,788 gdelt_gkg_v2 files (slightly ahead of the ~385,728 figure above, since GDELT keeps publishing; treat both as a moving target, not a fixed archive size), 4,122.8 days at the 15-minute cadence:

Basis	Total
Low (2020-01-01 rate)	~1.67 TB
High (2023-06-01 rate)	~3.39 TB
Average of both real days	~2.53 TB

That lands closer to the unpruned Parquet projection (~2.9 TB) than the pruned one (~220-380 GB): raw zip and unpruned-snappy Parquet both hold the full, unpruned content, just under different codecs, while pruning is a convert-time decision the raw archive never sees. convert --delete-source (see CLI Reference) removes each zip once its parquet output is confirmed written, the real lever to avoid holding both footprints on disk at once; filter --delete-source does the same for the converted parquet once its filtered output exists. Neither is on by default, and combined with any column-pruning or row-filtering setting, whatever that dropped can't be recovered later without redoing an earlier stage.

events-15min is a much lighter pull than GKG 2.1 at the same file count: a live master-file-list check counted 396,086 .export.CSV.zip files (2015-02-18 to present, same window as GKG 2.1/Mentions), totaling ~39.7 GB, ~100 KB/file average. File count, not raw size, dominates the cost here (~81x the daily events archive's file count for the same date range): Events rows are compact structured data, not GKG's free text, so the per-file and total-size story looks much closer to Mentions' ~67 GB than to GKG 2.1's multi-TB footprint.

Dtype narrowing: where it does and doesn't pay off

A natural next question after the above is whether narrowing individual column types saves more. The answer depends entirely on whether the column is low-cardinality or genuinely continuous, and the two cases point in opposite directions.

Low-cardinality integers (Actor1Geo_Type, QuadClass, ...): narrowing barely helps. Actor1Geo_Type/Actor2Geo_Type/ActionGeo_Type have exactly 6 distinct values in practice (0-5, confirmed against 8.3M real Events rows), and QuadClass has 4. The intuitive expectation is that declaring these as uint8 instead of int64 should save close to 8x. It doesn't: Parquet already dictionary-encodes low-cardinality integer columns by default regardless of the declared Arrow type, so a 6-value column gets stored as small bit-packed dictionary indices either way. Measured on a real 5M-row sample, int64 to uint8 gave a 1.00-1.01x change (noise level) for every one of these columns. IsRootEvent to bool was the one exception, a real 17% reduction on that column specifically, since Parquet's native boolean bit-packing beats even dictionary-encoded int64 for a 2-value column, but that column is too small a slice of a full row (these five candidate columns combined are only ~2% of a full Events file) for it to be worth a config option on its own. This was investigated but not implemented for that reason.

Continuous floats (AvgTone, lat/long, ...): narrowing to float32 saves real space, but is not lossless. Dictionary encoding can't compress a column with hundreds of thousands of distinct values the way it compresses a 6-value one, so float64 to float32 does save real space here: AvgTone measured 1.30-1.33x smaller, lat/long columns 1.02-1.03x.

The first pass at this reasoned "float32's ~7 significant digits should be plenty for a tone score" without checking GDELT's actual emitted precision. That assumption was wrong. A real downloaded Events file (20130401.export.CSV.zip) shows AvgTone values with up to 16 decimal places and 15 significant figures in source data, e.g. 0.0284010224368077, and a direct round-trip test (cast to float32, back to float64, compare to the original) against 6.5M real rows confirms the practical effect: the value changes on 31% of rows for GoldsteinScale, 96% for AvgTone, and literally 100% for FractionDate. Each individual change is tiny (on the order of float32's ~1.19e-7 relative precision floor), but it is a genuine, measurable change to the value, not just a smaller encoding of the same one.

That's exactly why filter.float32_columns exists as an explicit, per-dataset, per-column opt-in rather than a blanket setting or a new default: it's available for anyone who has decided that tradeoff is acceptable for their use case, but nothing is cast to float32 unless a column is named there.

Why compression defaults to zstd now, for every dataset

The GKG 2.1 codec numbers earlier in this page don't automatically transfer to Events, since it's a different content mix (mostly short codes, names, and dates rather than GKG's free text). Measured directly on 5.8M real Events rows, all 58 columns:

Codec	Size	bytes/row	Write time
snappy (previous default)	471.6 MB	81.1	64.0s
zstd (current default)	330.4 MB	56.8	50.5s

Roughly 30% smaller, and faster to write, not slower. Since zstd is lossless, this isn't a tradeoff to weigh the way float32_columns is: there's no case where snappy is the better default. filter.compression defaults to zstd for every dataset as of 2026-08-07; compression.<dataset> remains available to override to a specific codec if one is ever needed.

pandas vs polars: real measured throughput

The pipeline moved from pandas to polars for every DataFrame operation (convert's CSV parsing, filter's row/column pruning, sample's reservoir scanning, crossref's joins). Measured with a dedicated benchmark script via the real gdeltforge convert/gdeltforge filter/gdeltforge crossref CLI entry points, once from a pandas-based checkout and once from this one, against identical synthetic fixtures shaped like real Events/Mentions/GKG 2.1 data (Windows, single machine, one run per size, not averaged):

convert, a single Events-shaped file at each row count:

Rows	pandas	polars	Speedup
10,000	2.04s	1.63s	1.25x
100,000	3.30s	1.44s	2.30x
1,000,000	22.72s	3.45s	6.59x
10,000,000	796.46s	56.08s	14.20x

The gap widens sharply with size rather than staying fixed: at 10,000 rows both engines spend most of their wall-clock on process/interpreter startup, not CSV parsing, so there's little for a faster parser to win back yet. Past that, polars' advantage compounds, reaching over 14x at 10M rows, comfortably ahead of what the raw row-count growth (1,000x from 10k to 10M) alone would predict for a fixed-overhead explanation.

filter, a single already-converted Events file at each row count, dropping rows with a null in any of three geo lat/long pairs (roughly 39% of rows dropped, a chosen rate for exercising real work, not a measured real-world geocoding-failure rate):

Rows	pandas	polars	Speedup
10,000	1.78s	2.29s	0.78x
100,000	1.60s	1.65s	0.97x
1,000,000	3.93s	1.99s	1.97x
10,000,000	107.31s	4.04s	26.56x

filter is the one stage where polars is measurably slower at small sizes, not just less ahead: at 10,000 rows it's about 1.3x slower than pandas, and the two are within noise of each other at 100,000. The likely cause is architectural, not a regression: the polars port reports rows_before/rows_after as two separate lf.select(pl.len()) passes plus the actual sink_parquet write, three passes over the file, where the pandas implementation's single streaming batch loop (pyarrow.ParquetFile.iter_batches + per-batch dropna + write) made do with one. That fixed per-pass cost dominates at small files and is completely swallowed at scale: by 10M rows polars finishes in 4 seconds what takes pandas over a minute and a half, a 26.6x difference, the largest gap measured anywhere in this comparison.

crossref, Events joined against synthetic Mentions/GKG 2.1 (roughly 80% of events finding at least one match):

Events	pandas	polars	Speedup
1,000	0.78s	0.62s	1.26x
5,000	0.74s	0.65s	1.14x
10,000	0.79s	0.65s	1.22x
100,000	2.37s	2.11s	1.12x

crossref does not show the same widening pattern: the speedup stays in a narrow 1.1-1.3x band across two full orders of magnitude in event count, unlike convert's clear scaling trend. The most likely explanation is that this benchmark's own fixture is a single Mentions file and a single GKG 2.1 file per size, so the join itself (a hash join against an in-memory key set either engine handles well) is a smaller fraction of total wall-clock than process startup, config/schema loading, and Python-level orchestration, none of which the engine swap touches. This doesn't rule out a bigger real-world win at archive scale (thousands of Mentions/GKG 2.1 files, where _dataset's own per-file footer-schema read and predicate pushdown do proportionally more work), just that this benchmark's own fixture shape doesn't exercise that path; a genuine multi-file archive-scale crossref benchmark is a natural follow-up, not yet measured.

converter.compression defaults to zstd too, for the same reason: it wasn't independently re-measured against converter's own (unfiltered, wider-row-count) output, but a lossless codec with no measured downside on real GDELT data has no case for defaulting to snappy there either. It was previously hardcoded to snappy with no way to change it; it's now a normal per-dataset setting, same shape as filter.compression.

---

1. The v2 path has nothing to join before 2015-02-18: Mentions and GKG 2.1 didn't exist until GDELT 2.0 launched that day. v1/v1-counts reaches back further, to April 2013. ↩↩↩