Filtered sampling: filter syntax reference¶

FilteredSampler (the gdeltforge sample --mode filtered command) filters GDELT Parquet files using a flexible JSON dictionary passed via --filter. Filters define what rows are kept before sampling, and support:

simple equality
lists (IN)
numeric ranges
relational operators (>, <)
nested AND / OR logical blocks

Below is the complete specification. For runnable end-to-end examples built on it, see Recipes; to check that a filter value is a valid code before running, see gdeltforge codes.

Which columns you can filter on depends on your dataset

--dataset events reads GDELT's daily archive, still exported in the older, 58-column, GDELT-1.0-compatible schema for backward compatibility. --dataset events-15min reads the same underlying events in GDELT 2.0's native 61-column format instead, adding Actor1Geo_ADM2Code/Actor2Geo_ADM2Code/ActionGeo_ADM2Code, finer administrative-region geocoding than the ADM1Code both schemas already carry. --mode filtered's own --filter and --columns both reject one of those three fields against events at startup, before any file is even scanned: --filter with Invalid filter column: ..., --columns with Invalid columns: {...}, since filtered sampling validates both against the dataset's own declared schema up front. --mode indexed/calendar's --columns behaves differently: it isn't validated against a declared schema at all, so naming one of those three fields against events there just warns that it wasn't found in the scanned data and drops it, the same graceful handling any other absent-but-non-essential column already gets; those two modes don't take a --filter condition at all. See Configuration for the full schema comparison.

Basic filter types (single column)¶

Equality

{ "ActionGeo_CountryCode": "US" }

Equivalent to ActionGeo_CountryCode == "US".

IN list

{ "QuadClass": [1, 2, 3] }

Equivalent to QuadClass ∈ {1, 2, 3}.

Numeric range

Range filters require the explicit dictionary form (see below):

{ "GoldsteinScale": { "op": "between", "min": 0, "max": 5 } }

Equivalent to 0 ≤ GoldsteinScale ≤ 5.

Arrays are lists, never ranges

A JSON array such as [0, 5] is always treated as an IN-list (isin), matching exactly 0 or 5, not the range 0-5. Ranges need the explicit {"op": "between"} form above.

Dictionary operator (explicit)¶

All operator forms:

Operator	Example	Meaning
`equals`	`{ "IsRootEvent": { "op": "equals", "value": 1 } }`	`IsRootEvent == 1`
`in_list`	`{ "QuadClass": { "op": "in_list", "values": [1, 2] } }`	`QuadClass ∈ {1, 2}`
`gt`	`{ "NumArticles": { "op": "gt", "value": 20 } }`	`NumArticles > 20`
`lt`	`{ "NumMentions": { "op": "lt", "value": 5 } }`	`NumMentions < 5`
`between` / `range`	`{ "GoldsteinScale": { "op": "between", "min": -2, "max": 2 } }`	`-2 ≤ GoldsteinScale ≤ 2`

All of the above apply to any numeric or categorical GDELT column.

Logical groups¶

filter_dict can contain nested AND / OR blocks to build richer logic.

Top-level AND (default behavior: multiple keys are combined with AND)

{
  "ActionGeo_CountryCode": "US",
  "QuadClass": [1, 2]
}

Equivalent to ActionGeo_CountryCode="US" AND QuadClass in {1,2}.

Top-level OR

{
  "OR": {
    "ActionGeo_CountryCode": "US",
    "Actor1CountryCode": "USA"
  }
}

Equivalent to ActionGeo_CountryCode="US" OR Actor1CountryCode="USA".

Nested AND inside OR

{
  "OR": {
    "Actor1CountryCode": "BRA",
    "AND": {
      "Actor2CountryCode": "BRA",
      "ActionGeo_CountryCode": "BR"
    }
  }
}

Equivalent to Actor1="BRA" OR (Actor2="BRA" AND ActionGeo="BR").

Nested OR inside AND

Example: keep USA events and events where either actor is Russia:

{
  "AND": {
    "ActionGeo_CountryCode": "US",
    "OR": {
      "Actor1CountryCode": "RUS",
      "Actor2CountryCode": "RUS"
    }
  }
}

Equivalent to ActionGeo_CountryCode="US" AND (Actor1="RUS" OR Actor2="RUS").

Deeply nested example

You can combine arbitrarily:

{
  "OR": {
    "AND": {
      "IsRootEvent": 1,
      "QuadClass": [1, 2]
    },
    "OR": {
      "Actor1CountryCode": "CHN",
      "Actor2CountryCode": "CHN"
    }
  }
}

Equivalent to (IsRootEvent=1 AND QuadClass in {1,2}) OR (Actor1="CHN" OR Actor2="CHN").

Selecting specific columns¶

You may restrict the output to specific columns, which is a memory-friendly practice:

gdeltforge sample \
  --dataset events \
  --mode filtered \
  --filter '{"ActionGeo_CountryCode": "US"}' \
  --columns GlobalEventID Year Actor1Code \
  -n 1000

Sampling methods compatible with filters¶

Once the filter is applied, sampling works normally:

Random sample

gdeltforge sample --dataset events --mode filtered -n 5000 --filter '{"QuadClass":[1,2]}'

Stratified by column

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode":"US"}' \
    --stratify QuadClass \
    --n-per-group 500

Stratified output is not representative of the natural class distribution

--stratify draws the same --n-per-group count from every distinct value of the stratify column, on purpose, so the result is class-balanced regardless of how common each class actually is in the archive. That's the right shape for a model training set that needs coverage of a rare class, not a population-level estimate: computing an unweighted rate off a stratified sample (e.g. "what fraction of US events are QuadClass 4") reads back the sampling ratio you chose, not the real one. Use --mode filtered without --stratify (or --mode indexed) when the goal is an estimate of the natural distribution instead. See Limitations for the full picture across all three sampling modes.

Every stratified run also writes <out>.strata.json, recording each stratify value's true row count in the filtered archive regardless of --n-per-group. That's the number an unweighted rate above is missing: reweight each group by true_count / n_per_group before computing a population-level statistic, and the equal-allocation bias washes out.

Quick reference¶

Filter type	Example JSON	Meaning
equal	`"X": "USA"`	`X == "USA"`
in list	`"X": [1,2,3]`	`X ∈ {1,2,3}`
op:equals	`"X": {"op":"equals","value":10}`	explicit equality
op:gt	`"X": {"op":"gt","value":0}`	`X > 0`
op:lt	`"X": {"op":"lt","value":5}`	`X < 5`
op:between	`"X":{"op":"between","min":0,"max":10}`	`0 ≤ X ≤ 10`
AND block	`"AND": {...}`	all conditions must match
OR block	`"OR": {...}`	any condition may match
nested logic	`{"OR": {"X":1, "AND": {...}}}`	combine logic trees

See Recipes for complete, runnable examples built on this syntax.