togatoga/wikidata-grep

wikidata-grep (wdgrep)

A fast grep for Wikidata dumps: match entities by their claims, and by their place in the type graph.

wdgrep is a fast grep for Wikidata JSON dumps, inspired by wikibase-dump-filter. It reads a dump on stdin and writes the matching entities as newline-delimited JSON (NDJSON), one object per line. Beyond matching flat claims, it also follows the graph that properties like P31 (instance of) and P279 (subclass of) form, so you can match by reachability: for example, every entity that is a kind of chain (Q65553774) at any depth, which a single --claim pass cannot express.

Two kinds of matching:

• Per-entity filters (--claim, --sitelink, --type): keep an entity by

looking only at its own fields, fast.

• Graph reachability (--graph): build-graph distils the dump into a

compact property-graph once, then --graph-include/--graph-exclude keep an entity by whether its id can reach a target QID through that graph. See Graph grep.

# keep only humans (entities with a P31:Q5 claim)
cat latest-all.json | wdgrep --claim P31:Q5 > humans.ndjson

# directly from a compressed dump
gzip -dc latest-all.json.gz | wdgrep --claim P31:Q5 > humans.ndjson

# every chain (Q65553774), including all subclasses and their instances
wdgrep build-graph --properties P31,P279 < latest-all.json > graph.ndjson
wdgrep --graph graph.ndjson --graph-include Q65553774 < latest-all.json > chains.ndjson

Contents

• Install
• Get a Wikidata dump
• Usage
• Filters
• Claim expressions
• Qualifier matching

• Sitelink expressions
• Graph grep
• Step 1: build the graph
• Step 2: filter the full dump by graph reachability
• build-graph options
• Formatters

• Other options
• Benchmark
• License

Install

From crates.io (the crate is wikidata-grep; the installed command is wdgrep):

cargo install wikidata-grep

Or build from source:

cargo install --path .   # installs the `wdgrep` binary
# or
cargo build --release    # binary at target/release/wdgrep