📟 longeval-cli

Build dynamic test collections from search logs. longeval-cli turns raw search logs into a sequence of self-contained snapshots, each with its own queries, relevance labels (qrels), and documents. It is introduced in our paper Building Dynamic Test Collections: LongEval CLI and Scientific Retrieval Dataset (under review) to which we refer for the methodological background.

The pipeline

Each stage stores its results in the database or in files, so you can stop, inspect, and continue at any stage, and add a new snapshot without reprocessing the earlier ones.

Stage	What you provide	Commands	Paper
1. Ingest Logs	a parser for your log format	import-queries, import-clicks	§3.1 Log Ingestion
2. Select Queries	classifiers and a filter YAML per snapshot	classify, export queries	§3.2 Query Selection and Interaction Filtering
3. Create Labels	a CLAX click model (Python script)	export clicks, export qrels	§3.3 Pseudo Relevance Label Creation
4. Fetch Docs	a document source	collect-document-candidates, select-documents, fetch-documents	§3.4 Document Collection
5. Package	–	export snapshot (and export documents)	§3.5 Packaging
6. Retrieve	TIRA + Docker	retrieve	§3.6 Retrieval

Everything that depends on your search engine (parser, classifiers, document source) is a small Python file you pass on the command line, so the toolkit itself stays format-independent. The exported snapshots can be loaded with ir_datasets_longeval.

Installation

pip install longeval-cli                 # base install
pip install "longeval-cli[clax]"         # click-model training and scoring (JAX, CLAX)
pip install "longeval-cli[postgres]"     # if your database is PostgreSQL
pip install "longeval-cli[tira]"         # for `retrieve`

Extras combine, e.g. pip install "longeval-cli[clax,postgres]"; with uv, use uv add longeval-cli or uv tool install longeval-cli. On Linux, the clax extra pulls in PyTorch, whose default wheels bring ~4 GB of CUDA libraries which are not really needed. To skip them, install the CPU build first: pip install torch --index-url https://download.pytorch.org/whl/cpu.

To run the examples (the quickstart below and the tutorial) or to develop, work from a clone of this repository instead:

git clone https://github.com/clef-longeval/longeval-cli.git && cd longeval-cli
uv sync                      # base install
uv sync --extra clax         # likewise --extra postgres / --extra tira

The database defaults to a local SQLite file. Pass --database-url (or set LONGEVAL_DATABASE_URL) to use another file or PostgreSQL, e.g. postgresql+psycopg://user:password@host/database.

Quickstart

Run the first stages on the synthetic logs that ship with the repository:

export DB=sqlite:////tmp/minimal-example.sqlite3 E=examples/minimal-example

uv run longeval-cli init-db --database-url "$DB"
uv run longeval-cli import-queries $E/logs --parser $E/parser.py --database-url "$DB"
uv run longeval-cli import-clicks  $E/logs --parser $E/parser.py --database-url "$DB"
uv run longeval-cli classify --classifier $E/classifiers.py --filter $E/filter.yaml --database-url "$DB"
uv run longeval-cli database-status --database-url "$DB"

The tutorial continues from here through click-model training, qrels, documents, and the packaged snapshot, with the expected output of every step.

Documentation

Tutorial	the whole pipeline on a small synthetic dataset with known ground truth
Command reference	every command, its options and output formats, the filter YAML
LongEval Sci 2026 setup	the configuration used for the collection in the paper (CORE logs)
Agent skills	step-by-step guides that let an AI agent (e.g. Claude Code) run the toolkit for you

The agent skills double as how-to guides for the four tasks you will do most:

• longeval-log-parser: write a parser for a new log format
• longeval-classifier: write or configure classifiers
• longeval-snapshot-filter: write a snapshot's filter YAML
• longeval-build-snapshot: build and release a snapshot end to end

Roadmap

• Query intent classification
• Query recency classification
• Topic generation
• Sampling instead of pooling
• Generalize filters.yml into a snapshot config file to reduce options needed

License

MIT, see LICENSE.