FactOWL is a blazingly fast, open-source, and fully reproducible factuality evaluation tool for long-form Large Language Model (LLM) generations.
Built to overcome the limitations of pioneering tools like FActScore and SAFE, FactOWL eliminates the need for expensive proprietary APIs and outdated static knowledge bases. By leveraging Open LLMs (Llama-3-8B), real-time Wikipedia search, and Wikidata triple aggregation, FactOWL effectively resolves entity ambiguity and delivers human-aligned factuality precision at a fraction of the computational cost.
π Key Highlights:
- β‘ Blazingly Fast: Up to 10Γ faster than FActScore and 20Γ faster than SAFE.
- π Real-Time & Multi-Source: Dynamically queries live Wikipedia and Wikidata, solving the "outdated knowledge dump" problem.
- π° Cost-Efficient: 100% open-source pipeline. No paid search engines or proprietary LLM APIs required.
- π― High Alignment: Achieves stable, human-aligned factuality precision without complex non-parametric post-processing.
- π§© Modular Architecture: Easily swap retrievers, rerankers (GTR/BM25), or judge LLMs.
FactOWL decomposes factual evaluation into three transparent, robust stages:
- Atomic Fact Extraction: Prompts an open LLM to break down long-form text into self-contained, unambiguous atomic facts, explicitly resolving anaphoric pronouns and filtering out unrelated common-sense claims.
- Evidence Retrieval: Performs multi-page real-time Wikipedia searches and retrieves verbalized Wikidata triples to cover homonymous topics and rare entities. Passages are reranked using GTR or BM25.
- Fact Verification: Independently verifies each atomic fact against the retrieved context using a lightweight LLM (e.g., Llama-3-8B-Instruct) accelerated by
vLLM.
Install the latest stable release directly from PyPI:
pip install factowlTo run the full pipeline, including local Wikipedia dump indexing and Jupyter notebooks:
# 1. Clone the repository
git clone https://github.com/s-nlp/factowl.git
cd factowl
# 2. Create and activate the conda environment
conda env create -f environment.yml # or use your factowl.yml
conda activate factowl
# 3. Install the package in editable mode
pip install -e .
# 4. Download required NLP models for data preprocessing
python -m spacy download en_core_web_sm
---------
# FactOwl: Fast and modern factchecker
## Usage Examples
1. [Fact generation and verification using pre-computed contexts](https://github.com/s-nlp/factowl/blob/main/examples/eval_no_retrieve_ridic_hf.ipynb) only on RiDiC dataset
2. [Fact generation and verification on FactScore dataset](https://github.com/s-nlp/factowl/blob/main/examples/eval_factowl_factscore_data.ipynb) usign either fixed Wikipedia dump or Wikipedia search for context retrieval
3. LLM and rule-based [fact filtration](https://github.com/s-nlp/factowl/blob/main/examples/filter_facts.ipynb)
## Installation
### For development:
```bash
conda update --file factowl.yml
conda activate ltf
pip install -e ./factowl/
python -m spacy download en_core_web_sm
python factowl/factowl/download_data.pyand also download enwiki dump from this link and place it into ./data/ directory.
pip install factowl
To run the scorer on the provided sample dataset, simply use:
python -m factowl --input_file data/sample_input.json
You can also provide your own data into the script in the following format:
{
"topics": [
"{topic_id}": "{topic_name}"
],
"generations": [
"{topic_id}": "{generation}"
]
}
Explore our comprehensive Jupyter notebooks to understand the pipeline internals, experiment with context retrieval, and apply custom fact filtration rules.
FactOWL is designed to be highly accessible. While the original paper experiments were conducted on 3x NVIDIA RTX-3090 (24GB), the core verification pipeline utilizing Llama-3-8B and vLLM can comfortably run on a single consumer GPU (e.g., RTX 3060/4070 with 12GB+ VRAM) or Apple Silicon via MLX/Ollama integrations.
If you use FactOWL in your research, demo, or production systems, please cite our SIGIR 2026 paper:
## π Quick Start
Evaluate your LLM generations via the Command Line Interface (CLI). FactOWL uses `vLLM` under the hood for maximum throughput.
```bash
python -m factowl --input_file data/sample_input.json --retriever wikipedia_searchProvide your data in a structured JSON format. FactOWL handles batch processing automatically:
{
"topics": {
"topic_01": "William Post (lottery winner)",
"topic_02": "Lanny Flaherty"
},
"generations": {
"topic_01": "William Post (1949-1986) was an American lottery winner who won $16.2 million...",
"topic_02": "Lanny Flaherty is an American actor born on December 18, 1949..."
}
}Explore our comprehensive Jupyter notebooks to understand the pipeline internals, experiment with context retrieval, and apply custom fact filtration rules.
FactOWL is designed to be highly accessible. While the original paper experiments were conducted on 3x NVIDIA RTX-3090 (24GB), the core verification pipeline utilizing Llama-3-8B and vLLM can comfortably run on a single consumer GPU (e.g., RTX 3060/4070 with 12GB+ VRAM) or Apple Silicon via MLX/Ollama integrations.
If you use FactOWL in your research, demo, or production systems, please cite our SIGIR 2026 paper:
@inproceedings{10.1145/3805712.3808373,
author = {Sakhovskiy, Andrey and Sushko, Nikita and Marina, Maria and Konovalov, Vasily and Tutubalina, Elena and Panchenko, Alexander and Braslavski, Pavel},
title = {FactOWL: A Cost-Efficient Tool for Long-Form Factuality Evaluation},
year = {2026},
isbn = {9798400725999},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3805712.3808373},
doi = {10.1145/3805712.3808373},
abstract = {The presence of factual hallucinations in large language model (LLM) generations drives the development of retrieval-augmented factuality evaluation tools. However, existing implementations suffer from slow inference, outdated knowledge bases, and the use of paid search and LLM APIs, which hinders research and practical applications. To fill this gap, we propose FactOWL, a FActScore-based Factuality evaluation tool which adopts an Open LLM and real-time Wikipedia search for evaluation of Long-form LLM responses. FactOWL effectively addresses unreliable evidence, incomplete contexts, and entity ambiguity by performing multi-source aggregation over Wikipedia pages and Wikidata triples. FactOWL is at least 10\texttimes{} faster than the pioneering FActScore fact checking tool and its key modifications methods and is freely available on GitHub: https://github.com/s-nlp/factowl.},
booktitle = {Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {5209β5214},
numpages = {6},
keywords = {llm factuality, factuality evaluation tool, atomic fact generation, fact verification, wikipedia, wikidata, evidence retrieval},
location = {Australia},
series = {SIGIR '26}
}We pay our respects to the traditional owners of the lands where SIGIR 2026 was hosted, the peoples of the Woi Wurrung and Boon Wurrung language groups of the eastern Kulin Nation. We pay our respects to their Elders past and present, and extend that respect to all Aboriginal and Torres Strait Islander peoples today and their continuing connection to land, sea, sky, and community.