Home

Anansi is a framework for building web crawlers, and a web crawler built using that framework.

It consists of:

• libcrawl, a library for implementing web crawlers.
• crawld, a daemon which uses a relational database-based queue.
• Processors which can be enabled with crawld, in order to provide additional functionality.
• Some utilities built using libcrawl

Etymology

See the description of Anansi on Wikipedia.

Licence

Anansi is open source software, distributed according to the terms of the Apache License, 2.0.