The Crawler-Lib Engine is a general purpose workflow enabled task processor. It has evolved from a web crawler over data mining and information retrieval. It is throughput optimized and can perform thousands of tasks per second on standard hardware. Due to its workflow capabilities it allows to structure and parallelize even complex kind of work. Please visit the project page for the complete view of the Crawler-Lib Engine. A license for the Anonymous Edition is included in the package. A license for the more powerful free Community Edition can be generated on the project page. A unrestricted license is available too.
Project Url
View on NuGet:


Here are the packages that version 2.3.5544.21265 of CrawlerLib.Engine depends on.

HtmlAgilityPack : (1.4.6) Newtonsoft.Json : (6.0.3)



Installing with NuGet

PM> Install-Package CrawlerLib.Engine

Packages that Depend on CrawlerLib.Engine

PackageLatest VersionTags
CrawlerLib.Engine.TestHelper 2.3.5544.21265 crawler-lib test helper nunit mstest unit-test crawler spider bot scraper workflow http html xml json rss