crawl - Small, efficient web crawler with advanced features

Property Value
Distribution FreeBSD 11
Repository FreeBSD Ports Quarterly i386
Package filename crawl-0.4_14.txz
Package name crawl
Package version 0.4
Package release 14
Package architecture i386
Package type txz
Category www
Download size 27.88 KB
Installed size 69.57 KB
The crawl utility starts a depth-first traversal of the web at the
specified URLs. It stores all JPEG images that match the configured
constraints.  Crawl is fairly fast and allows for graceful termination.
After terminating crawl, it is possible to restart it at exactly
the same spot where it was terminated. Crawl keeps a persistent
database that allows multiple crawls without revisiting sites.
The main reason for writing crawl was the lack of simple open source
web crawlers. Crawl is only a few thousand lines of code and fairly
easy to debug and customize.
Some of the main features:
- Saves encountered JPEG images
- Image selection based on regular expressions and size contrainsts
- Resume previous crawl after graceful termination
- Persistent database of visited URLs
- Very small and efficient code
- Supports robots.txt
===>   NOTICE:
The crawl port currently does not have a maintainer. As a result, it is
more likely to have unresolved issues, not be up-to-date, or even be removed in
the future. To volunteer to maintain this port, please create an issue at:
More information about port maintainership is available at:


Package Version Architecture Repository
crawl-0.4_14.txz 0.4 amd64 FreeBSD Ports Latest
crawl-0.4_14.txz 0.4 i386 FreeBSD Ports Latest
crawl-0.4_14.txz 0.4 amd64 FreeBSD Ports Quarterly
crawl - - -


Name Value
db5 = 5.3.28_7
libevent = 2.1.11 -


Type URL
Binary Package crawl-0.4_14.txz
Source Package www/crawl

Install Howto

Install crawl txz package:

# pkg install crawl

See Also

Package Description
crc32c- CRC32C implementation supporting CPU-specific acceleration
cre-1.2.b.r2_1.txz Corewars Redcode Evolver
cream-0.43.txz Gvim extension with many features
create-cert-2.7.txz Create openssl client key and certificates
create-rb-port-0.0.3.txz Script to ease creation of rubygem- ports
createtorrent-1.1.4.txz Create BitTorrent files from the command line
credis-0.2.3.txz Library for communicating with Redis servers
credns-0.2.10_2.txz Verifier performing validation in the DNS notify/transfer-chain
creduce-2.10.0.txz Produces small test cases
cretan-hubris-0.2_6.txz Maze on the Isle of Crete
crex-0.2.5_2.txz Explore, test, and check regular expressions in the terminal
crf++-0.54_1.txz Yet Another CRF toolkit
cricket-1.0.5_12.txz High performance, flexible monitoring system
crimson-1.1.3_2.txz Implements the Java API for XML Processing (JAXP)
crimson-fields-0.5.3_5.txz Tactical war game in the tradition of Battle Isle