- Protocols : http(s)/ftp(s)/tor hidden service/magnet URIs
- Automatic authentication handling for : ftp/http basic/http digest/form html
- Customs handling rules (domain specific)
- Customs harvesting rules
- Smart crawling in order to spare bandwidth
- Optional secure connection(ssl) between crawling nodes
- Slave : node harvesting the remote services(ex: web pages).
- Master : dividing the work over slaves (based on URIs) and handling harvesting rules.
- Monitor : handling the loadbalacing over manager. Only one active leader.
- Admin : used in order to monitor the crawling cluster.
Time and Space-Efficient Algorithms for Mobile Agents in an Anonymous Network,
Efficient leader election in complete networks,
Using Name-Based Mappings to Increase Hit Rates,
IEEE/ACM Trans. Netw. 6(1),