You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen 2de159719b added an option to set 'obey nofollow' for links with rel="nofollow" ... attribute in the <a> tag for each crawl. This introduces a lot of changes because it extends the usage of the AnchorURL Object type which now also has a different toString method that the underlying DigestURL.toString. It is therefore not advised to use .toString at all for urls, just just toNormalform(false) instead.		11 years ago
..
data	added an option to set 'obey nofollow' for links with rel="nofollow"	11 years ago
retrieval	added an option to set 'obey nofollow' for links with rel="nofollow"	11 years ago
robots	new Strategies in Balancer:	11 years ago
Balancer.java	- added a new Crawler Balancer: HostBalancer and HostQueues:	11 years ago
CrawlStacker.java	added an option to set 'obey nofollow' for links with rel="nofollow"	11 years ago
CrawlSwitchboard.java	added an option to set 'obey nofollow' for links with rel="nofollow"	11 years ago
HarvestProcess.java	fix for wrong display of error urls in HostBrowser	12 years ago
HostBalancer.java	enhanced crawler host round-robin strategy	11 years ago
HostQueue.java	Crawler hostbalancer does not delete finished queue files,	11 years ago
LegacyBalancer.java	special strategy for balancer: do not remove targets with zero wait time	11 years ago