4 Commits (f510fb82ddbe2f1edbfa6a5112d67e49ed4a5455)

Author	SHA1	Message	Date
Michael Peter Christen	2de159719b	added an option to set 'obey nofollow' for links with rel="nofollow" ... attribute in the <a> tag for each crawl. This introduces a lot of changes because it extends the usage of the AnchorURL Object type which now also has a different toString method that the underlying DigestURL.toString. It is therefore not advised to use .toString at all for urls, just just toNormalform(false) instead.	11 years ago
Michael Peter Christen	f23c4142e0	added option to configure a custom user agent within allip networks	11 years ago
Michael Peter Christen	a2fba6584f	use submitted default userAgent if cloning a crawl	11 years ago
orbiter	d29b6db270	made crawl start pages public since they do not reveal individual ... information and they are also not used as servlet to actually start the crawl (which is Crawler_p.html).	11 years ago