You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen 06afb568e2 new Strategies in Balancer: ... - doublecheck cache now records the crawl depth as well - doublecheck cache is available from the outside (made static) - no more need to crawl hosts with lowest depth first, instead all hosts which have only singleton entries are preferred to reduce the number of files.		11 years ago
..
RobotsTxt.java	new Strategies in Balancer:	11 years ago
RobotsTxtEntry.java	support for multiple sitemaps in robots.txt	11 years ago
RobotsTxtParser.java	support for multiple sitemaps in robots.txt	11 years ago