You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen 8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and ... reduced time-out of robots.txt load limit		12 years ago
..
RobotsTxt.java	reduced locking situation in crawler: shifted synchronized location and	12 years ago
RobotsTxtEntry.java	better balancing and duetime-cumputation also for no-delay intranet	12 years ago
RobotsTxtParser.java	fix for sitemap detection: the sitemap url was not visible if it	12 years ago