You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen 8f3bd0c387 fix for smb crawl situation (lost too many urls)		12 years ago
..
RobotsTxt.java	fix for smb crawl situation (lost too many urls)	12 years ago
RobotsTxtEntry.java	better balancing and duetime-cumputation also for no-delay intranet	12 years ago
RobotsTxtParser.java	replaced the custom robots.txt loader by the standard http loader	12 years ago