You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen 038f956821 fix for sitemap detection: the sitemap url was not visible if it ... appeared after the declaration of robots allow/deny for the crawler because the sitemap parser terminated after the allow/deny rules had been found. Now the parser reads the robots.txt until the end to discover also sitemap rules at the end of the file.		12 years ago
..
RobotsTxt.java	fix for wrong robots.txt loading for https protocol	12 years ago
RobotsTxtEntry.java	better balancing and duetime-cumputation also for no-delay intranet	12 years ago
RobotsTxtParser.java	fix for sitemap detection: the sitemap url was not visible if it	12 years ago