3 Commits (5c7ddc67feb204af4d2aa6397423375ae7550d54)

Author	SHA1	Message	Date
Michael Peter Christen	038f956821	fix for sitemap detection: the sitemap url was not visible if it ... appeared after the declaration of robots allow/deny for the crawler because the sitemap parser terminated after the allow/deny rules had been found. Now the parser reads the robots.txt until the end to discover also sitemap rules at the end of the file.	12 years ago
Michael Peter Christen	2d9e577ad0	replaced the custom robots.txt loader by the standard http loader	12 years ago
Michael Peter Christen	00c1c777fa	refactoring	13 years ago