Commit Graph

110 Commits (6dad227bdf0bf7973faae7954faea86a26dec7ee)

Author SHA1 Message Date
orbiter 05c26d58d9 fixed missing remove operation in balancer
17 years ago
orbiter 606b323a2d fixed bug that appeared when a new crawl ist started
17 years ago
orbiter 28d5703f8a - fixed a bug in Robots.txt loader which could have caused that robots.txt files had been loaded from the same domain more than once
17 years ago
orbiter a6719dfd2b - refactoring of robots parser
17 years ago
orbiter 474e29ce4a added options to configure the 'corporate identity'-icons, the home page link and the greeting line from
17 years ago
orbiter 474659a71f - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order
17 years ago
orbiter b928ae492a some code-cleanup and possible speed enhancements in different core methods
17 years ago
danielr 7feae906aa - organize imports
17 years ago
orbiter dd75b3cabc - patch for bad profiles
17 years ago
orbiter 1689030ee8 refactoring: moved all crawler classes into their own package
17 years ago