5 Commits (b2ffd498170e65a8e92efe21fae0c40fa5504370)

Author	SHA1	Message	Date
Michael Peter Christen	c25d7bcb80	- added concurrency for robots.txt loading ... - changed data model for domain counter	13 years ago
Michael Peter Christen	2d9e577ad0	replaced the custom robots.txt loader by the standard http loader	13 years ago
Michael Peter Christen	a33e2742cb	- removed unnecessary synchronized and deadlock in crawler ... - removed problem with monitoring object on Balancer.wait - added missing user agent settings	13 years ago
Michael Peter Christen	5f0ab25382	removed the option to prevent removal of & parts inside of the ... MultiProtocolURI during normalform computation because that should always be done and also be done during initialization of the MultiProtocolURI Object. The new normalform method takes only one argument which should be 'true' unless you know exactly what you are doing.	13 years ago
Michael Peter Christen	00c1c777fa	refactoring	13 years ago