8 Commits (dbfa8657003b96610afe09d9cc9981c02b64901d)

Author	SHA1	Message	Date
Michael Peter Christen	57ffdfad4c	added a crawl option to obey html-meta-robots-noindex. This is on by ... default.	12 years ago
Michael Peter Christen	25499eead5	- added a new field for the regular expression in crawl start ... - added the field in crawl profile - adopted logging end error management - adopted duplicate document detection - added a new rule to the indexing process to reject non-matching content - full redesign of the expert crawl start servlet The new filter field can now be seen in /CrawlStartExpert_p.html at Section "Document Filter", subsection item "Filter on Content of Document"	12 years ago
Michael Peter Christen	0fe8be7981	enhaced data structures for balancer and latency computation which ... should produce a bit better prognosis about forced waiting times.	12 years ago
Michael Peter Christen	ac9540dfb6	removed options for stopwords which are not used	12 years ago
Michael Peter Christen	5f0ab25382	removed the option to prevent removal of & parts inside of the ... MultiProtocolURI during normalform computation because that should always be done and also be done during initialization of the MultiProtocolURI Object. The new normalform method takes only one argument which should be 'true' unless you know exactly what you are doing.	13 years ago
Michael Peter Christen	2f536cb54d	code cleanup: removed unised methods and made more methods and objects ... private	13 years ago
Michael Peter Christen	1533bfd63b	refactoring	13 years ago
Michael Peter Christen	00c1c777fa	refactoring	13 years ago