c25d7bcb80
- added concurrency for robots.txt loading
...
- changed data model for domain counter
12 years ago
85ca07b90e
when a new crawl is started, an equal crawl, if still running, is
...
terminated and the corresponding crawl profile is deleted (this also
clears the crawl queue entries for that crawl profile)
12 years ago
5f0ab25382
removed the option to prevent removal of & parts inside of the
...
MultiProtocolURI during normalform computation because that should
always be done and also be done during initialization of the
MultiProtocolURI Object. The new normalform method takes only one
argument which should be 'true' unless you know exactly what you are
doing.
12 years ago
53789555b9
fix for crawl start filter
12 years ago
76d218fbef
fixes to crawl profiles
12 years ago
1533bfd63b
refactoring
13 years ago
8219a445f3
refactoring
13 years ago
f879a344e7
fix for no depth limit default value
13 years ago
00c1c777fa
refactoring
13 years ago
Michael Peter Christen