8f3bd0c387
fix for smb crawl situation (lost too many urls)
12 years ago
1e002ab18e
added another blacklist-cleaner into balancer
12 years ago
fa27e5820f
- check blacklist (again) when taking urls from the crawl stack because
...
the blacklist may get extended during crawling
- removed debug output
12 years ago
5e182a566f
- added another enumeration method in kelondro data structure to get a
...
more random access to data for the balancer
- added random access inside the balancer
12 years ago
75dd706e1b
update to HostBrowser:
...
- time-out after 3 seconds to speed up display (may be incomplete)
- showing also all links from the balancer queue in the host list (after
the '/') and in the result browser view with tag 'loading'
12 years ago
0fe8be7981
enhaced data structures for balancer and latency computation which
...
should produce a bit better prognosis about forced waiting times.
12 years ago
0833937c1c
better balancing and duetime-cumputation also for no-delay intranet
...
hosts
12 years ago
c326aa8f67
disabled writing new entries to crawl stacks to prevent that a domain
...
with many documents block refreshing of the crawl queue
12 years ago
c25d7bcb80
- added concurrency for robots.txt loading
...
- changed data model for domain counter
12 years ago
2d9e577ad0
replaced the custom robots.txt loader by the standard http loader
12 years ago
a33e2742cb
- removed unnecessary synchronized and deadlock in crawler
...
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings
12 years ago
8952153ecf
update to Balancer algorithm:
...
- create a load list from the current list of known hosts
- do not create this list for each Balancer.pop access
- create the list from those hosts which have a zero-waiting time
- select 1/3 from that list which have the most urls waiting
- get hosts from the wainting list in random order
- fixes for some delta-time computations
- always load all urls from hosts which have never been loaded before
12 years ago
1533bfd63b
refactoring
12 years ago
8219a445f3
refactoring
12 years ago
00c1c777fa
refactoring
12 years ago
Michael Peter Christen