Commit Graph

13 Commits (10527e28ae488da3beaa95c122f4402b1a023448)

Author SHA1 Message Date
Michael Peter Christen fa27e5820f - check blacklist (again) when taking urls from the crawl stack because
12 years ago
Michael Peter Christen 5e182a566f - added another enumeration method in kelondro data structure to get a
12 years ago
Michael Peter Christen 75dd706e1b update to HostBrowser:
12 years ago
Michael Peter Christen 0fe8be7981 enhaced data structures for balancer and latency computation which
12 years ago
Michael Peter Christen 0833937c1c better balancing and duetime-cumputation also for no-delay intranet
12 years ago
Michael Peter Christen c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain
12 years ago
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading
12 years ago
Michael Peter Christen 2d9e577ad0 replaced the custom robots.txt loader by the standard http loader
12 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler
12 years ago
orbiter 8952153ecf update to Balancer algorithm:
12 years ago
Michael Peter Christen 1533bfd63b refactoring
13 years ago
Michael Peter Christen 8219a445f3 refactoring
13 years ago
Michael Peter Christen 00c1c777fa refactoring
13 years ago