Commit Graph

23 Commits (0f3d8890db83b90e4dd869fef7f00a47973e93c1)

Author SHA1 Message Date
orbiter 268a36aaff emergency fix for crawler: this will otherwise cause loss of complete
11 years ago
Michael Peter Christen 735a66eff3 enhancements to crawler
11 years ago
Roland Haeder 841a28ae76 Added 'final' for all exception blocks as this helps the Java compiler
11 years ago
Michael Peter Christen c6a6f159e8 fix for crawl stack domain counter
12 years ago
Michael Peter Christen 5878c1d599 - refactoring of log to ConcurrentLog:
12 years ago
Michael Peter Christen fa08bd9d5a hack to prevent long waiting times in crawler
12 years ago
Michael Peter Christen 8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and
12 years ago
Michael Peter Christen bb4bf3d8fd infinity timeout bug protection patch
12 years ago
Michael Peter Christen 8f3bd0c387 fix for smb crawl situation (lost too many urls)
12 years ago
Michael Peter Christen 1e002ab18e added another blacklist-cleaner into balancer
12 years ago
Michael Peter Christen fa27e5820f - check blacklist (again) when taking urls from the crawl stack because
12 years ago
Michael Peter Christen 5e182a566f - added another enumeration method in kelondro data structure to get a
12 years ago
Michael Peter Christen 75dd706e1b update to HostBrowser:
12 years ago
Michael Peter Christen 0fe8be7981 enhaced data structures for balancer and latency computation which
12 years ago
Michael Peter Christen 0833937c1c better balancing and duetime-cumputation also for no-delay intranet
12 years ago
Michael Peter Christen c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain
12 years ago
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading
12 years ago
Michael Peter Christen 2d9e577ad0 replaced the custom robots.txt loader by the standard http loader
12 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler
12 years ago
orbiter 8952153ecf update to Balancer algorithm:
12 years ago
Michael Peter Christen 1533bfd63b refactoring
12 years ago
Michael Peter Christen 8219a445f3 refactoring
12 years ago
Michael Peter Christen 00c1c777fa refactoring
12 years ago