Commit Graph

79 Commits (b85db72a73da4797d09dc72155c56ca00dd5da0f)

Author SHA1 Message Date
Michael Peter Christen 75dd706e1b update to HostBrowser:
12 years ago
Michael Peter Christen 0716a24737 added more / all new crawl profile fields into crawl profile editor
12 years ago
Michael Peter Christen 4a14122ba7 in case that a crawl profile has a collection assigned, use the
12 years ago
Michael Peter Christen 0fe8be7981 enhaced data structures for balancer and latency computation which
12 years ago
Michael Peter Christen ac9540dfb6 removed options for stopwords which are not used
12 years ago
Michael Peter Christen b2ffd49817 less latency
12 years ago
Michael Peter Christen 0833937c1c better balancing and duetime-cumputation also for no-delay intranet
12 years ago
Michael Peter Christen c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain
12 years ago
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading
12 years ago
Michael Peter Christen a87811bc38 more auto-commit calls when a search interface is opened, but not when a
12 years ago
Michael Peter Christen 2d9e577ad0 replaced the custom robots.txt loader by the standard http loader
12 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler
12 years ago
orbiter 8952153ecf update to Balancer algorithm:
12 years ago
Michael Peter Christen 85ca07b90e when a new crawl is started, an equal crawl, if still running, is
12 years ago
Michael Peter Christen ae6feb5610 showing the web structure graph as animation in the crawl monitor
12 years ago
Michael Peter Christen ccc3760a47 Refactoring and redesign of data architecture to make URIMetadataRow
12 years ago
Michael Peter Christen e5b3c172ff removed hack which translated Solr documents to virtual RWI entries
12 years ago
Michael Peter Christen 43f3345c90 - removed dependencies from URIMetadataRow and made direct access to
12 years ago
Michael Peter Christen 21fe8339b4 - enhanced generation of url objects
12 years ago
Michael Peter Christen 5f0ab25382 removed the option to prevent removal of & parts inside of the
12 years ago
Michael Peter Christen 53789555b9 fix for crawl start filter
12 years ago
Michael Peter Christen a06930662c replaced some more .getBytes() with UTF8/ASCII.getBytes()
12 years ago
Michael Peter Christen 4b5e0c1500 added an url rewriter which can be used to remove session ids from urls
12 years ago
Michael Peter Christen 76d218fbef fixes to crawl profiles
12 years ago
Michael Peter Christen 1533bfd63b refactoring
12 years ago
Michael Peter Christen 872f83ebe0 refactoring
12 years ago
Michael Peter Christen 8219a445f3 refactoring
12 years ago
Michael Peter Christen f879a344e7 fix for no depth limit default value
12 years ago
Michael Peter Christen 00c1c777fa refactoring
12 years ago