Commit Graph

63 Commits (89ede0fe847c5010cd563dff97285cf40e5d5186)

Author SHA1 Message Date
Michael Peter Christen 91a0401d59 introduced a second core named 'webgraph'. This core will hold the link 12 years ago
Michael Peter Christen b6de1f42dc Full redesign of solr connection architecture. This was done to support 12 years ago
Michael Peter Christen 0b6566a389 optimizations when starting large crawl requests with many start urls in 12 years ago
Michael Peter Christen 0fe7b6fd3b migrated the index export methods from the old metadata to solr. Now 12 years ago
Michael Peter Christen 9ccdd21d76 Merge remote-tracking branch 'aleksejs/fixtrans' 12 years ago
Michael Peter Christen af465cdca5 fix for wrong robots.txt loading for https protocol 12 years ago
Michael Peter Christen 38d3feae65 added separate delete commands for the local+remote solr index, the old 12 years ago
reger 276e63401e small sanitary fixes 12 years ago
Michael Peter Christen 8f3bd0c387 fix for smb crawl situation (lost too many urls) 12 years ago
Michael Peter Christen 7c3de8b4cd - fix for localhost detection 12 years ago
orbiter 712cc37c40 if maxFileSize < 0 then the file size limit is without limit. 12 years ago
Michael Peter Christen a3cd3852ab introduced a better place to update the lastacc time value in latency 12 years ago
Michael Peter Christen 864abcd33d removed Latency update after URL selection because that causes 12 years ago
Michael Peter Christen dd241d03bb latency fix: only set last-visit time if access was actually by the 12 years ago
Michael Peter Christen 1e002ab18e added another blacklist-cleaner into balancer 12 years ago
Michael Peter Christen 10527e28ae fix for wrong display of error urls in HostBrowser 12 years ago
Michael Peter Christen 756772fbd3 fix for waitingtime computation for intranet configuration 12 years ago
Michael Peter Christen fa27e5820f - check blacklist (again) when taking urls from the crawl stack because 12 years ago
Michael Peter Christen 72f165d58b added a Boost class which stores solr query boost values. The class can 12 years ago
Michael Peter Christen 3de784c8dd replaced more split and replaceAll missing pattern pre-compilation with 12 years ago
Michael Peter Christen d48e9788d2 enhanced search result processing behavior 12 years ago
Michael Peter Christen eca68fa197 added debug code to crawler monitor 12 years ago
orbiter 5aa5202adf fixes for filesystem indexing 12 years ago
Michael Peter Christen efd2c4622d added a new fail type attribute for the index to distinguish two 12 years ago
Michael Peter Christen 5e182a566f - added another enumeration method in kelondro data structure to get a 12 years ago
Michael Peter Christen d6b82840f8 added a feature to find similarities in documents. 12 years ago
Michael Peter Christen f5ca5cea44 - added field options to all solr queries. This can be used to restrict 12 years ago
cominch d2a94cc55e refactor package 13 years ago
Michael Peter Christen 8b1c9cba3d fixed a problem with non-terminating crawls 13 years ago
Michael Peter Christen 71ed8e5e07 bugfixes for crawler 13 years ago
Michael Peter Christen 158732af37 automatically delete entries from the crawl profile list if crawl is 13 years ago
Michael Peter Christen d481abd087 added the visualization of error-urls to host browser 13 years ago
Michael Peter Christen 791e1dcfdf when a new crawl is started, delete all entries about error-urls for 13 years ago
orbiter 354ef8000d - added 'deleteold' option to crawler which causes that documents are 13 years ago
Michael Peter Christen 75dd706e1b update to HostBrowser: 13 years ago
Michael Peter Christen 0716a24737 added more / all new crawl profile fields into crawl profile editor 13 years ago
Michael Peter Christen 4a14122ba7 in case that a crawl profile has a collection assigned, use the 13 years ago
Michael Peter Christen 0fe8be7981 enhaced data structures for balancer and latency computation which 13 years ago
Michael Peter Christen ac9540dfb6 removed options for stopwords which are not used 13 years ago
Michael Peter Christen b2ffd49817 less latency 13 years ago
Michael Peter Christen 0833937c1c better balancing and duetime-cumputation also for no-delay intranet 13 years ago
Michael Peter Christen c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain 13 years ago
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading 13 years ago
Michael Peter Christen a87811bc38 more auto-commit calls when a search interface is opened, but not when a 13 years ago
Michael Peter Christen 2d9e577ad0 replaced the custom robots.txt loader by the standard http loader 13 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler 13 years ago
orbiter 8952153ecf update to Balancer algorithm: 13 years ago
Michael Peter Christen 85ca07b90e when a new crawl is started, an equal crawl, if still running, is 13 years ago
Michael Peter Christen ae6feb5610 showing the web structure graph as animation in the crawl monitor 13 years ago
Michael Peter Christen ccc3760a47 Refactoring and redesign of data architecture to make URIMetadataRow 13 years ago