Commit Graph

373 Commits (8041742e48a3deb61ef0db171da107a23c8a3e63)

Author SHA1 Message Date
Michael Peter Christen 8041742e48 added matching of path to query pattern
12 years ago
Michael Peter Christen 570e42c4e3 fix for filetype naviagtor
12 years ago
Michael Peter Christen 71ed8e5e07 bugfixes for crawler
12 years ago
Michael Peter Christen 12c0db20e5 fixed npe for surrogate import
12 years ago
Michael Peter Christen 52df6ee369 more logging
12 years ago
Michael Peter Christen 158732af37 automatically delete entries from the crawl profile list if crawl is
12 years ago
Michael Peter Christen 15d1460b40 added information about the reason of pausing of crawls
12 years ago
Michael Peter Christen 2371ef031c added solr faceted search support to YaCy search results
12 years ago
Michael Peter Christen d481abd087 added the visualization of error-urls to host browser
12 years ago
Michael Peter Christen 791e1dcfdf when a new crawl is started, delete all entries about error-urls for
12 years ago
Michael Peter Christen 619bf7e875 fixed filetype modified for media types in text search
12 years ago
Michael Peter Christen 97f82994a6 automatically pause the crawler if there is a problem with solr
12 years ago
Michael Peter Christen 8fb370d9f8 renovated the way how search results are count. should be correct now...
12 years ago
orbiter 354ef8000d - added 'deleteold' option to crawler which causes that documents are
12 years ago
Michael Peter Christen 75dd706e1b update to HostBrowser:
12 years ago
Michael Peter Christen e2c4c3c7d3 migration to solr 4.0.0
12 years ago
Michael Peter Christen b764de424a code cleanup
12 years ago
Michael Peter Christen 9330ad4838 - fixed the delete option in host browser
12 years ago
Michael Peter Christen 1168d09de8 more refactoring - integrated the code of SnippetProcess into
12 years ago
Michael Peter Christen 6629e37685 tried to clean up the search process mess
12 years ago
Michael Peter Christen c5f67a5d6d fixed a problem with local search from solr results: now all results
12 years ago
Michael Peter Christen f8f05ecba7 - added a delete button in host browser to delete a complete subpath
12 years ago
Michael Peter Christen 4a14122ba7 in case that a crawl profile has a collection assigned, use the
12 years ago
Michael Peter Christen 0833937c1c better balancing and duetime-cumputation also for no-delay intranet
12 years ago
Michael Peter Christen c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain
12 years ago
Michael Peter Christen 6905182d41 - fix for number of words log message
12 years ago
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading
12 years ago
Michael Peter Christen 2d9e577ad0 replaced the custom robots.txt loader by the standard http loader
12 years ago
Michael Peter Christen 799d71bc67 enhanced solr caching:
12 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler
12 years ago
Michael Peter Christen 8e1248ffe3 force a commit in advance of a search for the administrator to get most
12 years ago
Michael Peter Christen 3b48c78190 added an option to force a commit to solr.
12 years ago
orbiter 276dd6452b removed warnings
12 years ago
sixcooler 47ae7e322e smaller dhtDispatcher.cloudSize
12 years ago
Michael Peter Christen ce0e5b1e17 - more refactoring / private methods
12 years ago
Michael Peter Christen ccc3760a47 Refactoring and redesign of data architecture to make URIMetadataRow
12 years ago
Michael Peter Christen e5b3c172ff removed hack which translated Solr documents to virtual RWI entries
12 years ago
Michael Peter Christen 5d16c23a1f specified more URIMetadata as URIMetadataNode
12 years ago
Michael Peter Christen 43f3345c90 - removed dependencies from URIMetadataRow and made direct access to
12 years ago
Michael Peter Christen cc98496ff3 enhanced the HostBrowser:
12 years ago
Michael Peter Christen 21fe8339b4 - enhanced generation of url objects
12 years ago
Michael Peter Christen 1b02408936 use less cache
12 years ago
Michael Peter Christen 36c13ed15b less solr prefetch
12 years ago
Michael Peter Christen 5f0ab25382 removed the option to prevent removal of & parts inside of the
12 years ago
Michael Peter Christen 7e3e45fd04 added Open Graph Metadata default fields, see http://ogp.me/ns#
12 years ago
Michael Peter Christen c3e5f667a7 added schema.org breadcrumb counter to parser and solr schema
12 years ago
Michael Peter Christen a06930662c replaced some more .getBytes() with UTF8/ASCII.getBytes()
12 years ago
Michael Peter Christen bd769de604 since the solr index is now used for all pages that are indexed locally,
12 years ago
Michael Peter Christen 4b5e0c1500 added an url rewriter which can be used to remove session ids from urls
12 years ago
Michael Peter Christen 76d218fbef fixes to crawl profiles
12 years ago