Commit Graph

410 Commits (fe75f326d8db7db083313a560a7464b8016249d7)

Author SHA1 Message Date
Michael Peter Christen 0dda979801 adopted network image drawing to increased number of peers 11 years ago
Michael Peter Christen d9858e1b8a removed warnings and superfluous logging 11 years ago
Michael Peter Christen d2b8f2b477 enhancements for staticIP and ipv6 handling 11 years ago
orbiter 0002abd583 fix for OOM during remote search and too high load protection 11 years ago
sixcooler 5a917e13c6 use less ram on dht-URL transfer by not using a URIMetadataNode[] 11 years ago
sixcooler 4d77ca52c9 workaround to let dht-out run on smal Systems like a Pi 11 years ago
Michael Peter Christen be5e808236 - removed hardcoded load-test which is now handled in BusyQueues 11 years ago
Michael Peter Christen 1ea17bd9f3 - removed old metadata database and all migration code 11 years ago
reger 97e84439fb adjusted ConfigHeuristic and changed QueryGoal.getOriginalQueryString to .getQueryString 11 years ago
Michael Peter Christen 022c6d3ce1 do YaCy p2p connections using a timeout-request which covers the http 11 years ago
orbiter fd4abc0565 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 11 years ago
orbiter d5b8e473c8 added load limit for DHT transfer: RWI acceptance only if local load is 11 years ago
reger 2614fa7aeb Skip remote Solr search if last try showed error 11 years ago
orbiter a07e9b3582 concurrency-solid version of transmission limitation 11 years ago
orbiter 60ead31273 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 11 years ago
orbiter 52bf7d1ac8 reduce load during dht transfer 11 years ago
Michael Peter Christen 0bf3cab8c7 - better 'extra'-peer selection 11 years ago
Michael Peter Christen ba44eb1160 when scaling the number of remote peers, also consider the machine load 11 years ago
Michael Peter Christen f8ce7040ab remote search peer selection schema change: 11 years ago
Michael Peter Christen 47a82e471c less blocking in SeedDB which caused deadlocks in peer ping 11 years ago
reger 6932aa4d7a use configured admin-username for api calls 11 years ago
orbiter 3cb6c7861f fixed shutdown authenticaton problem 11 years ago
Michael Peter Christen 1c56befb93 fixed mess with test on localhost (which means local hosts for some 11 years ago
reger dd8ea0cdd6 fix "add to blacklist" button style in IndexControlRWIs_p 11 years ago
Michael Peter Christen 09412ea3a4 counting search requests in solr interface 11 years ago
Michael Peter Christen 79771c60c0 IPv6 fixes 11 years ago
Michael Peter Christen 9a27bf6e82 removed filter computation in Protocol class for remote searches because 11 years ago
Michael Peter Christen f1b5db2c45 - performance graph does not shop peer ping in memory monitor any more 11 years ago
Michael Peter Christen 2c39b65409 fixes for searches containing stopwords. The fix was done using a 11 years ago
orbiter 037cd0a57c using the BinaryResponseWriter which is supported within the YaCy solr 11 years ago
Michael Peter Christen ccf2f4e43b refactoring of seed attributes (introduced more constants) 11 years ago
orbiter b7f1e5af51 added new servlet which generates the same file as the principal peers 11 years ago
Michael Peter Christen e1c1e57877 less overhead calling exist() with only one hash 12 years ago
Michael Peter Christen 9bb7eab389 hacks to prevent storage of data longer than necessary during search and 12 years ago
orbiter d2effd21db fix for npe during location search 12 years ago
Michael Peter Christen 61c5e40687 - replaced the properties object in AnchorURL with distinct variables 12 years ago
Michael Peter Christen 5e31bad711 - the webgraph shall store all links which appear on a web page and not 12 years ago
Michael Peter Christen 049c3b3f2e added an option to exclude image search results from text search. This 12 years ago
Michael Peter Christen cb85b22725 redesign of the image search process (with much better results, 12 years ago
Michael Peter Christen 3c5abedabf NPE during shutdown fix 12 years ago
Michael Peter Christen 765943a4b7 Redesign of crawler identification and robots steering. A non-p2p user 12 years ago
Michael Peter Christen 47b1c81d08 - refactoring 12 years ago
sixcooler 8a96140f92 fix / workaround for 12 years ago
orbiter e24016e30a added the property federated.service.solr.indexing.timeout to yacy.init 12 years ago
Roland Haeder aaedc0405d Fixes and avoid of catching bad exceptions (some): 12 years ago
Roland Haeder 841a28ae76 Added 'final' for all exception blocks as this helps the Java compiler 12 years ago
Michael Peter Christen bcc623a843 refactoring of load_delay: this is a matter of client identification 12 years ago
Michael Peter Christen 5091d627bc fixed parsing of peer flags 12 years ago
Michael Peter Christen 5878c1d599 - refactoring of log to ConcurrentLog: 12 years ago
Michael Peter Christen 64140f35cd fix for solr requests if no query part is given (prevent npe) 12 years ago
Michael Peter Christen 8caaf6203a fixed false multiple-generation of remote facet search which 12 years ago
Michael Peter Christen 1762911f57 added synchronizations and timeouts in solr api; missing 12 years ago
Michael Peter Christen 164603b946 cleanup 12 years ago
Michael Peter Christen 409d6edf53 Store node/solr search threads to be able to send them an interrupt 12 years ago
Michael Peter Christen a1644ca0fd new workflow processor in Segment to enqueue indexing documents to solr 12 years ago
Michael Peter Christen 0c1a018bbd removed 'later' tactic because it used too much RAM, reduced number of 12 years ago
Michael Peter Christen 709e9b8ce7 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 12 years ago
Michael Peter Christen 1eb9626cca less logging 12 years ago
orbiter da621e827e prevent NPE in case RWI is disabled 12 years ago
Marc Nause 8fb1b1e290 *) simplified banner creation code 12 years ago
Michael Peter Christen 8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and 12 years ago
reger 97ab5b90e8 - odt & ooxml (office document) parser correction to add content to fulltext index 12 years ago
Michael Peter Christen 8dbc80da70 redesign of index.exist-test: this shall now not be done using a single 12 years ago
Michael Peter Christen 44e363f37f refactoring of WorkflowProcessor, added process counter, update of 12 years ago
Michael Peter Christen b9b446bca6 - added ssl configuration sign (a lock) to network statistic/table 12 years ago
Michael Peter Christen f7f3e28c5e prevent that the size of the index is computed too many times. 12 years ago
Michael Peter Christen ed1d5bace6 draw the names of other peers which receive/send dht into the network 12 years ago
Michael Peter Christen b528448332 enlarge network graph circle according to image height and reduce the 12 years ago
Michael Peter Christen bb4bf3d8fd infinity timeout bug protection patch 12 years ago
Michael Peter Christen 3502b4c697 refactoring (renaming) of yacy-solr api 12 years ago
orbiter e1bfe9d07a - reduction of the concurrently running processes to make YaCy more 12 years ago
Michael Peter Christen fc2095ac67 some extensions to raster plotter to transform a RGB picture to an 12 years ago
Michael Peter Christen c1a2175fbc added transparency to gif image animation and the integration to the 12 years ago
reger 6a9d0b60a3 make sure configured port is reported on recreated mySeed.txt 12 years ago
Michael Peter Christen 2d36a7eaf5 - do not create a new query for all remote peers 12 years ago
Michael Peter Christen addba047e2 changes in ranking computation 12 years ago
reger 38f46eb33d set RootNodeFlag only if EmbeddedSolr is connected (as RootNodes may receive direct Solr queries) 12 years ago
Michael Peter Christen 25300913fa fixes to search debugging after testing with the different search 12 years ago
orbiter b1140e3d82 added debug switches for detailed search testing 12 years ago
orbiter 2562f052b9 do not put the fulltext field text_t into the search cache because it is 12 years ago
Michael Peter Christen 221ed7d764 - enhanced concurrency during search without IO blocking 12 years ago
Michael Peter Christen 3b1d9dc884 made index storage from DHT search result concurrently. This prevents 12 years ago
orbiter f13c0b2abd fix for search 12 years ago
orbiter 9c09fd7d0b better/less requests to local solr; the request is made in chunks which 12 years ago
orbiter d74472f562 corrected result counter 12 years ago
Michael Peter Christen c95a84103a complete redesign of search process: 12 years ago
Michael Peter Christen 008288719c fix for schema export to consider also automatically generated 12 years ago
Michael Peter Christen 089dee1770 - generalized SchemaConfiguration into super-class Configuration and 12 years ago
Michael Peter Christen 14cceb6b17 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 12 years ago
reger f291d60c5f on remote Solr search take only locally enabled schema fields from remote solrdocument for the inputdocument added to local index 12 years ago
Michael Peter Christen 788288eb9e added the generation of 50 (!!) new solr field in the core 'webgraph'. 12 years ago
Michael Peter Christen 91a0401d59 introduced a second core named 'webgraph'. This core will hold the link 12 years ago
Michael Peter Christen b6de1f42dc Full redesign of solr connection architecture. This was done to support 12 years ago
Michael Peter Christen c34af7fe94 extended JSON Response Writer and Opensearch Response Writer for the 12 years ago
Michael Peter Christen e1da39245a when searching the network, do not search on robinson peers with the old 12 years ago
Michael Peter Christen 6f6ddaf7e7 A robinson peer does not need to write RWI data if such peers are only 12 years ago
Michael Peter Christen 7806680ab8 fixed a problem with re-feeding of already indexed documents whith 12 years ago
Michael Peter Christen 19c46e4acf catch more exceptions 12 years ago
Michael Peter Christen 4f270d89e2 another NPE 12 years ago
Michael Peter Christen e8f7b85b98 fixes to internal RWI usage if RWI is switched off (NPE etc) 12 years ago
Michael Peter Christen 4ca1b76627 less search overhead when first result set is smaller than requested 12 years ago
Michael Peter Christen 4589afe056 fix NPE when solr does not deliver snippets 12 years ago
Michael Peter Christen c3d50d91f8 relaxing site operator for www prefix: 12 years ago
reger d456f69381 SeedUpload url : check to reject localhost url included in saveSeedList (same check as in / copied from Seed.isProper() ), to prevent identity change on next startup (due to rejected seeduploadurl). 12 years ago
Michael Peter Christen 433143ba40 removed protocol, tld, ext from the urlmask and created specific 12 years ago
Michael Peter Christen 01200f06cc using the author field as solr-native facet. this makes it necessary to 12 years ago
Michael Peter Christen bab573361f - using a filter query for facet restriction 12 years ago
Michael Peter Christen 34f8786508 removed dependency of vocabulary navigation from Jena and it's 12 years ago
Michael Peter Christen 9319b90d8a - fixes for host navigation 12 years ago
Michael Peter Christen cb5cbec14d distinguishing modified query string and original query string 12 years ago
Michael Peter Christen d48e9788d2 enhanced search result processing behavior 12 years ago
orbiter 5dfd6359cb redesign of the QueryParams class: introduced QueryGoal which holds the 12 years ago
Michael Peter Christen d64445c3cb because we have the inurl:<term> - searchmodifier, we don't actually 12 years ago
Michael Peter Christen 93001586a0 removed warnings, removed too-fast pausing of crawls 13 years ago
Michael Peter Christen 2371ef031c added solr faceted search support to YaCy search results 13 years ago
Michael Peter Christen 8fb370d9f8 renovated the way how search results are count. should be correct now... 13 years ago
Michael Peter Christen 6629e37685 tried to clean up the search process mess 13 years ago
Michael Peter Christen c5f67a5d6d fixed a problem with local search from solr results: now all results 13 years ago
Michael Peter Christen f8f05ecba7 - added a delete button in host browser to delete a complete subpath 13 years ago
Michael Peter Christen 0fe8be7981 enhaced data structures for balancer and latency computation which 13 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler 13 years ago
sixcooler 2d972f289a rise commitWithinMs to default-value from SwitchBoard 13 years ago
Michael Peter Christen f2d0418218 because the new PngEncoder had a problem with the PixelGrabber which is 13 years ago
Michael Peter Christen d5d64019e5 - added a method for the RasterPlotter to draw arrow endings to lines 13 years ago
orbiter 276dd6452b removed warnings 13 years ago
Michael Peter Christen ae6feb5610 showing the web structure graph as animation in the crawl monitor 13 years ago
Michael Peter Christen 39317a6c66 enhanced webstructure image: introduced 13 years ago
sixcooler 57ddd63888 not hold a expensive cache of references for DHT-out,but but load them 13 years ago
Michael Peter Christen ccc3760a47 Refactoring and redesign of data architecture to make URIMetadataRow 13 years ago
Michael Peter Christen e5b3c172ff removed hack which translated Solr documents to virtual RWI entries 13 years ago
Michael Peter Christen 43f3345c90 - removed dependencies from URIMetadataRow and made direct access to 13 years ago
Michael Peter Christen 21fe8339b4 - enhanced generation of url objects 13 years ago
Michael Peter Christen 584663ae8c - redesign of solr query construction 13 years ago
Michael Peter Christen 016ffa7434 increased strength of crawling waves in network image 13 years ago
Michael Peter Christen 24d2ee3c52 - better date ranking 13 years ago
Michael Peter Christen 562183932b - removed ip_s from default profile since that needs a DNS lookup to 13 years ago
Michael Peter Christen c913b2ba77 - fix for NPEs during remote solr configuration 13 years ago
Michael Peter Christen 1533bfd63b refactoring 13 years ago
Michael Peter Christen 872f83ebe0 refactoring 13 years ago
Michael Peter Christen 5683162bd3 simplifications in DHT Distribution class and more documentation 13 years ago
Michael Peter Christen e57bf2ca39 simplified DHT classes 13 years ago
Michael Peter Christen 8219a445f3 refactoring 13 years ago
Michael Peter Christen 00c1c777fa refactoring 13 years ago
orbiter 563d584420 removed more dependencies in cora from kelondro 13 years ago
Michael Peter Christen f75b3f8a47 added more patches to work without RWI data structure 13 years ago
Michael Peter Christen e8acd542b5 - added faceted drill-down for host and geolocation to solr queries 13 years ago
Michael Peter Christen 4716546ef5 - reduced memory usage in index transmission using a transformation of 13 years ago
Michael Peter Christen 653645c1cf corrected solr query syntax 13 years ago
Michael Peter Christen f42a57cd7d gsa format update 13 years ago
Michael Peter Christen b3aad6cc35 bugfix for remote search when search is done to solr 13 years ago
Michael Peter Christen ff3eaa21b0 added remote search to solr on YaCy peers! 13 years ago
Michael Peter Christen a06123aec6 more abstraction and less parameter overhead for remote search 13 years ago
Michael Peter Christen f00733186b code simplifications 13 years ago
orbiter 404b0aab09 refactoring in remote search and stub for remote node peer selection 13 years ago
Michael Peter Christen 0cab06c47c refactoring 13 years ago
Michael Peter Christen 18f989dfb1 - refactoring (load -> getMetadata) 13 years ago
Michael Peter Christen a16206e38b more attempts to clean the index (cleaning is faster then) 13 years ago
Michael Peter Christen 703f427303 fixed some peer-ping connection details 13 years ago
Michael Peter Christen 597bb76e4f get the peer location more quickly 13 years ago
Michael Peter Christen b51df6c7e8 - added coordinate storage in solr schema 13 years ago
orbiter e816b88b55 changed behaviour of metadata storage: in case that any solr is 13 years ago
Michael Peter Christen f9c0e6e950 - Implemented and integrated the URIMetadataNode object which is a 13 years ago
orbiter 67edfd991c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 13 years ago
orbiter d9173ba7ed added more solr fields to integrate values from URIMetadataRow. All 13 years ago
Michael Peter Christen 24d9db1613 snippet retrieval loading processes may use a smaller minimum load time 13 years ago
Michael Peter Christen 1687737771 Abstraction of HandleMap and HandleSet 13 years ago
Michael Peter Christen 6f1ddb2519 Moved solr index-add method to the same method where the YaCy index is 13 years ago
Michael Peter Christen 1f41d9c6f5 bugfix for a NPE 13 years ago
Michael Peter Christen d3f243e2e1 fixed node type calculation for principal peers 13 years ago
orbiter 69e743d9e3 - more abstraction for the RWI index as preparation for solr integration 13 years ago
orbiter c00a3cf74d less usage of generic logger to avoid logger generation overhead 13 years ago
orbiter 0cbda0b2b8 - replaced all length() == 0 and size() == 0 with isEmpty() 13 years ago
orbiter 62202e2d71 refactoring of query attribute variable names for better consistency 13 years ago
Michael Peter Christen b0c408788b made class methods static where possible 13 years ago
Michael Peter Christen 0301aba1e9 removed unused method parameters 13 years ago
Michael Peter Christen 241dd8410a removed snippet pattern filter - it was not used 13 years ago
Michael Peter Christen d3964253ae - added @SuppressWarnings to unused servlet method parameters 13 years ago
Michael Peter Christen ea10766bfd cleaned unnecessary nested code 13 years ago
Michael Peter Christen 1481037820 replaced non-generic array with collection 13 years ago
Michael Peter Christen 613b45f604 - better data structures in secondary search 13 years ago
Michael Peter Christen 1825f165b8 better integration of blacklist according to use case 13 years ago
Michael Peter Christen 96aeb127e3 generalized localhost naming. 13 years ago
Michael Peter Christen 77f795756c fixing redirects and status codes: storing of status code in 13 years ago
Michael Peter Christen b9d42fd9c8 using com.google.common.io.Files instead of homebrew methods 13 years ago
Michael Peter Christen 3f55dc7c1e - added solr core and libraries that solr needs (lucene is missing, will 13 years ago
Michael Peter Christen 82a682b31d fixed problem with seed when switching network 13 years ago
Michael Peter Christen 8e97ada7c9 IPv6 bugfix 13 years ago
Roland 'Quix0r' Haeder edaa09b9b1 Rewrote all String blacklist types to enum 'BlacklistType', closes bug 13 years ago
Michael Peter Christen 3b992e6b00 using utf8 String compression in Webstructure database 13 years ago
Michael Peter Christen a1fe65b115 performance hacks 13 years ago
Michael Peter Christen e0d8643226 - performance hacks 13 years ago
Michael Peter Christen 7c1feefb28 introduced a default 10 second time-out in rwi normalization time 13 years ago
Michael Peter Christen 65d37e6a20 only ASCII needed in seed bitflags 13 years ago
Michael Peter Christen 0f82fb3628 using double instead float for a better release ordering 13 years ago
Michael Peter Christen 71c3163f3d - fixes to node identification 13 years ago
Michael Peter Christen ad222be7f8 added node state icon in network list 13 years ago
Michael Peter Christen 3c2bec681f added a root node flag: identifies peers with short ping time 13 years ago
Michael Peter Christen f294f2e295 bugfix to http://bugs.yacy.net/view.php?id=181 13 years ago
Michael Peter Christen 89142d1e8d removed (not all) warnings 13 years ago
Michael Peter Christen 15db703808 added missing serialization to remove all warnings 13 years ago