2103 Commits (ab9583d42976822bcb15d313891defe5a81ef25e)

Author	SHA1	Message	Date
Michael Peter Christen	a4214694df	We assert that no other metadata storage than solr is used now. ... Therefore a property like solrConnected() must be true all the time. Removal of this method causes removal of all write operations to the old metadata index.	13 years ago
Michael Peter Christen	0cec7e761a	enhanced snippet extractor to find snippets also inside of tokens of an ... url	13 years ago
sixcooler	6c50d016ed	pdf- and zipParser should not use forced Memory-Limits	13 years ago
Michael Peter Christen	562183932b	- removed ip_s from default profile since that needs a DNS lookup to ... create an document entry. This makes remote search much slower. - removed synchronization of add method if ip_s is activated to prevent that a user configuration causes bad behavior. The disadvantage of that is, that a index dump can cause data loss if an indexing is running during index dump - catched more exceptions and more NPE - better abstraction in MirrorSolrConnector - slight performance enhancement when only the index count is requested (rows=0 is sufficient to get a total count)	13 years ago
Michael Peter Christen	24f4ca4d85	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
apfelmaennchen	116f429e35	fix for java.lang.RuntimeException: TableColumnIndex not available...	13 years ago
Michael Peter Christen	5ac61591f3	better abstraction for solr query params	13 years ago
Michael Peter Christen	c913b2ba77	- fix for NPEs during remote solr configuration ... - fixed remote solr setting switch - added more logging	13 years ago
Michael Peter Christen	1533bfd63b	refactoring	13 years ago
Michael Peter Christen	e49359cc95	removed tenant query attribute since it is not used any more and is ... replaced by the site-operator in the GSA interface. This operator can also be simulated in the Solr interface using the collections_sxt field.	13 years ago
Michael Peter Christen	872f83ebe0	refactoring	13 years ago
Michael Peter Christen	fb9460f0a8	using the search filter to drill down search to file types. ... A search like "mp3 filetype:mp3" will now maybe surprise you.	13 years ago
Michael Peter Christen	15ea053c3a	- added xml output in IndexControlURLs to get the storage page of index ... dump commands - adjusted the apicall.sh script to get the downloaded text as output to stdout which is necessary to parse the content out of it - added indexdump.sh script which creates a solr dump and prints out the storage path for the index dump - added synchronization to the Fulltext class to prevent that data is stored to a non-existing solr index while this index is disabled during the storage of the dump	13 years ago
Michael Peter Christen	1b474139dd	used the new zip writer/reader to add a solr dump process: the whole ... solr index can be written to a zip dump and also restored during runtime	13 years ago
Michael Peter Christen	4a3e684f8c	added a directory-to-zip writer and zip-to-directory reader	13 years ago
Michael Peter Christen	d9ebf4a40f	a bit more logging	13 years ago
Michael Peter Christen	5683162bd3	simplifications in DHT Distribution class and more documentation	13 years ago
Michael Peter Christen	e57bf2ca39	simplified DHT classes	13 years ago
orbiter	a053b356ee	added new classes to renovate the YaCy protocol based on simple data ... structures in cora: - added the Peer object, which is a fresh version of Seed - added the Peers object, which is a fresh version of Network - added the Network api access class to retrieve a list of peers based on the Network.xml servlet in all YaCy peers.	13 years ago
Michael Peter Christen	8219a445f3	refactoring	13 years ago
Michael Peter Christen	f879a344e7	fix for no depth limit default value	13 years ago
Michael Peter Christen	00c1c777fa	refactoring	13 years ago
orbiter	563d584420	removed more dependencies in cora from kelondro	13 years ago
orbiter	aa65282259	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	63762d8f89	removed kelondro dependencies from cora	13 years ago
orbiter	6e0f4557f8	added ftp to getName	13 years ago
cominch	23204d2245	change parameter to support the smw extension for list import	13 years ago
Michael Peter Christen	c235d5c0f1	fixed size parsing in RSS message parser (for YaCy size parameter)	13 years ago
Michael Peter Christen	5bc8f34150	fix for success query counter	13 years ago
orbiter	60b1e23f05	added new crawl options: ... - indexUrlMustMatch and indexUrlMustNotMatch which can be used to select loaded pages for indexing. Default patterns are in such a way that all loaded pages are also indexed (as before) but when doing an expert crawl start, then the user may select only specific urls to be indexed. - crawlerNoDepthLimitMatch is a new pattern that can be used to remove the crawl depth limitation. This filter a never-match by default (which causes that the depth is used) but the user can select paths which will be loaded completely even if a crawl depth is reached.	13 years ago
orbiter	4987921d3d	fixed the size() method which counted also failed pages (which are also ... inside the solr index)	13 years ago
Michael Peter Christen	6ec02deec6	added new crawl attributes in crawl profile (not active yet)	13 years ago
Michael Peter Christen	975bc95ddf	added default facet fields for json response format (stub)	13 years ago
Michael Peter Christen	0504b01bdc	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	9413f77b65	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	a55e77a115	added twitter search heuristic	13 years ago
Michael Peter Christen	e54ac38095	- some corrections in usage of getFile() and getFileName() ... - added more attributes in json response writer according to yacy servlet	13 years ago
Michael Peter Christen	62add1d564	added the protocol and the file name extension to the solr fields since ... these fields are probably facets in file search	13 years ago
Michael Peter Christen	e072632a54	no complaints about memory if the database is empty	13 years ago
Michael Peter Christen	b846f585fa	fixed a bug with size_i field usage	13 years ago
Michael Peter Christen	9db032664e	activate two solr fields which will be used by administration interface ... (later)	13 years ago
orbiter	fcd5c7eec3	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	6171143b4a	added facet stub in JsonResponseWriter	13 years ago
Michael Peter Christen	e84ffdb4f3	enhanced solr writers	13 years ago
Michael Peter Christen	5df553c152	- added a json writer for solr (yes there was one using xslt but this ... one writes the same way as yacysearch.json) - using the new json solr result to change the ajax search in IndexControlURLs to the new solr search	13 years ago
Michael Peter Christen	4634f0e626	fix for images_withalt	13 years ago
Michael Peter Christen	e65cecc419	- updated lucene libraries to 3.6.1 ... - added lucene-grouping which enables faceted search; try this: http://localhost:8090/solr/select?q=*:*&start=0&rows=3&facet=true&facet.field=host_s	13 years ago
Michael Peter Christen	4d29f59a27	removed warnings	13 years ago
Michael Peter Christen	8c099d2106	Merge remote-tracking branch 'origin/master' ... Conflicts: htroot/api/ymarks/import_ymark.java source/de/anomic/data/ymark/YMarkEntry.java source/de/anomic/data/ymark/YMarkTables.java	13 years ago
apfelmaennchen	d31a632951	- added dmoz RDF dump importer ... - added indexing to Tables columns to support larger bookmark collections - added RDF output (HTTP) for public bookmarks at /YMarks.rdf - YMarkRDF also provides a Jena RDF Model as "internal" API - various other changes/fixes for YMarks (mainly backend)	13 years ago
Michael Peter Christen	10b911eed4	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	be67c70a47	added Solr fields: ... inboundlinks_text_chars_val inboundlinks_text_words_val inboundlinks_alttag_txt outboundlinks_text_chars_val outboundlinks_text_words_val outboundlinks_alttag_txt	13 years ago
orbiter	d73fff0e0e	added solr field images_withalt_i	13 years ago
sixcooler	e78fe3f477	also do a clearcache on the solr-connector-caches	13 years ago
sixcooler	9ee2e09983	statistics for solr-cache	13 years ago
Michael Peter Christen	d8425e6809	added collections to crawl monitor	13 years ago
Michael Peter Christen	ee23fc7a32	added h1..h6 counter fields	13 years ago
Michael Peter Christen	b2b516cc3e	added a collection attribute to crawls and searches: ... - a solr field collection_sxt can be used to store a set of crawl tags - when this field is activated, a crawl tag can be assigned when crawls are started - the content of the collection field can be comma-separated, all of them are assigned to the documents when they are indexed as result of such a crawl start - a search result can be drilled down to a specific collection; this is currently only available in the solr interface and also in the gsa interface using the 'site' option - this adds a mandatory field for gsa queries (the google api demands that field all the time)	13 years ago
Michael Peter Christen	4815713ec7	added synchronization to solr server requests since lucene is not ... thread-safe. We experienced problems as described in http://stackoverflow.com/questions/5327978/lockobtainfailedexception-updating-lucene-search-index-using-solr	13 years ago
Michael Peter Christen	f75b3f8a47	added more patches to work without RWI data structure	13 years ago
Michael Peter Christen	a427a68bac	removed many warnings	13 years ago
Michael Peter Christen	c72c435517	- moved the gsa search interface from /gsa/searchresult? to /gsa/search? ... - fixed the NB field data	13 years ago
Michael Peter Christen	31d4d38804	- extended the solr interface by a references-by-word-count method ... - reduced danger that a non-existing RWI database causes NPEs - added Solr queries to did-you-mean: this makes it possible that our did-you-mean algorithm works together with only Solr and without RWIs	13 years ago
Michael Peter Christen	528d6763fa	- added new solr fields: ... title_count_i, title_chars_val, title_words_val description_count_i, description_chars_val, description_words_val - added many asserts to ensure data type correctness from YaCy to Solr and vice versa - made many fixes according to new findings from these asserts (!)	13 years ago
Michael Peter Christen	3142e675e8	fixed problems with GSA api: ... - better FS attribute - highlightning of searched words in title	13 years ago
Michael Peter Christen	3b19fe7b52	- fixed num parameter in GSA api ... - changed FS attribute in GSA api	13 years ago
Michael Peter Christen	2ddc33646a	added new field for solr: ... url_paths_sxt url_parameter_i url_parameter_key_sxt url_parameter_value_sxt url_chars_i	13 years ago
Michael Peter Christen	75d5e3475d	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
cominch	dc468dad01	add content control features for custom filter lists	13 years ago
Michael Peter Christen	316b5fe116	- added a solr type definition verifier ... - fixed type definition found by the verifier - added multivalue-string fields for solr with extension 'sxt' - added multivalue-integer fields for solr with extension 'val' - renamed some solr attributes from txt to sxt - changed solr query line to an explicit AND/OR structure - added a country code second level domain list to Domains class; with parser - added a host string parser to get domain class name, country-code second-level domain and subdomain out of it - removed old coordinate attributes	13 years ago
orbiter	a3d5959981	Merge commit '65d49df865f60511d22d86fb15c33a082176e7ab'	13 years ago
Michael Peter Christen	4521d63c92	added boosts to solr search queries	13 years ago
Michael Peter Christen	e8acd542b5	- added faceted drill-down for host and geolocation to solr queries ... - added a new geolocation field to index schema, the old values are migrated if possible	13 years ago
Michael Peter Christen	f00168ecc5	added gsa result attribute 'has'	13 years ago
reger	65d49df865	security fix: clear automtic password only if adminAccountForLocalhost=false to prevent remote access to protected pages after restart. ... if adminAccountForLocalhost=true leave automatic password unchanged so access from local host is granted but remote access is preventet from the 1st second.	13 years ago
orbiter	2094df2e4e	- correct length computation for BStringObject (bugfix suggested by ... apfelmaennchen) - using ASCII for string conversion for Strings generated from Integer	13 years ago
orbiter	67f2866cd0	small fixes	13 years ago
orbiter	ce156a01ba	Merge commit 'c2341a175fdd755a34965ff63c7ea437b380352d'	13 years ago
David Rubio	c2341a175f	Fixed a bug that prevented Yacy from indexing files with non ASCII filenames in FTP servers. ... Previously Yacy could read file listings in UTF-8, but couldn't send commands to the FTP server in UTF-8 (the second byte of every multi-byte character was ignored), which caused a lot of errors on the server side. Now it handles UTF-8 correctly.	13 years ago
orbiter	3ebc4264c5	fixed concurrent query	13 years ago
orbiter	29171e2f6c	fixed generation of ontologies from index enumerations	13 years ago
orbiter	7cd302de3e	omit xml parsing when using the embedded solr server	13 years ago
orbiter	787e1c6836	added the ... QueryResponse query(SolrParams params) method to the SolrServerConnector which is necessary to use facets in solr search.	13 years ago
orbiter	01a63ef595	redesign of YaCySchema and SolrDoc handling	13 years ago
orbiter	479bfca571	refctoring	13 years ago
Michael Peter Christen	48a82bc705	log queries anonymous from gsa+solr requests	13 years ago
Michael Peter Christen	ab6ec4ec52	added snippet computation to solr/rss and gsa result writer	13 years ago
Michael Peter Christen	4716546ef5	- reduced memory usage in index transmission using a transformation of ... Node to Row objects - removed peerDeparture in solr remote search in case that peer does not answer (this may be normal because it is allowed to switch this off)	13 years ago
Michael Peter Christen	06b0081fdc	fix for NPE during host navigation computation	13 years ago
Michael Peter Christen	feb99bc291	fixed GSA format	13 years ago
Michael Peter Christen	653645c1cf	corrected solr query syntax	13 years ago
Michael Peter Christen	08ae142a3d	- enhanced caching after search queries to solr ... - reduced caching after short memory	13 years ago
orbiter	716ea0cfe2	sorted the solr schema into mandatory and optional fields; reduced ... number of used field to reduce solr index size	13 years ago
orbiter	9b8c8c0f47	fix from gaston in ... http://forum.yacy-websuche.de/viewtopic.php?p=26909#p26909	13 years ago
orbiter	acb9f04e80	removed unused classes	13 years ago
Michael Peter Christen	0ad52ac4c3	gsa bugfix for date parser	13 years ago
Michael Peter Christen	3ce4c2f937	fixes for gsa result format	13 years ago
Michael Peter Christen	67d235fae9	added gzip encoding to solr2sor http interface, client side (server ... already works)	13 years ago
Michael Peter Christen	a049761e0c	fixed double-check	13 years ago
Michael Peter Christen	f42a57cd7d	gsa format update	13 years ago
Michael Peter Christen	b3aad6cc35	bugfix for remote search when search is done to solr	13 years ago
Michael Peter Christen	ff3eaa21b0	added remote search to solr on YaCy peers! ... - when doing a remote search, node peers are selected for solr queries - the solr query is done concurrently to the standard YaCy rwi search - the solr search result is feeded into the same data structure that prepares the rwi search result - the same remote seach that is done to several outside peers is done to the local solr index - the search process works now also without any 'old' RWI data using solr	13 years ago
Michael Peter Christen	a06123aec6	more abstraction and less parameter overhead for remote search	13 years ago
Michael Peter Christen	f00733186b	code simplifications	13 years ago
Michael Peter Christen	755f5e76cf	removed strange assert statements and simplified code in metadata ... transformation	13 years ago
Michael Peter Christen	db0d438709	fix for http://bugs.yacy.net/view.php?id=206	13 years ago
orbiter	404b0aab09	refactoring in remote search and stub for remote node peer selection	13 years ago
orbiter	d7ea45f698	- get nice text_t values from metadata conversions that are stored into ... solr as fulltext search index. - added slow migration from old metadata to solr index entries: each entry from the old metadata is removed from that data structure and written into solr.	13 years ago
orbiter	99ef57f103	reduced sleep times	13 years ago
orbiter	780f8974e7	added ramaining iteration methods for solr in fulltext class	13 years ago
orbiter	acd2dc3575	hack to removed StringBuilder overhead in query construction	13 years ago
orbiter	ee01c12e56	fixes for putDocument and putMetadata	13 years ago
orbiter	cc47a0876e	reverted bf55f69176 ... to have a fall-back option in case that memory problems as reported in http://forum.yacy-websuche.de/viewtopic.php?p=26901#p26901 for full-solr installation are too strong and we have to work with an 'small memory footprint' peer system.	13 years ago
Michael Peter Christen	0904afe8fb	added concurrent iterator methods to the solr connectors	13 years ago
Michael Peter Christen	d54b80327a	refactoring	13 years ago
Michael Peter Christen	f9fc5cfaba	better check for bad urls in url transmission	13 years ago
Michael Peter Christen	d39463a85c	added deleteByQuery to solr connectors	13 years ago
Michael Peter Christen	0cab06c47c	refactoring	13 years ago
Michael Peter Christen	bf55f69176	removed write methods to old metadata file type; all metadata now goes ... to solr	13 years ago
Michael Peter Christen	40c0856489	refactoring	13 years ago
Michael Peter Christen	06a78eecb7	code simplification	13 years ago
Michael Peter Christen	54bea21c02	bugfix for solr connector, possibly a cause for ... http://forum.yacy-websuche.de/viewtopic.php?p=26893#p26893	13 years ago
Michael Peter Christen	9bece5ac5f	enhanced snippet fetch - removed a bug that caused documents to be ... parsed even if a solr text was available	13 years ago
Michael Peter Christen	18f989dfb1	- refactoring (load -> getMetadata) ... - added getDocument to retrieve Solr documents which shall replace getMetadata	13 years ago
Michael Peter Christen	395b78a0d8	using the solr search index to concurrently search within solr and the ... rwis during local search requests.	13 years ago
Michael Peter Christen	6197caf698	added clear-text search words in query params	13 years ago
Michael Peter Christen	23226676c6	FOR THE BRAVE.. this is a forced migration to solr which is now ready ... for production as a replacement of the metadata-db. This intermediate release 1.041 will switch on the previously optional solr index and the old metadata-db will still work as it did before. Solr+metadata are accessed in mixed mode, no migration is done yet. If this causes not a catastrophe until the end of the weekend, we will do a YaCy 1.1 main release containing this as default.	13 years ago
Michael Peter Christen	a1b2c9a67d	doctype2mime fix, influences metadata conversion between old metadata ... and solr	13 years ago
Michael Peter Christen	a16206e38b	more attempts to clean the index (cleaning is faster then)	13 years ago
Michael Peter Christen	703f427303	fixed some peer-ping connection details ... - larger time-out - removed too old seedlist - fixed a bug in connection test	13 years ago
Michael Peter Christen	597bb76e4f	get the peer location more quickly	13 years ago
Michael Peter Christen	1641835fef	replaced yacy xml encoding by solr xml encoding	13 years ago
Michael Peter Christen	89fe13e73d	enhanced GSA and RSS output format: corrected date, added some missing ... fields, added xml encoding for utf8	13 years ago
Michael Peter Christen	ea49a8aa8c	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	d988ba50cf	added a very rudimentary, incomplete, non-verified GSA response writer ... for solr. Try this: http://localhost:8090/gsa/searchresult?q=pdf&site=col1&num=10	13 years ago
Michael Peter Christen	aab0b680c3	- added xslt support for solr result formats. ... try i.e. http://localhost:8090/solr/select?q=*:*&start=0&rows=10&wt=xslt&tr=json.xsl - added servlet-side mime-type configuration for streamed servlets. this is used for the result formatters in solr result formats	13 years ago
cominch	e2119f4e76	augmented browsing: replace htmlparser by jsoup, which is more stable ... and reliable	13 years ago
Michael Peter Christen	9448d9a8a2	ups	13 years ago
Michael Peter Christen	e5ef840f40	- renamed DoubleSolrConnector to MirrorSolrConnector and added a ... hit/miss/document cache to the MirrorSolrConnector. - more abstraction to SolrDocument in Connector interface - bugfixes in Solr field reader	13 years ago
Michael Peter Christen	94a334f128	another fix to the Solr metadata reading process and to the shutdown ... process	13 years ago
Michael Peter Christen	b51df6c7e8	- added coordinate storage in solr schema ... - fixed shutdown process - fixed some solr-to-metadata reading - added a large number of metadata attributes in ViewFile.html	13 years ago
Michael Peter Christen	da851c6071	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	bd4f03bc85	removed unused class	13 years ago
orbiter	39f8eb60c3	tried to prevent calls to bad-hack getSize() method and reduced overhead ... of that method a bit.	13 years ago
orbiter	e816b88b55	changed behaviour of metadata storage: in case that any solr is ... attached, the metadata is not written to the metadata-db, even if it is enabled but instead to solr. This prevents that metadata is written in two store systems at the same time. It is also the next step to migrate the current metadata-db to solr.	13 years ago
orbiter	2571e0d47a	removed unused classes	13 years ago
Michael Peter Christen	f9c0e6e950	- Implemented and integrated the URIMetadataNode object which is a ... metadata representation from the solr index. This shall replace metadata from the built-in database in the future. - added the Solr-driven metadata into the search index of YaCy which makes it now possible to run YaCy without the old metadata index. This is a major stept forward to a full migration to Solr.	13 years ago
Michael Peter Christen	b2b480fff2	more abstraction of the YaCySchema -> Opensearch matching process	13 years ago
Michael Peter Christen	24462e9baa	set the title every time, it is possible that it has changed	13 years ago
Michael Peter Christen	dcc72799c4	better abstraction for result writers using controlled vocabularies and ... URIRefs	13 years ago
Michael Peter Christen	136fcb1ad9	refactoring	13 years ago
Michael Peter Christen	a12f693ec9	added two response writer for embedded solr interface: ... a rss/opensearch writer and an enhanced solr xml writer. The enhanced solr writer has less configuration overhead than the original writer and should by slightly faster. The rss/opensearch writer is at this time slightly incomplete compared with the already existing rss search result form YaCy and also snippets are missing at this time. To test the new interface, open for example: http://localhost:8090/solr/select?wt=rss&q=olympia The wt-code for the new result writers are= wt=rss for opensearch wt=exml for the enhanced solr xml writer. Additionally, the SRU search parameters had been added to the solr interface which can now also be used for a normal solr/xml search.	13 years ago
Michael Peter Christen	bca4a16603	replaced the multivalue generic string field name suffix _ss by _txt ... because _ss is not part of the standard solr example schema.	13 years ago
orbiter	67edfd991c	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	d9173ba7ed	added more solr fields to integrate values from URIMetadataRow. All ... writings to the Metadata-DB are now also done to solr. This includes metadata transfer during search and rwi transfer. The new/added solr fields are: ## time when resource was loaded load_date_dt ## date until resource shall be considered as fresh fresh_date_dt ## id of the host, a 6-byte hash that is part of the document id host_id_s ## ids of referrer to this document referrer_id_ss ## the md5 of the raw source md5_s ## the name of the publisher of the document publisher_t ## the language used in the document; starts with primary language language_ss ## an external ranking value ranking_i ## the size of the raw source size_i ## number of links to audio resources audiolinkscount_i ## number of links to video resources videolinkscount_i ## number of links to application resources applinkscount_i	13 years ago
Michael Peter Christen	3276508d1b	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	3ce04cecf3	bad hack to prevent a bug appearing in solr	13 years ago
sixcooler	f32aa9a49c	prevent merge of blobs that can't be handled in memory	13 years ago
Michael Peter Christen	bbd242afb4	fix for a NPE	13 years ago
Michael Peter Christen	24d9db1613	snippet retrieval loading processes may use a smaller minimum load time ... value than crawling processes. This speeds up the search result preparation dramatically.	13 years ago
Michael Peter Christen	ef488a15f7	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	1687737771	Abstraction of HandleMap and HandleSet	13 years ago
sixcooler	76b037a20a	check content domain fix: ... search image/media should not show pages containing image/media search text should show all/text but image/media	13 years ago
Michael Peter Christen	e432bb9cd9	better calculation of possible saving in HeapReader index data structure	13 years ago
Michael Peter Christen	9549984c65	documentation/comments	13 years ago
Michael Peter Christen	3bcd9d622b	cleaned up classes and methods which are either superfluous at this time ... or will be superfluous or subject of complete redesign after the migration to solr. Removing these things now will make the transition to solr more simple.	13 years ago
Michael Peter Christen	6f1ddb2519	Moved solr index-add method to the same method where the YaCy index is ... written. Also done some code-cleanup.	13 years ago
Michael Peter Christen	315d83cfa0	cleanup	13 years ago
Michael Peter Christen	1f41d9c6f5	bugfix for a NPE	13 years ago
Michael Peter Christen	76202f068e	extended abstraction of local and remote solr index using one front-end ... for index administration and querying.	13 years ago
Michael Peter Christen	d3f243e2e1	fixed node type calculation for principal peers	13 years ago
Michael Peter Christen	826967513b	changed options in IndexFederated_p to switch on/off parts of the index ... individually. The settings are experimental and the values of the settings will be overwritten when an index migration from urldb to solr starts.	13 years ago
Michael Peter Christen	cba4ab862e	fix for http://bugs.yacy.net/view.php?id=202	13 years ago
orbiter	69e743d9e3	- more abstraction for the RWI index as preparation for solr integration ... - added options in search index to switch parts of the index on or off	13 years ago
orbiter	05a3ffd03a	patches to ensure that solr connectors are active ony if they have a ... solr object assigned and vice versa	13 years ago
orbiter	5a3c829872	embedded solr is only initiated if it is activated with ... IndexFederated_p.html	13 years ago
Michael Peter Christen	97b7bcf2a6	added a solr search index ... - by default, a (empty) solr storage instance is created at SEGMENTS/solr_36 - the index is written if in /IndexFederated_p.html the flag "embedded solr search index" is switched on - a standard solr query interface is available now with a new servlet at http://127.0.0.1:8090/solr/select To test this, do the following: - switch to webportal mode - switch on the feature as described - do a crawl. this fills the solr index. The normal YaCy search will NOT work now! - do a solr query, like: http://127.0.0.1:8090/solr/select?q=*:* http://127.0.0.1:8090/solr/select?q=text_t:Help play with different search fields as you can see in /IndexFederated_p.html You can use the standard solr query attributes as described in http://wiki.apache.org/solr/SearchHandler	13 years ago
Michael Peter Christen	f0a079ac9f	allow larger log entries	13 years ago
Michael Peter Christen	784a4abb18	enhancement in internal data organization which should generate less ... synchronizations in database access	13 years ago
Michael Peter Christen	f78ce93a80	collection of speed and memory saving hacks	13 years ago
orbiter	c00a3cf74d	less usage of generic logger to avoid logger generation overhead	13 years ago
orbiter	a196f24f60	prevent enqueueing of non-loggeable logging entries	13 years ago
orbiter	482afed07c	reduced logging overhead (a bit)	13 years ago
orbiter	e76159040b	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	bbfa497a3c	replaced more size() > 0 by !isEmpty()	13 years ago
Michael Peter Christen	58e7d1952f	reduction of logging to prevent too much IO caused be logging	13 years ago
Michael Peter Christen	83da68c4c1	fixed a memory leak inside the logger which appeared if the log was ... writter faster that the logger is able to print this out to its out stream. A very large collection of unwritten log outputs had been seen during strong crawling. The new ArrayBlockingQueue is limited to prevent this case.	13 years ago
orbiter	0cbda0b2b8	- replaced all length() == 0 and size() == 0 with isEmpty() ... - replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be done automatically - implemented some isEmpty() methods	13 years ago
orbiter	28b30231c3	fix for url matcher of multiple amp& in an url, see: ... http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4439&p=26650#p26650	13 years ago
Roland 'Quix0r' Haeder	aef9dd0350	- removed cleaning of blacklist cache on startup ... - added cleaning of blacklist cache if cache is modified in interface - extended cache saving to all cache types - moved cache location to DATA/LISTS - fixed static file path which was relative to the application path but should be relative to data path - which is different in debian and mac implementations	13 years ago
orbiter	c7afa8bc48	using SwitchboardConstants for solr attributes	13 years ago
orbiter	c6d8950651	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
orbiter	5f3b8dc040	fix for RSS reader	13 years ago
orbiter	62202e2d71	refactoring of query attribute variable names for better consistency ... with (next) stored query words	13 years ago
Michael Peter Christen	1addbc792c	use less memory for md5 cache	13 years ago
Michael Peter Christen	f32de94723	more logging	13 years ago
Michael Peter Christen	d09d9f2364	filter old peers from bootstrap (now stronger: 60 minutes instead of ... 240).	13 years ago
Michael Peter Christen	434ee90c59	added classification for control file types which shall not be loaded ... but placed onto the noload-queue	13 years ago
Michael Peter Christen	a90bcb48f6	added webm	13 years ago
Michael Peter Christen	801972fe6f	fix for url camel case parser and sentence reader	13 years ago
Michael Peter Christen	fbc1a2030d	fix for sitemap importer: can now also import very large sitemaps within ... small memory configurations	13 years ago
Michael Peter Christen	92731e5287	fix for sevenzip parser	13 years ago
Michael Peter Christen	45641b0c23	catch and log a warning in RasterPlotter	13 years ago
Michael Peter Christen	8efc1c1078	- fixed a memory leak (or bad usage) during parsing/snippet fetch ... - more logging for errors	13 years ago
Michael Peter Christen	c3db015410	prevent loading of content from the cache when retrieval with IFFRESH is ... used and cache is stale. Should speed up snippet generation when cache strategy is IFFRESH.	13 years ago
Michael Peter Christen	b1e7c11fba	fix for pattern matcher in html parser	13 years ago
Michael Peter Christen	8a6edc0031	fix for solr shutdown	13 years ago
Michael Peter Christen	b8bcc06283	fix for urls beginning with "//"	13 years ago
Michael Peter Christen	b0c408788b	made class methods static where possible	13 years ago
Michael Peter Christen	5bd3c90907	- removed unnecessary semicolons ... - added default case for switch	13 years ago
Michael Peter Christen	132afaf687	removed unaccessible code	13 years ago
Michael Peter Christen	7c1ba99755	removed more unused method parameters	13 years ago
Michael Peter Christen	83701a1b4c	removed unused ImageReference package	13 years ago
Michael Peter Christen	0301aba1e9	removed unused method parameters	13 years ago
Michael Peter Christen	241dd8410a	removed snippet pattern filter - it was not used	13 years ago
Michael Peter Christen	d3964253ae	- added @SuppressWarnings to unused servlet method parameters ... - removed unnecessary casts - removed unnecessary throw statements	13 years ago
Michael Peter Christen	ea10766bfd	cleaned unnecessary nested code	13 years ago
Michael Peter Christen	1481037820	replaced non-generic array with collection	13 years ago
orbiter	fc0f9543fe	More SentenceReader cleanup	13 years ago
orbiter	586bb0eb6a	Simplified SentenceReader (no more Reader inside..)	13 years ago
orbiter	7f851d62a7	replaced HashARC with SizeLimited Objects which are less costly	13 years ago
orbiter	d4291ac1f3	more tolerance when creating solar document	13 years ago
orbiter	78fc3cf8f8	refactoring and new usage of SentenceReader: this class appeared as one ... of the major CPU users during snippet verification. The class was not efficient for two reasons: - it used a too complex input stream; generated from sources and UTF8 byte-conversions. The BufferedReader applied a strong overhead. - to feed data into the SentenceReader, multiple toString/getBytes had been applied until a buffered Reader from an input stream was possible. These superfluous conversions had been removed. - the best source for the Sentence Reader is a String. Therefore the production of Strings had been forced inside the Document class.	13 years ago
orbiter	bb8dcb4911	automatically adopt size of word cache to available memory	13 years ago
Michael Peter Christen	ad09b786bf	clean up parser data	13 years ago
Michael Peter Christen	276a66a793	Adding a limit of 1000 links that a parser shall store during indexing. ... A limit was necessary because some web pages have such huge numbers of links that it can easily cause a OOM just by the number of links. The quesion if the number of 1000 links is sufficient or too weak must be answered with the result of testing this feature.	13 years ago
Michael Peter Christen	613b45f604	- better data structures in secondary search ... - fixed a big memory leak in secondary search	13 years ago
Michael Peter Christen	de903a53a0	parser refactoring & hacks	13 years ago
Michael Peter Christen	8a82609360	- smaller caches to save memory ... - close cloneable iterators to free memory	13 years ago
Michael Peter Christen	7249d9c9de	bugfix for concurrent seed loader	13 years ago
Michael Peter Christen	c72d3b12cd	concurrently initialize the seed list during p2p network bootstrap	13 years ago
Michael Peter Christen	1825f165b8	better integration of blacklist according to use case	13 years ago
Michael Peter Christen	c18fa9fa75	Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1	13 years ago
Michael Peter Christen	ce8d4b87d9	fixes for new eclipse 'Juno' warning 'Resource leak'.	13 years ago
Michael Peter Christen	0c345d1559	giving threads name so its easier to see whats happening during ... debugging and within a thread dump	13 years ago
reger	067728bccc	add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)	13 years ago
Michael Peter Christen	2f46085be0	more logging	13 years ago
Michael Peter Christen	6ea77e5d7a	Merge branch 'master' of ssh://gitorious.org/yacy/rc1 into jetty	13 years ago
Michael Peter Christen	65f56b1fd4	Merge branch 'master' of ssh://gitorious.org/yacy/rc1 into jetty ... Conflicts: .classpath build.xml htroot/Status.java source/de/anomic/http/server/HTTPDProxyHandler.java source/net/yacy/yacy.java	13 years ago
Michael Peter Christen	03280fb161	removed segments-concept and the Segments class: ... the segments had been there to create a tenant-infrastructure but were never be used since that was all much too complex. There will be a replacement using a solr navigation using a segment field in the search index.	13 years ago
Michael Peter Christen	508a81b86c	added solr field 'refresh_s' which stores the refresh url contained in ... the meta-refresh html header field.	13 years ago
Michael Peter Christen	f3167def64	do not fill the keywords with title content if keywords do not exist.	13 years ago
Michael Peter Christen	9116013c64	- allow lazy initialization of solr value (if using 'lazy', then no ... 0-values and no empty strings are written). This may save a lot of memory (in ram and on disc) if excessive 0-values or empty strings appear) - do not allow default boolean values for checkboxes because that does not make sense: browsers may omit the checkbox attribute name if the box is not checked. A default value 'true' would not comply with the semantic of the browsers response. - add a checkbox in IndexFederated_p for the lazy initialization of solr fields.	13 years ago
sixcooler	97f60010d8	fix crawl start from file	13 years ago
Michael Peter Christen	0294a53459	- add canonical field only if requested by solr schema ... - remove canonical url from in/outbound urls if present	13 years ago
Michael Peter Christen	3fd4a01286	added option to record urls that are forwarded to the solr index	13 years ago
Michael Peter Christen	d763e4d94b	fixed bad referer computation in SSIs which causes a NPE during host ... computation. This error was there before the latest IPv6 hack but did not cause a NPE. The IPv6 hack was not the cause for this bug, but it discovered the misconfiguration of the 'referer' referrer.	13 years ago
Michael Peter Christen	358b04885e	more IPv6 hacks	13 years ago
Michael Peter Christen	96aeb127e3	generalized localhost naming. ... this is also a preparation for a better IPv6 implementation.	13 years ago
Michael Peter Christen	77f795756c	fixing redirects and status codes: storing of status code in ... ResponseHeader to make it available for late evaluations, like storage in solr.	13 years ago
Michael Peter Christen	8dd469b9dd	added option to configure the autocommit delay time of solr on-the-fly	13 years ago
Michael Peter Christen	b9dfca4b0a	- fixed IndexFederated Servlet / a embedded Solr can now be selected ... - added code stub for an embedded Solr but generation of Solr store is still commented out (it works but is not yet ready for usage)	13 years ago
Michael Peter Christen	fad3b14813	added jetty libraries, needed for future use as web server and as ... application server for the solr search interface	13 years ago
Michael Peter Christen	a38b0a2c46	extended embedded solr tests to ensure that it will be usable within a ... jetty instance	13 years ago
Michael Peter Christen	b9d42fd9c8	using com.google.common.io.Files instead of homebrew methods	13 years ago
Michael Peter Christen	a5eb91fa60	refactoring	13 years ago
Michael Peter Christen	1be0025a9c	- added test for EmbeddedSolrConnector ... - added needed libraries for this test this includes most (all) files needed for an embedded solr	13 years ago
Michael Peter Christen	dbdd697f4d	moved RDFaParser.xsl configuration file to defaults	13 years ago
Michael Peter Christen	90b82ce994	using guava for host resolution (non-blocking for ips) and time-out	13 years ago
Michael Peter Christen	e12bb254b4	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	3f55dc7c1e	- added solr core and libraries that solr needs (lucene is missing, will ... follow later) - added embedded solr connector which can connect to solr programmatically (without using a server in between)	13 years ago
Michael Peter Christen	c337190a00	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
cominch	c63c3a4495	Show additional interaction elements in footer section on each page, if ... activated in ConfigPortal.html. This footer is also visible in augmented browsing proxy mode.	13 years ago
Michael Peter Christen	786be7d175	better integration of RDFaParser	13 years ago
Michael Peter Christen	de3ef8ad73	removed unimportant warnings	13 years ago
Michael Peter Christen	82a682b31d	fixed problem with seed when switching network	13 years ago
Michael Peter Christen	8c544edee4	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	7dc59979bc	fix for npe, possibly for http://bugs.yacy.net/view.php?id=195	13 years ago
Michael Peter Christen	1d4e206b2b	bugfix in vocabulary generation	13 years ago
cominch	2c89975378	Merge remote-tracking branch 'original yacy/master'	13 years ago
Michael Peter Christen	52f5d40043	better abstraction of document model generation	13 years ago
Michael Peter Christen	8b7c4d3144	produce a rdf output containing the triplestore with yacydoc; ie: ... http://localhost:8090/api/yacydoc.rdf?urlhash=yOiCM7Fh1hyQ	13 years ago
cominch	f7160dae5c	Merge remote-tracking branch 'original yacy/master'	13 years ago
cominch	e4555cbee3	Augmented browsing: Pass on additional action parameter	13 years ago
Michael Peter Christen	24bbe359ca	integrate also geonames library files for less cities. these are more ... useful for tagging since less normal words are false-identified as location	13 years ago
Michael Peter Christen	223a5440ab	preventing that an empty pnd is inserted into the vocabularies	13 years ago
Michael Peter Christen	8e97ada7c9	IPv6 bugfix	13 years ago
Michael Peter Christen	963f92ed9a	- merged files ... - changed behaviour of delete button in vocabulary edit - fixed size numbe in vocabulary listing	13 years ago
Michael Peter Christen	dd88d0ace2	more logging	13 years ago
Michael Peter Christen	94d54e2d91	added recognition of multi-word terms in vocabulary matching ... this makes the PND usable: it is now possible to recognize persons and navigate with a 'Persons' facet.	13 years ago
Michael Peter Christen	64c0268b2b	show triplestore metadata in yacydoc and viewfile	13 years ago
Michael Peter Christen	0fbd749207	ipv6 update	13 years ago
Michael Peter Christen	c2f0d16d2c	fixed vocabulary initialization	13 years ago
Michael Peter Christen	df3531f8d5	added the generation of virtual vocabularies using the pnd	13 years ago
Michael Peter Christen	a0f1decd82	- added loading of the dbpedia pnd triplestore in the dictionary loader ... - renamed the dictionary loader to knowledge loader - some refactoring in the library provider method names	13 years ago
cominch	2ac7a5c1f2	Augmented browsing: Add overlay bar which shows the vocabulary tags	13 years ago
cominch	3c255c025b	Show tags in search results (if activated in ConfigPortal_p.html)	13 years ago
Michael Peter Christen	16d8f33795	added objectlink generation to vocabulary generation and editor	13 years ago
cominch	f49d92d8da	Cleanup of interaction class and helper routines	13 years ago
cominch	56b0115054	Triplestore: modify routines to access per user store	13 years ago
cominch	a95127c9af	Triplestore: initalize per-user triplestores	13 years ago
Michael Peter Christen	d45718251e	refactoring (Localization -> Location)	13 years ago
Michael Peter Christen	b8b3c87ba7	- renamed localization to location (that was confusing) ... - renamed 'Locale' navigator to 'Location' - produce Location navigation only if geolocation libraries are loaded	13 years ago
Michael Peter Christen	e89747bb67	- added automated generation of vocabularies from url stubs ... - added clear of all terms for vocabularies - added deletion of vocabularies	13 years ago
Michael Peter Christen	79464189a4	The 'Locale' vocabulary, which is generated by geo data, has now the ... objectspace "http://dbpedia.org/resource/"	13 years ago
Michael Peter Christen	eca38c53e7	added a vocabulary editor	13 years ago
Michael Peter Christen	61bb52d55c	- using http://purl.org/dc/terms/references to refer from an ... auto-annotated document to a 'pseudo-linked' document which has an url created with an object-prefix as defined in the vocabulary file	13 years ago
Michael Peter Christen	2bbb6c52cf	added option to clean the triplestore when deleting the index	13 years ago
Michael Peter Christen	50c576599b	allow multiple parser options instead of printing an error	13 years ago
Michael Peter Christen	c02d742e53	proper namespaces in triplestore dump	13 years ago
Michael Peter Christen	8b53771db2	changed behavior of navigation processing: ... - vocabulary annotation is not done any more into the metadata of urldb - vocabularies are written into the jena triplestore using a rdf vocabulary - vocabularies for rdf tripel must be updated; refactoring done - with the new navigation tags in the triplestore a faster pre-urldb-lookup is possible: navigation is processed now within the RWI during pre-ranking retrieval - added also a Owl vocabulary stub to add the plain-text url to the triplestore using the owl:sameas predicate	13 years ago
Michael Peter Christen	5fc6524ca8	- moved triple store to net.yacy.cora.lod (should be generalized there ... later - added abstract add, delete, get methods in the triplestore - added generation of triples after auto-annotation - migrated all MultiProtocolURI objects to DigestURI in the parser since the url hash is needed as subject value in the triples in the triple store	13 years ago
cominch	8d2e6355f8	augmented browsing: remove non-existing external snippet file	13 years ago
cominch	c90f174799	preparation and generalization of augmented browsing methods	13 years ago
Michael Peter Christen	bef823c247	close the reader if finished	13 years ago
Michael Peter Christen	4ee6fb1de9	added missing blacklist dht cache storage (maybe due to mistakes in ... cherry picking)	13 years ago
Roland 'Quix0r' Haeder	e4d36fa5eb	Fix to make all values lower-case (this should make all existing blacklists compatible with the new enum)	13 years ago
Roland 'Quix0r' Haeder	edaa09b9b1	Rewrote all String blacklist types to enum 'BlacklistType', closes bug ... #143 Conflicts: htroot/Supporter.java htroot/yacy/crawlReceipt.java htroot/yacy/transferRWI.java htroot/yacy/transferURL.java source/de/anomic/crawler/CrawlStacker.java source/de/anomic/data/ListManager.java source/net/yacy/peers/Protocol.java source/net/yacy/repository/Blacklist.java source/net/yacy/repository/LoaderDispatcher.java source/net/yacy/search/Switchboard.java source/net/yacy/search/index/MetadataRepository.java source/net/yacy/search/index/Segment.java source/net/yacy/search/query/RWIProcess.java source/net/yacy/search/snippet/MediaSnippet.java	13 years ago
Roland 'Quix0r' Haeder	af5a597e47	Scroogle is not comming back, remove dead code ... Conflicts: source/net/yacy/search/Switchboard.java	13 years ago
Michael Peter Christen	ca93835713	removed usage of deprecated methods	13 years ago
Michael Peter Christen	23e38bd918	do not load the "_triplestore.rdf" files which are of special use in ... sciencenet	13 years ago
Michael Peter Christen	90c6fc4b63	load all - but not the persistent local.rdf - triples from ... DATA/TRIPLESTORE at startup time. The local.rdf is loaded only if the persistent switch is on (as before).	13 years ago
cominch	bbfc53b663	bugfix	13 years ago
cominch	65c5826d93	bugfix ... Conflicts: source/net/yacy/document/parser/augment/AugmentParser.java	13 years ago
cominch	aa0295917c	augmentation ... Conflicts: source/net/yacy/interaction/AugmentHtmlStream.java	13 years ago
cominch	ed2ea0f08e	augmented browsing modification ... Conflicts: htroot/interaction/OverlayInteraction.html source/net/yacy/interaction/AugmentHtmlStream.java	13 years ago
cominch	6b32f7c1f6	re-enable augmented proxy	13 years ago
cominch	3b08edec2e	bugfix ... Conflicts: source/net/yacy/interaction/AugmentHtmlStream.java	13 years ago
cominch	5f8ba7f4f2	small changes ... Conflicts: source/net/yacy/document/parser/augment/AugmentParser.java source/net/yacy/interaction/Interaction.java	13 years ago
cominch	300b235ce8	Updated Demo Servlet ... Conflicts: htroot/About.html htroot/DemoServlet.html htroot/DemoServlet.java htroot/interaction/interaction.js source/net/yacy/interaction/Interaction.java	13 years ago
cominch	90512640bf	Added config switches for custom parser ... Conflicts: source/net/yacy/document/TextParser.java	13 years ago
cominch	df47f31235	interaction: add special table interaction ... Conflicts: source/net/yacy/interaction/Interaction.java	13 years ago
cominch	e14f2881ae	interaction: add special table interaction ... Conflicts: source/net/yacy/interaction/Interaction.java	13 years ago
cominch	d7326079a8	interaction: add global variable store ... Conflicts: source/net/yacy/interaction/Interaction.java	13 years ago
cominch	4e4e7a99f8	interaction: add global variable store ... Conflicts: source/net/yacy/interaction/Interaction.java	13 years ago
cominch	bde07ed7a8	Add tagging overlay element ... Conflicts: htroot/env/templates/jqueryheader.template htroot/yacysearchitem.java source/net/yacy/interaction/Interaction.java	13 years ago
cominch	b0bc0b4572	Add new demonstration module for client-side key-value store (backend: ... triplestore): /DemoServletInteraction.html Conflicts: source/net/yacy/interaction/Interaction.java	13 years ago
cominch	c9dc6cda02	Demonstration: include value from interaction in search results ... Conflicts: htroot/interaction/OverlayInteraction.html htroot/yacysearchitem.java	13 years ago
cominch	bcbd8eee33	Add several parsers, for RDFa and rdf files. ... Conflicts: source/net/yacy/document/TextParser.java	13 years ago
cominch	9ef5a80f4e	add interaction for triples and selector for augmented browsing ... Conflicts: htroot/interaction/interaction.js source/net/yacy/interaction/Interaction.java	13 years ago
cominch	282c1620d6	Allow TripleStore to be persistent after reboot	13 years ago
cominch	5d20cd324a	Add Triplestore and RDF query interface ... Conflicts: build.xml defaults/yacy.init source/net/yacy/interaction/AugmentHtmlStream.java	13 years ago
cominch	bc9a618e0a	augmented browsing: ignore js and css, integrate more user interaction ... Conflicts: htroot/interaction/Footer.html source/net/yacy/interaction/AugmentHtmlStream.java	13 years ago
cominch	b21048892b	augmentedParser add features and integrate external html parser to ... modify existing web pages Conflicts: addon/YaCy.app/Contents/Info.plist build.xml	13 years ago
cominch	9cbfc1a1c0	augmentedProxy, which forwards every proxy request to a ... rewrite engine to customize existing webpages. originally implemented by Florian Richter. Conflicts: source/de/anomic/http/server/HTTPDProxyHandler.java	13 years ago
Michael Peter Christen	3b992e6b00	using utf8 String compression in Webstructure database	13 years ago
Michael Peter Christen	26301a538d	bugfix in Domains - dns-lookup	13 years ago
Michael Peter Christen	cde20911bb	saved a bit more ram using UTF8 String compression for OpenGeoDB and ... Geonames data files.	13 years ago
Michael Peter Christen	225ee42879	made the GeoLocation into an interface with the current ... integer implementation as accuracy implementation of 1.863cm	13 years ago
Michael Peter Christen	2280a7b276	- changed initialization order to prefer allocation of memory for table ... files first - bugfixes in memory amount calculation	13 years ago
Michael Peter Christen	0746308bc2	only the metadata tables shall be able to use the tail cache	13 years ago
Michael Peter Christen	7ec9bef0c3	fix for OOM	13 years ago
Michael Peter Christen	41c02cb10e	- less restrictions for usage of Table RAM copy ... - new limit to use the table copy (instead of flag): 400MB available. If less is available, then a copy is never used. If more is available, then it can be used if there is a remaining space of at least 200MB - flush caches more often: flush the Digest cache	13 years ago
Michael Peter Christen	b8f56a9803	npe bugfix	13 years ago
Michael Peter Christen	dd14b19c26	lazy initialization of block rank table ... only normal web search uses ... this. When interactive search or location search is used, the block rank is switched off	13 years ago
Michael Peter Christen	ba10caf89a	lazy initialization of database tables	13 years ago
Michael Peter Christen	701b9a28a0	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git ... Conflicts: htroot/PerformanceMemory_p.java	13 years ago
Michael Peter Christen	ab7107b34b	fixed RWIProcess queue limits: now discovering hidden results for mass ... result retrieval	13 years ago
Michael Peter Christen	10c9c17d51	fixed handlemap spread factor and null iterator handling	13 years ago
Michael Peter Christen	b0095c8d3c	flush the compressor cache when a cleanup is done	13 years ago
Michael Peter Christen	a61f44f9e4	lazy initialization of block rank table. ... this causes that the table is not initialized when there is no search is done. the effect is most strong if YaCy is started headless which causes no browser pop-up which otherwise would load the search page and therefore trigger the initialization of the table.	13 years ago
Michael Peter Christen	96e9d77270	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git ... Conflicts: source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java	13 years ago
Michael Peter Christen	00f2df1120	a variety of possible memory leak fixes	13 years ago
Michael Peter Christen	3dd8376825	added automatic cleaning of cache if metadata and file database size is ... not equal. It might happen that these data is different because one of that caches is cleaned after a while or when it is too big. The metadata is then not cleaned, but now wiped after a checkup process at every application start. This should cause a bit less memory usage.	13 years ago
Michael Peter Christen	d0ec8018f5	fixes for bad long computation	13 years ago
Michael Peter Christen	6bb07afcc3	accept also files with other file prefix; used to read 'foreign' cache ... files	13 years ago
Michael Peter Christen	96c8119b50	added GeoLocation / GeoPoint classes which uses less memory than ... Location/Coordinates and has initializers with correct order of lat,lon coordinates	13 years ago
Michael Peter Christen	461a0ce052	removed warnings	13 years ago
Michael Peter Christen	62ae9bbfda	allow more POIs, get more at once	13 years ago
Michael Peter Christen	407fdf6968	more bug fixes and performance hacks for search process	13 years ago
Michael Peter Christen	a1fe65b115	performance hacks	13 years ago
Michael Peter Christen	2fe207f813	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	5aee19daa4	added show from cache in search results (not yet finished)	13 years ago
Michael Peter Christen	5e562dcdb7	adopted vocabulary usage within anotation/naviagtion feature of search ... to new SimpleVocabulary class	13 years ago
Michael Peter Christen	514700291a	moved Vocabulary to cora package (added in git ... 964406ad17)	13 years ago
Michael Peter Christen	0284a4d88f	more fixes for double precision of coordinates	13 years ago
Michael Peter Christen	964406ad17	added concurrency enhancement to xml parser	13 years ago
Michael Peter Christen	240045cf7c	fix for bad distance computation	13 years ago
Michael Peter Christen	e0d8643226	- performance hacks ... - added log warnings in case that search processes run into time-out situations - better concurrency for Integer formatter (used a non-synchronized formatter before) - bugfix for search termination (a poison pill was missing) - added timeout parameters for search (again) -> target is, that they are never reached.	13 years ago
Michael Peter Christen	7a329465b3	using pre-compile pattern in blacklist; should enhance search speed	13 years ago
Michael Peter Christen	6e83b02b83	- bugfix for surrogate file reader ... - bugfix for location search: suppress empty search	13 years ago
Michael Peter Christen	9b4c699526	ehanced location search: ... - search request are now made using a map boundary - search results are only computed for the map boundary - the number of results is adopted to the results in the visible range - added a double-buffering for the search result markers - added a search query option for the search results: /radius/<lat>/<lon>/<radius>	13 years ago
Michael Peter Christen	834dc6b263	store more data from interface access	13 years ago
Michael Peter Christen	1f48d1528b	performance hacks	13 years ago
Michael Peter Christen	c70aaccdc9	better location to generate a guid for rss messages	13 years ago
Michael Peter Christen	10da7335ea	performance hack: use a hash cache for all hashes that are computed by a ... byte array. If this hash is used in a HashMap (which is very often the case) then this hack eliminates a lot of re-computations of the same hash.	13 years ago
Michael Peter Christen	f8a0cf6d7c	RSSMessages do not need a concurrent hash map -> removed overhead	13 years ago
Michael Peter Christen	07ca7e4dd1	enhanced RSS parsing by ensuring that it is parsed with a buffered input ... stream	13 years ago
Michael Peter Christen	7c1feefb28	introduced a default 10 second time-out in rwi normalization time ... uring search process to prevent endless deadlocks after a very long running search	13 years ago
Michael Peter Christen	8d997d55b6	better logging	13 years ago
Michael Peter Christen	65d37e6a20	only ASCII needed in seed bitflags	13 years ago
Michael Peter Christen	0f82fb3628	using double instead float for a better release ordering	13 years ago
Michael Peter Christen	43c2c6e588	better logging	13 years ago
sixcooler	56087c1f23	bump to httpclient- httpcore-, httpmime- 4.2	13 years ago
Michael Peter Christen	71c3163f3d	- fixes to node identification ... - added link to node in network list - added marking of portal search node peers	13 years ago
Michael Peter Christen	4d3cc02168	replaced old bzip2 library against better documented commons-compress ... package from http://commons.apache.org/compress/	13 years ago
Michael Peter Christen	ad222be7f8	added node state icon in network list	13 years ago
Michael Peter Christen	3c2bec681f	added a root node flag: identifies peers with short ping time	13 years ago
Michael Peter Christen	c846e9ca14	redesign of the crawler monitor page: show crawled pages instead of ... queue of urls that shall be crawled	13 years ago
Michael Peter Christen	c15fcde1c8	add-on to latest commit	13 years ago
Michael Peter Christen	cf47d94888	performance hack to parse numbers inside of substrings without actually ... generating a substring. This avoids the allocation of a String object ech time a substring is parsed. Should affect CPU load during RWI transmission.	13 years ago
Michael Peter Christen	7e0ddbd275	added a "fromCache" flag in Response object to omit one cache.has() ... check during snippet generation. This should cause less blockings	13 years ago
Michael Peter Christen	81737dcb18	removed stack trace from swf parser since we cant do anything there	13 years ago
Michael Peter Christen	7bf421b9dd	- fixed image search page navigation ... - removed some deadlocks and ConcurrentModificationExceptions during DidYouMean collection	13 years ago
Michael Peter Christen	c6a09eab0b	synchronization needed	13 years ago
Michael Peter Christen	fb94b47b1a	changed queue sizes to have less memory occupied during indexing	13 years ago
Michael Peter Christen	76157dc2c3	bugfix for http://bugs.yacy.net/view.php?id=173	13 years ago
reger	6696cb1313	bugfix: lookup of peernames no result for active peer in page IndexControlRWIs_p.html -> Transfer RWI to other Peer ... SeedDB.lookupByName searche for lowercase peerNames, while MapColumnIndex.getIndex uses peername as is in the keyset. Changed the index init to insert lowercase peer names as key	13 years ago
Michael Peter Christen	c6558cba08	more classification bugs	13 years ago
Michael Peter Christen	082831b9d6	search contentdom was checked in wrong way - fixed	13 years ago
reger	ee553d971e	correct typo in scripts_txt comment	13 years ago
Michael Peter Christen	f294f2e295	bugfix to http://bugs.yacy.net/view.php?id=181 ... tried to make a bit less 'noise' to dns server also included: less processes in snippet fetch to reduce load during search on small computers	13 years ago
Michael Peter Christen	acf8d521a2	fix for http://bugs.yacy.net/view.php?id=126	13 years ago
Michael Peter Christen	bb88878b4d	the last commit was incomplete..	13 years ago
Michael Peter Christen	d320a31ae1	bugfix for http://bugs.yacy.net/view.php?id=186	13 years ago
Michael Peter Christen	fa735f4f04	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	3e1bc9477f	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	6f8a2fef1f	small speed enhancement using a column factory	13 years ago
Roland 'Quix0r' Haeder	d10627d591	More sync in close() methods ... Conflicts: source/net/yacy/kelondro/logging/GuiHandler.java source/net/yacy/kelondro/workflow/InstantBusyThread.java	13 years ago
Roland 'Quix0r' Haeder	b3ae2aa41f	With or without 'final'? At least please try it in other methods ... Conflicts: source/de/anomic/tools/tarTools.java	13 years ago
Roland 'Quix0r' Haeder	fbb946f913	Made a method static (Eclipse suggested it), removed unused import, pk=null check does now output a warning in logfile	13 years ago
Michael Peter Christen	52d307c735	prevent that the snippet fectch process removes catchall entries	13 years ago
Michael Peter Christen	7eece0256f	moved yacy.logging to defaults according to request in ... http://bugs.yacy.net/view.php?id=55	13 years ago
Michael Peter Christen	89142d1e8d	removed (not all) warnings	13 years ago
Michael Peter Christen	5deebd02ea	added serialization	13 years ago
reger	b2175ea4ef	Add possibility to set custom Solr field names for the YaCy default Solr attributes. ... - Changing the format of YaCy's solr.key.list while maintainig backward compatibility Federated index config screens adjusted accordingly - modified the Solr update request to use a 3 min Solr autocommit intervall	13 years ago
Michael Peter Christen	15db703808	added missing serialization to remove all warnings	13 years ago
Michael Peter Christen	1795a7325b	made HandleSet serializable	13 years ago
Michael Peter Christen	e7e381d110	added configuration to switch off redirection following in crawler	13 years ago
Michael Peter Christen	2717c1b749	fixed bug in solr interface	13 years ago
Michael Peter Christen	f150bc218b	fixed bug in solr error document	13 years ago
Michael Peter Christen	cb54c1737b	solrj connector bugfix	13 years ago
Roland 'Quix0r' Haeder	a093ccf5eb	Now used synchronization in all close() methods to make sure all objects ... are 'closed' in an ordered way Conflicts: source/de/anomic/http/server/ChunkedInputStream.java source/de/anomic/http/server/ChunkedOutputStream.java source/de/anomic/http/server/ContentLengthInputStream.java source/net/yacy/cora/protocol/Domains.java source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java source/net/yacy/document/content/dao/PhpBB3Dao.java source/net/yacy/document/parser/html/AbstractTransformer.java source/net/yacy/kelondro/blob/BEncodedHeap.java source/net/yacy/kelondro/blob/HeapReader.java source/net/yacy/kelondro/index/RAMIndexCluster.java source/net/yacy/kelondro/io/ByteCountInputStream.java source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java source/net/yacy/kelondro/table/SQLTable.java	13 years ago
Michael Peter Christen	49cab2b85f	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	0d58fea210	made multiple connector default	13 years ago
Michael Peter Christen	7740c02c56	- enhanced the solr connector ... - added new multiple connector (to replace singleConnector)	13 years ago
Michael Peter Christen	0cf3d36eae	more tolerance in case of corrupted file	13 years ago
Michael Peter Christen	acc6db28ff	added missing classes for solr interface	13 years ago
Michael Peter Christen	adeb33bb36	better abstraction for solr objects	13 years ago
Michael Peter Christen	8864141872	more abstraction in solr connection classes	13 years ago
Michael Peter Christen	c00efc2717	made the solr connection more generic	13 years ago
Michael Peter Christen	ea2bd43b28	patch for broken configurations	13 years ago
Michael Peter Christen	e5ca7f22b1	enhancement in circle drawing	13 years ago
Michael Peter Christen	34f4225d7e	less 'wellformed' calls without asserts	13 years ago
Marc Nause	a691023d04	*) better formatting for network QPM ... *) refactoring	13 years ago
Michael Peter Christen	77f8e9fb9b	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	ba6aaabc51	refactoring + parser bugfixes	13 years ago
Michael Peter Christen	2a0434efa4	Merge commit 'c1f6b4fb5226d3d2f8b2bec9e361f6b3476e03ff'	13 years ago
Michael Peter Christen	942896fe46	removed methods not supported by new solrj connector for httpclient 4 ... Error was: java.lang.UnsupportedOperationException: Client was created outside of HttpSolrServer at org.apache.solr.client.solrj.impl.HttpSolrServer.setDefaultMaxConnectionsPerHost(HttpSolrServer.java:614) at net.yacy.cora.services.federated.solr.SolrSingleConnector.<init>(SolrSingleConnector.java:128) at net.yacy.cora.services.federated.solr.SolrShardingConnector.<init>(SolrShardingConnector.java:55) at net.yacy.search.Switchboard.<init>(Switchboard.java:657) at net.yacy.yacy.startup(yacy.java:222) at net.yacy.yacy.main(yacy.java:1018)	13 years ago
Michael Peter Christen	22e1f68c0b	solrj user authentication patch	13 years ago
Michael Peter Christen	09484955dc	added new entry class for embed tags	13 years ago
Michael Peter Christen	62f2554a01	- fixed build problems (deprecated methods using httpclient 3.1) ... - removed httpclient 3.1 lib which was used by solrj (solrj now uses httpclient 4)	13 years ago
Michael Peter Christen	a6d60fc21f	concurrency enhancement in ConfigurationSet	13 years ago
Michael Peter Christen	453010bd68	- solved problems with backpath normalization ... - redesigned in/outbound link handover - removed iframe links from inbound/outbound in solr scheme	13 years ago
Michael Peter Christen	5f5ed33ed8	patch for media search (audio, video apps)	13 years ago
Michael Peter Christen	7860c1df80	fix needed for new solrj library	13 years ago
Michael Peter Christen	0e13022147	- enhanced solr field documentation ... - added xml api button to IndexFederated_p - the solr schema.xml file can be generated by YaCy	13 years ago
Michael Peter Christen	19efbf1b0f	- apply directDocByURL to NOLOAD Queue ... - choose pushing to NOLOAD as default for site crawl	13 years ago
Michael Peter Christen	659178942f	- Redesigned crawler and parser to accept embedded links from the NOLOAD ... queue and not from virtual documents generated by the parser. - The parser now generates nice description texts for NOLOAD entries which shall make it possible to find media content using the search index and not using the media prefetch algorithm during search (which was costly) - Removed the media-search prefetch process from image search	13 years ago
Michael Peter Christen	a3badd3205	changed search process for images: no more media snippet load process, ... show only links from index which had been on the text search page before. This creates a superfast search process for images!	13 years ago
reger	c1f6b4fb52	lookupByIP: prevent comparing of port parameter if called with port -1 (=unknown)	13 years ago
Michael Peter Christen	f8cd57c92f	new indexing strategy: ALL links that appear anywhere are indexed, not ... only links where the content can be parsed. All non-parseable links are placed into the noload queue. The search process must therefore be able to filter out non-text search results. - This fixes the problem that image search results appeared in the text search. - The interactive search can retrieve now ALL types of links - The p2p interface is now extended to retrieve only certain types of links (text, image, video, apps) - The search process has an extension to filter the right document type according to the search query	13 years ago
Michael Peter Christen	14f67f217c	refactoring of ContentDomain: now subclass of Classification	13 years ago
Michael Peter Christen	8a08c96a82	removed dependency from logging	13 years ago
Michael Peter Christen	a1a5b015d8	refactoring: moved document Classification to cora package	13 years ago
Michael Peter Christen	33d1062c79	refactoring: the cache belongs to the crawler	13 years ago
Michael Peter Christen	4d5da75814	fix for parser problem if a <a>-tag is 'within' html tags with unclosed ... tags. That prevented the <a> tags from beeing recognized. This is a fix for http://forum.yacy-websuche.de/viewtopic.php?p=25516#p25516	13 years ago
Michael Peter Christen	91a86f0b06	fixed to network graph testing	13 years ago
Michael Peter Christen	7b5b9baee0	added citation rank to ranking profile	13 years ago
Michael Peter Christen	046f3a7e8d	check if httpc has decompressed the release file and rename the file ... from .tar.gz to .tar if that happened	13 years ago
Michael Christen	02e4dedff2	fix to url citation collection	13 years ago
Michael Christen	e32055aa15	added stub classes for ... - a new database for url reference data ('seen links') - a new database extending the references to the full url metadata attributes set which shall replace the old metadata database if it is finished - migration help classes stub to use old and new metadata databases simultanously	13 years ago
Michael Christen	ac5d124ee0	experimental implementation of a citation ranking as post-ranking ... method. (ranking coefficient fixed, need to be made configurable)	13 years ago
Michael Christen	8fc86fe397	added storage of full anchor link structure: ... the links between all pages are now stored. The same index structure as used for the word index is used to make a reverse link index. The new file(s) in SEGMENT/default/citation.index.*.blob store the citation index. This will be used to create much more detailed link structures for the YaCy apis and to create a better ranking. A ranking using the citation.index should provide better results especially for portal indexes and initranets.	13 years ago
Lotus	0b3f39136e	allow custom ppm lower than minimum button on /Crawler_p.html ... fixes http://bugs.yacy.net/view.php?id=166	13 years ago
Michael Peter Christen	532c7cf827	added physics experiment to the graph plotter. not active by default	13 years ago
Michael Peter Christen	aba9b1bfa0	better names for elements of a linked graph	13 years ago
Michael Peter Christen	2fc8ecee36	ConcurrentLinkedQueue has a VERY long return time on the .size() method. ... See http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html and the following test programm: public class QueueLengthTimeTest { public static long countTest(Queue<Integer> q, int c) { long t = System.currentTimeMillis(); for (int i = 0; i < c; i++) { q.add(q.size()); } return System.currentTimeMillis() - t; } public static void main(String[] args) { int c = 1; for (int i = 0; i < 100; i++) { Runtime.getRuntime().gc(); long t1 = countTest(new ArrayBlockingQueue<Integer>(c), c); Runtime.getRuntime().gc(); long t2 = countTest(new LinkedBlockingQueue<Integer>(), c); Runtime.getRuntime().gc(); long t3 = countTest(new ConcurrentLinkedQueue<Integer>(), c); System.out.println("count = " + c + ": ArrayBlockingQueue = " + t1 + ", LinkedBlockingQueue = " + t2 + ", ConcurrentLinkedQueue = " + t3); c = c * 2; } } }	13 years ago
Michael Peter Christen	8aba045ba1	if a new pop-up page is set in config portal, then this page applies ... also to the default page configuration for the httpd if no path is given.	13 years ago
Michael Peter Christen	8c06925984	animation of the web structure picture	13 years ago
Michael Peter Christen	898fa7c3f3	use tld heuristic to check if a domain is local or global	13 years ago
Michael Peter Christen	213c8d97f2	use less proccesses in process pool	13 years ago
Michael Peter Christen	c639248c23	protection against strange answers from remote peers during search	13 years ago
Michael Peter Christen	36e4d82b27	changed ranking	13 years ago
Michael Peter Christen	096c17e7cd	added test code	13 years ago
Michael Peter Christen	665626a51b	catch OOM errors during scanning	13 years ago
Michael Peter Christen	1cd711d005	added classes for citation references (for new citation ranking)	13 years ago
Michael Peter Christen	33a405dab8	ipv6 bugfix	13 years ago
Michael Peter Christen	c6c61be3f0	fix for http://bugs.yacy.net/view.php?id=148	13 years ago
Michael Peter Christen	e0f1e7d904	added new citation reference data structure that shall be used for a ... citation ranking	13 years ago
Michael Peter Christen	e18a4f6b74	more tolerant merge iterator	13 years ago
Michael Peter Christen	e101c2e0e2	added changes from copperdust (submitted by email): ... 1. Improved and fixed language detection: 1.1 Identificator.java - recognition fix (improved) 1.2 DCEntry.java - fix (changed detection order due to detection from tld in many cases is incorrect) 1.3 MultiProtocolURI.java - fixed and enhanced language from tld detection (all currently used top-level domains; ccTLD added but not tested). 2. Ukrainian language update. 3. Main Slavic languages langstats (tested and works fine).	13 years ago
Michael Peter Christen	8d63a5887c	bugfixes	13 years ago
Michael Peter Christen	9ad1d8dde2	complete redesign of crawl queue monitoring: do not look at a ... ready-prepared crawl list but at the stacks of the domains that are stored for balanced crawling. This affects also the balancer since that does not need to prepare the pre-selected crawl list for monitoring. As a effect: - it is no more possible to see the correct order of next to-be-crawled links, since that depends on the actual state of the balancer stack the next time another url is requested for loading - the balancer works better since the next url can be selected according to the current situation and not according to a pre-selected order.	13 years ago
Michael Peter Christen	7e4e3fe5b6	free some memory after parsing html	13 years ago
Michael Peter Christen	4540174fe0	memory hacks	13 years ago
Michael Peter Christen	b4409cc803	small redesign of blob column index and usage	13 years ago
Michael Peter Christen	d5c1f2746e	performance hack	13 years ago
Michael Peter Christen	803963aebd	performance hack: better space grow in CharBuffer (speeds up html ... parser)	13 years ago
Michael Peter Christen	8b0920b0b5	tried to fix the ipv6 problem as reported in bug ... but this did not solve all problems because a bug in the apache http client prevented that it worked. Thread dump: Caused by: java.lang.NumberFormatException: For input string: "1450:400c:c01:0:0:0:69" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:458) at java.lang.Integer.parseInt(Integer.java:499) at org.apache.http.client.utils.URIUtils.extractHost(URIUtils.java:310) at org.apache.http.impl.client.AbstractHttpClient.determineTarget(AbstractHttpClient.java:764) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at net.yacy.cora.protocol.http.HTTPClient.execute(HTTPClient.java:597) at net.yacy.cora.protocol.http.HTTPClient.getContentBytes(HTTPClient.java:558) at net.yacy.cora.protocol.http.HTTPClient.GETbytes(HTTPClient.java:341) at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:131) at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:74) at net.yacy.repository.LoaderDispatcher.loadInternal(LoaderDispatcher.java:274) at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:164) at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:150) at net.yacy.repository.LoaderDispatcher.loadDocument(LoaderDispatcher.java:355) at getpageinfo_p.respond(getpageinfo_p.java:97)	13 years ago
Michael Peter Christen	e2f8f263e8	changed storage of search words: keep order	13 years ago
Michael Peter Christen	ed39ef2890	changed generation of protocol information	13 years ago
Michael Peter Christen	0b67a0a5d8	added a column index for tables in blob files. This is heavily used ... during receiving of DHT submissions and when answering remote search requests. Both events together may have caused IO-deadlocking and this commit shall fix that.	13 years ago
Michael Peter Christen	2e5cd6a1b2	fixed parser extension deny list generation and usage	13 years ago
Michael Peter Christen	8bee1472c9	there is no noindex, only nofollow in links	13 years ago
Michael Peter Christen	3cd6dcd352	do not add new solr fields as activated fields	13 years ago
Michael Peter Christen	e3bb73c3d6	serialized some database access methods	13 years ago
Michael Peter Christen	7e728867e5	added a synchronization around iterations to prevent IO-deadlocking ... during concurrent remote search requests	13 years ago
Michael Peter Christen	355ecf330f	reduced target file site to 64mb	13 years ago
Michael Peter Christen	10ae6d94a1	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	2ea585d616	fix for host navigator	13 years ago
Michael Peter Christen	2f6dde92e2	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Peter Christen	c560a582ac	fix for single-word vocabulary lines	13 years ago
Michael Peter Christen	4c5edab1ec	added option to have exception search result windows	13 years ago
Michael Peter Christen	046d7de95b	Merge remote branch 'reger/master'	13 years ago
reger	a95f645a61	Bugfix class repository.Loaddispatcher fixed download file limit of 10000 ... line 355: final Response response = this.load(request, cachePolicy, 10000, true);	13 years ago
Michael Peter Christen	ef78f22ee1	performance hack	13 years ago
Michael Peter Christen	41536eb4a2	performance hack	13 years ago
Michael Peter Christen	f91487fc50	added delete-button for host navigation	13 years ago
Michael Peter Christen	e8d24fd802	author navigator can be switched off	13 years ago
Michael Peter Christen	558ab7bd4e	made the protocol navigator reversible	13 years ago
Michael Peter Christen	96cb75f1d4	made the filetype navigator be able to deselect the search constraint	13 years ago
Michael Peter Christen	1f4f60654a	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git ... Conflicts: source/net/yacy/document/parser/pdfParser.java	13 years ago
reger	32104360ce	PDFParser - return at least first 3 pages of PDF ... fix for pdf parsing without returning parsed text due to interruption by time out.	13 years ago
Michael Peter Christen	ef5192f8c9	using the generic document parser for crawl starts instead of the html ... parser. This makes it possible that every type of document can be a crawl start point, not only text documents or html documents. Testet this with a pdf document.	13 years ago
Michael Peter Christen	a02fdf8625	better error messages	13 years ago
Michael Peter Christen	eadb58dd87	small enhancements in pdf parser	13 years ago
Michael Peter Christen	c6ba44468e	timeout = 5000 instead 3000	13 years ago
reger	b616de5973	PDFParser - return at least first 3 pages of PDF ... fix for pdf parsing without returning parsed text due to interruption by time out.	13 years ago
Lotus	c73af39e54	refactoring of tray icon class, ... now uses Java 6 methods natively	13 years ago
Michael Peter Christen	4eff0e26f1	npe bugfix	13 years ago
low012	8776b84c10	*) small fix to make password change function of reconfigureYACY.sh work ... again	13 years ago
Michael Peter Christen	1a0b6b3913	get more navigation details to search results	13 years ago
Michael Peter Christen	7f9b6b7a0c	added switches to ConfigParser to accept/deny documents by their ... extension	13 years ago
Michael Peter Christen	4901cee3cc	suppress auto-tagged subject entries when sending out or receiving ... metadata from other peers	13 years ago
Michael Peter Christen	83009d86f7	added the vocabulary navigator. It can be very simply tested by ... switching on the locale dictionaries.	13 years ago
sixcooler	985b78cf89	correct 'avaiable()' to use max of young / eden	13 years ago
sixcooler	4da8746275	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
sixcooler	c9aaa9e00a	respect non-reserved Memory in GenerationMemoryStrategy ... and enable it again	13 years ago
Michael Peter Christen	37f2d1b3e9	replaced Thread initialization with ExecutorService pool for delete ... method. This is much faster and produces less blocking when using the Compressor class which is used by the HTCache. I.e. picture search is much faster now.	13 years ago
Michael Peter Christen	a58dc4a91f	added autotagging to document condenser: ... - tags that are automatically generated now enrich the dc:subject - auto-generated tags have a '$' at the beginning of the tag - auto-generated tags lead the tag name with a vocabulary name each tag has the form $<vocabulary-name>:<tag-printname-space-replaced-by-'_'>	13 years ago
Michael Peter Christen	0d6176804b	emergency disabling of GenerationMemoryStrategy because of non-working ... available-method	13 years ago
Lotus	411aab02e3	Windows installer now detects reliably whether YaCy runs. A file lock on ... the yacy.running file has been implemented.	13 years ago
Michael Peter Christen	87f0210480	enriched log output to find NPE in HeapReader	13 years ago
Michael Peter Christen	987b412491	updated solr scheme: generic declaration of solr schemes	13 years ago
Michael Peter Christen	254adea51c	small fixes	13 years ago
Michael Peter Christen	49be60a7c8	WorkflowProcess is forced to make small pauses if shortMemoryStatus is ... reached.	13 years ago
Michael Peter Christen	b7bb84c0bb	set a limit to CharBuffer object size to fight against bad/too large ... content	13 years ago
Michael Peter Christen	c602eaaf46	enhanced search process	13 years ago
Michael Peter Christen	087f97d4c0	less noise if a browser cannot be opened	13 years ago
Michael Christen	eff966f396	fix for search process (it was aborted too early during remote search)	13 years ago
Michael Christen	e6d51363ee	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Marek Otahal	a231d0eeb9	Run from Java the whole app YACY ... start for java webStart allow for better integration with IDE Conflicts: source/net/yacy/gui/framework/Browser.java	13 years ago
Marek Otahal	72adbeae90	!Important: move from Hashtable to HashMap ... Hashtable is an obsolete collection v1, now since v2 offers HashMap with same or better functionality. Please review, almost all code was already moved, so only a few changes. That is not the issue, but I found notices that some (ugly big) helper classes had to be created in past to compensate missing Hashtable's functionality. I'd like input if we can remove some of them. look for //FIX: if these commits Signed-off-by: Marek Otahal <markotahal@gmail.com>	13 years ago
Marek Otahal	f40efb39af	Blacklist loadList() remove duplicates by using Set ... Signed-off-by: Marek Otahal <markotahal@gmail.com>	13 years ago
Marek Otahal	f75b5e40e0	little fix in copy() ... Signed-off-by: Marek Otahal <markotahal@gmail.com>	13 years ago
Marek Otahal	1dc5d9f0f3	make ConnectionInfo comparable and sort list of connections in Connections_p ... ConnectionInfo compare by initTime Connections_p implement wish to sort connections, descending Signed-off-by: Marek Otahal <markotahal@gmail.com>	13 years ago
Michael Christen	fa8da7f89d	vocabularies are now also used as source for a did-you-mean computation	13 years ago
Michael Christen	eaec14ecc4	Dictionaries from words caches can now be used as autotagging vocabulary	13 years ago
Michael Peter Christen	91940fdf56	redesign of WordCache to be prepared to hold multiple ... independent dictionaries. Such dictionaries can then be also used as simplified vocabularies.	13 years ago
Michael Christen	bd40a10230	added autotaggig stub .. only reading and parsing of vocabularies at ... this time	13 years ago
Michael Peter Christen	2ee8cbeb2c	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git ... Conflicts: source/net/yacy/search/Switchboard.java	13 years ago
Michael Peter Christen	992dbdf4bb	added noload statistic to servlets	13 years ago
Michael Christen	eebc02f5c1	fix	13 years ago
Michael Christen	216a287a85	Merge commit '6d4e08ed06c5cd28c45981b2ebe31c7f7ec6fd83' into quix0r ... Conflicts: source/de/anomic/crawler/CrawlQueues.java	13 years ago
stbrumm	d18095dc48	Patch fuer Issue 0000102 ... and fixes to Patch (private peer status is a property of a peer, not a status)	13 years ago
stbrumm	9f1b1b4604	Type for Robinson-Mode/Private Perr added	13 years ago
Michael Christen	20962a4ed7	added metadata node stub for metadata from blobs	13 years ago
Michael Christen	575dbbaa93	enhancements in Blob retrieval: try to use less CPU resources by testing ... a blog first that most certainly has wanted entries.	13 years ago
Michael Christen	585a8f3c44	fixed a bug in search sequence (caused emtpy results)	13 years ago
Michael Christen	361146dd7a	better error handling for file loader	13 years ago
Roland 'Quix0r' Haeder	6d4e08ed06	Rewrote filesize() to (hopefully) avoid a NPE, rewrote Blacklist class to concurrent classes to avoid a CME	13 years ago
Roland 'Quix0r' Haeder	fa08ed5ae5	Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check	13 years ago
Roland Haeder	319fd1f4aa	A concurrent access can happen on the blacklist (with latest introduced blacklist check in media snippet computation)	13 years ago
Roland 'Quix0r' Haeder	a3083d13bf	Blacklist checks are now always turned on, in media searches (e.g. image search) images matching blacklist entries are no longer shown to the user	13 years ago
Michael Christen	52184a1170	fix for search process	13 years ago
Michael Christen	85bd4cc8bc	better lookup for peer names	13 years ago
Michael Christen	20e3084bd4	redesign of fining of peers by ip: more leightweight method to read the ... seed databases	13 years ago
Michael Christen	0797b0de99	new handling of remote search processes: looking for seeds will now not ... block the whole search process any more. A deadlock with a DHT selection process may have been the cause for interface lockings in the past.	13 years ago
Michael Christen	ee9aae5cc0	more about CreativeCommons license vocabulary	13 years ago
Michael Christen	ecd74fe34f	less dramatic upnp failures	13 years ago
Michael Christen	c75e1a3125	Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git	13 years ago
Michael Christen	13f5b5f80d	the component part in the YaCy Metadata is filled using the Dubling Core ... vocabulary	13 years ago
Michael Peter Christen	8d2cbfb685	more vocabularies and more semantics for lod data structures	13 years ago
Michael Christen	9cd36b4c44	added vocabulary for geolocalization as used in georss	13 years ago
Michael Christen	9e5894c784	Removed handling of components objects for URIMetadataRows. ... This is a preparation to replace this rows with nodes from the node store.	13 years ago
Michael Christen	66ab51f89d	added rdf vocabulary	13 years ago
Michael Christen	c04bfaa51b	refactoring	13 years ago
Michael Peter Christen	136b514f52	added a Triple Store based on Nodes that fit to the new storage classes. ... Added also a first Vocabulary for the node store - Dublin Core.	13 years ago
Michael Peter Christen	613ab6a69d	added BEncodedHeapBag and BEncodedHeapShard which are storage container ... for a new metadata store. An abstraction of the content for this storage is defined with MapStore. A MapStore is an abstraction of a RDF Node store.	13 years ago
Michael Christen	6fecd0db88	one more performance hack to prevent costly md5 computation	13 years ago
Michael Christen	e13441b069	better digest pool size (smaller by default but unlimited)	13 years ago
Michael Christen	1f4afb4dc0	performance hacks	13 years ago
Michael Christen	675d557e88	removed debug logging	13 years ago
Michael Christen	e9dc99fe15	added rules to set specific RWIs as private RWIs which are not ... transmitted to remote peers. This will be used for private index copies and phonetic indexes.	13 years ago
Michael Peter Christen	4243ace863	added phonetic classes	13 years ago
Michael Peter Christen	0bcef2d156	added feature as requested in ... http://forum.yacy-websuche.de/viewtopic.php?f=18&t=3461 The search can now be configured with a non-display host list. the search will always exlude the given list of host unless they are requested directly using the host navigation	13 years ago
Michael Christen	204c29f010	small bugfixes for search result display and cache display	13 years ago
Michael Christen	17f962fceb	translator updates: ... - config string for chinese - do not copy the language file to DATA/LOCALE any more (and do not use them there, this is really confusing for new translators)	13 years ago
Michael Christen	078fcde0dd	bad initialization	13 years ago
Michael Christen	14e45e90fd	patch for a bug that I don't understand by now.	13 years ago
Michael Christen	3eccdca63c	protection against too long running snippet fetch processes	13 years ago
Michael Christen	86b3385847	fixed a deadlock during secondary remote search	13 years ago
Michael Christen	c715d19c09	fixes for dependency on svn	13 years ago
Michael Christen	404758698a	less io operations	13 years ago
Michael Christen	0bc5d76bee	ups	13 years ago
Michael Christen	044f83feed	added some pauses into the search process which shall produce ... better-ranked search results. without that pauses the result page will only contain links from the peer that answers first which is not a good average picture of all the peers that provided results	13 years ago
Michael Christen	943b670738	less terrible warning if uPnP fails	13 years ago
sixcooler	448656087a	probably fix for http://bugs.yacy.net/view.php?id=94 ... (don't know how to force this exception)	13 years ago
Michael Christen	f14faf503b	better ranking because we wait a very little time during the search ... process more to get better remote sear results into the ranking priority stack	13 years ago
Michael Christen	762e0ecfb6	fixed localization dictionaries, see ... http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3418&view=next	13 years ago
Michael Christen	d35bdc2df6	removed npe	13 years ago
Michael Christen	e7e429705a	- less automatic indexing after a search (needs to reset the default ... crawl profiles) - fix for concurrency problem in storage of serverSwitch Properties - markup update	13 years ago
Michael Christen	9cd469e6d6	added pull request from als plus an NPE fix	13 years ago
admin	484c4ad339	Merge branch 'master' of git://github.com/f1ori/yacy	13 years ago
orbiter	402e9d71ef	changed ording on release files: main criteria is not the svn any more; releases are now ordered by ... - release number - date - svn number additionally there is a new option to remove the svn number completely git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8135 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
admin	56ce8488e4	Merge branch 'master' of git://github.com/f1ori/yacy	13 years ago
orbiter	4b8ff84705	- search bugfixes (page counter and number of results per page; recognition of new search) ... - experiments to speed-up the network image production (commented out) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8130 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
sixcooler	aeeae75b8a	the timeout of httpclient is not absolut, but till a connection is ... established or between bytes send trying this to reduce count of client-connections to /yacy/search.html of other peers	13 years ago
hermens	2ac272cfbf	Fix for PeerSelection.seedsByAge() for big networks (>1000 Peers) ... To get the most(least) recent peers search those with highest(lowest) LastSeen instead of the first by peerhash git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8129 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	83335c3b09	fix for http://bugs.yacy.net/view.php?id=78 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8127 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
Al Sutton	39898cb94a	Added try/finally protection to ensure streams are closed. Added initial size guess for the CharBuffer	13 years ago
Al Sutton	4c67a964a1	Added try/finally protection to ensure streams are closed. Added initial size guess for the CharBuffer	13 years ago
Al Sutton	3f9b9f953f	Added close() to ensure buffer close actions are invoked	13 years ago
Al Sutton	d73c84f9a0	Allow initial buffer size definition in TransformWriter, and use available() method to set it in htmlParser. In this situation a ByteArrayInputStream is used so the available() method gives a good size estimation and avoid the buffer needing to be continually grown	13 years ago
Al Sutton	f02ea27b31	Added missing closure of ByteArrayInputSteam	13 years ago
orbiter	0796b54601	- some speed hacks for network image ... - panic patch for 'AD' hashes until it is clear where the problem comes from git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8126 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	f9216e388c	- faster ping to clean up old peers faster ... - clean up more news git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8125 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	35a9e8f307	- fixed network graphic ... - debuged evaluation tables - changed cache settings in template engine - some speed hacks - changed int angles for peer positions in network graphic to double angles git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8124 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
Al Sutton	8993cac4d8	Initial performance improvements	13 years ago
orbiter	d9c066227a	fix for npe ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8122 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	8895d8c1cd	removed unnecessary log entries ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8117 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	550c881d80	remove more news (all older than one day) because they can be a performance problem if we have too many peers sending news ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8112 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	ebd840ebf6	- enhanced description on search front page ... - fixed language and heuristic modifier - added hint to crawl start that we can do also ftp and smb crawls - added a protocol extension to remote crawls to transport all search modifiers to remote peers git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8108 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	e22f8497c9	- tested the ARC methods ... - removed strict authentication (if password is empty; this was buggy and not useful; can be switched on if necessary globally and not for each interface method) - increased speed of CrawlResults page (no dns lookup any more) - increased speed of favicon display (removed dns lookup) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8104 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	bc5df0eef5	updated ranking tables (fresh computation) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8103 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	5a55397f99	some last-minute performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	c9216d5adf	fixed secondary remote search (the process that finds distributed join situations) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8098 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	64fd20b857	new default ranking profile ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8097 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	0cf9ebc3b0	speed enhancements when parsing RWI rows (makes search slightly faster) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8096 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	c9a0dbd25a	added a security check ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8094 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	ee8b1d4de1	fixed unresolved pattern and unwanted local/global switch when using votes on search results ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8093 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	c584db991f	creating a bookmark from the search results now works again .. with new YMarks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8092 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	1120f0c93c	update to network graphics: slightly less crawling activity, slightly stronger color for query activity ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8089 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	6cd27473f5	- better default values for caching and cache usage ... - set new caching and verification behavior according to use case automatically git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8087 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	709013385a	fix for language fix ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8086 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	1019c36dad	bug fixes and speed enhancements for search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8085 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	507c9d478d	much better timing when search globally; less blocking; more results earlier! ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8084 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	8e0b2c5832	fixed cluster search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8083 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	c0c6e9e7a5	fix for bad language encoding ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8082 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
apfelmaennchen	564374d1fe	- included YMarks in addition to old bookmarks in yacysearchitem.html; don't get confused by the old bookmark dialog, the ymark is automatically added silently beforehand. ... - reworked bookmark creation on crawlstart - many smaller adjustments to ymarks git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8072 6c8d7289-2bf4-0310-a012-ef5d649a1542	13 years ago
orbiter	05f34a3fa7	added a full, complete, database insert, update and delete API for the tables. ... Please see this example: list all database tables: http://localhost:8090/api/table_p.xml now create a new table and insert some values into 'mytable' http://localhost:8090/api/table_p.xml?table=mytable&pk=&commitrow=&col_termin=Release%20Machen&col_datum=24.11.2011&col_status=ongoing list the table content: http://localhost:8090/api/table_p.xml?table=mytable&pk= update the table and change a single value inside. You must refer to the row using a primary key 'pk' http://localhost:8090/api/table_p.xml?table=mytable&pk=000000000001&commitrow=&col_datum=29.11.2011 you can also select rows using a search operator http://localhost:8090/api/table_p.xml?table=mytable&pk=&count=10&search= now lets delete the row: http://localhost:8090/api/table_p.xml?table=mytable&pk=&deleterows=pk_000000000001 and we can also delete the complete table: http://localhost:8090/api/table_p.xml?table=mytable&deletetable= You can use this to administrate the robots, bookmarks and API steering using an outside application! git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8071 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
lotus	3cc93325f0	temporary remove compare search from tray ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8070 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	804e48888b	smaller bug fixes for search behavior; should produce less unnecessary removals and an exact number of results as shown in counter ... should also be a little bit faster git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8057 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	84c3fc9d97	local/global fixes in search, better abstraction ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8054 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	06352b8d6b	more logging ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8047 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	017a01714d	- enhanced logging in robots.txt parser for remote debugging ... - robots.txt is now more robust against database operations git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	3a15e58e28	- increased stability when opening the robots table ... - increased stability when deleting tables git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8034 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	775b44017e	refactoring ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8033 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	e914a30099	fix for npe ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8032 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	78ce3b13be	typo ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8027 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	85d6bf4ac4	fixed urls to media content during indexing ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8021 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	0d858d48ec	replaced String with StringBuilder in suggestion process ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	3a807e10cf	- added a cache for active crawl profiles to the crawl switchboard ... - moved the domain cache for domain counter from the crawl switchboard to the crawl profiles. the crawl domain counter is now therefore relative for each crawl start, not for the whole crawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8018 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	37e35f2741	normalization of url using urlencoding/decoding ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8017 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	e58438c01c	- added a new retry connector for solr (for cases where solr responses are slow) ... - added a new exist property into the metadataRepository which includes solr entries git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8016 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	d8d9735b4f	stability bugfix ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8012 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	c31564ef08	stability bugfixes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8011 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	f121f4bb45	fix for link in Supporter and Suftipps page ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8010 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	94eab08794	- updated opensearchdescription text and icon ... - removed automatic setting of maxitems during search (can be set now elsewhere) - updated RSSMessage.java git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8009 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	279482a76d	fix for npe ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8007 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	1b86d06d1e	fix for http://bugs.yacy.net/view.php?id=62 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8004 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	eb9c9edb01	enhanced table method (used by almost all yacy api interfaces) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8000 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	4ad9fc2bff	new snippet strategy for search hits in metadata: show beginning of text instead of hit position ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7999 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	a9838f8b99	fix for http://bugs.yacy.net/view.php?id=59 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7997 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
hermens	d3df03838a	make sure myself-target is always inserted at its appropriate position ... this was previously omitted if the own peer should have been the first target or the peer was the last peer before the rotation to AAAAAAAAAAAA git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7996 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
hermens	c3e7efa846	added sender side prevention of rwi flooding as mentioned in SVN 7993 ... saves memory and speeds up enqueueContainers by limiting the size of transfer.Chunk saves network bandwidth by not transmitting RWIs that would get discarded at the target anyway git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7995 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	5af9598bd1	enhanced exported row parsing during row import ... this affects the search and dht receive speed git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7994 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	7598a9e26b	fix for thread dump ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7992 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	8eef8722d1	update to ThreadDump analysis: freerunner and thread state recognition ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7990 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	1df43b137d	another performance hack ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7989 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	7df0643f0e	performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7988 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	a7df70221e	refactoring ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	1b45e33f04	added robots tag parser to solr scheme ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7986 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	cf4fd525ee	added directDocByURL attribute in crawl profile ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7985 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	813f297a95	another performance hack: re-use of known host addresses for isLocal property; avoids look-up in local hash ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7983 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	035ebfbf3b	- performance hacks (should affect the crawl balancer and reduce CPU load during crawl stack re-fill) ... - this may have also (good) performance side effects on other parts of YaCy git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7982 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	b250e6466d	implemented crawl restrictions for IP pattern and country lists ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7980 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	57d5529a01	performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7977 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	47a8c69745	added a new feature to MultiProtocolURIs to get the locale for each url: ... This is done using a new library InetAddressLocator.jar which is NOT added by default to YaCy because it is very old and with that library we will never get a debian package. However, some people want that functionality and it can be made available if the library is taken from http://javainetlocator.sourceforge.net/ and placed into the /lib directory where it will be found using reflection. The new feature will be used to extend the crawler steering. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7975 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	2c3161b4ac	refactoring: ... RankingProcess -> RWIProcess ResultFetcher -> SnippetProcess git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7974 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	d2ea250d99	refactoring: ... - moved many classes from de.anomic to net.yacy - made more sub-packages for search classes git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
low012	277b454a62	*) added comments ... *) minor refactoring git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7971 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	6b22865dbc	- removed some warinings ... - removed a dead update location git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7970 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	0c6d95e57b	- more tolerance against failure of table opening ... - more connections for solrj git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7968 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	6b02b696b0	- add number of search results to end of rss and json output to reflect latest status of retrieval ... - distinguish search access with different verify state in access of search cache git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7965 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	ce2a76d603	performance hack for search process ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7961 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	aaf7a0feaa	yet another cache strategy ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7959 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	8a428d3e77	ensure termination of pdf parser to avoid deadlocking of other processes during search result preparation ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7958 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	2c4a672fe2	bugfixes and performance hacks for tabe index ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7957 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	dad5b586a4	added a concurrent warmin-up of Table data structures. that should speed-up the start-up process but may also cause stronger CPU load at that time. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7956 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	734059d33e	performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7955 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	23e81b28b2	synchronization enhancements ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7954 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	dd4635e323	patches ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7953 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	85a5487d6d	YaCy can now use the solr index to compute text snippets. This makes search result preparation MUCH faster because no document fetching and parsing is necessary any more. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7943 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	0819e1d397	protection against OOM cases in image parser. See also bugs.yacy.net/view.php?id=54 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7942 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	2cba860693	- fix for wrong entries in NOLOAD indexing queue (that caused that urls had been only indexed based on their url and not loaded) ... - patch for better urls to solr admin interface git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7938 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	2842ce30d6	added synchronization in ReferenceContainer and logging for shrinking ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7937 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
orbiter	cec3836e73	added reference limitation to IndexControlRWIs_p.html servlet ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7936 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago
sixcooler	ecb4986b38	refactored stuff from last commit to ReferenceContainer ... see: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3353&p=23163#p23163 the limiting of references is disabled per default to enable this set yacy.conf - index.maxReferences to a value of e.g. 100000 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7935 6c8d7289-2bf4-0310-a012-ef5d649a1542	14 years ago