7993 Commits (3a44ee74ba8089a54c30fd6e4d669e33f90e88c8)

Author	SHA1	Message	Date
reger	a6617ad887	expand initRemoteCrawler() to terminate worker threads if called to deactivate ... remote crawl. On startup we save the resources for remote crawler if disabled. Once started threads are running idle after disable remote crawl. Now threads are terminated to save the resources also while disabeling during runtime. + remove empty class Channels	9 years ago
reger	2048b7e057	support scraping start-/enddate from html tag with property "datetime" ... This may be used in html5 <time> tag (which we don't explicite support yet for date in content scraping).	9 years ago
reger	900d4584ba	complet resource cleanup of lists in contentscraper's close()	9 years ago
reger	1f18653de0	pass parsed swf content trough htmlscraper ... Swf may contain subset of html tags which shoul'd appear as text. Especially <font> tag may totally screw up metadata servlet if not filtered out.	9 years ago
reger	18ecf57792	add support of compressed swf to swfParser ... from JavaSWF2 (source compatible to WebCat). Moved swf file signature check to parser Changed use of synced vector to list swf InStream	9 years ago
sixcooler	5cb7ba0dc4	fix for connections not getting closed to get favicon.ico during seach	9 years ago
reger	ed3e16e092	apply remote result count config value to Bookmark Autosearch ... + prepare to make the widely unused Bookmark feature optional	9 years ago
Ryszard Goń	a98c395023	Add the Autocrawl thread	9 years ago
Ryszard Goń	1728cd30c6	Create autocrawl profiles	9 years ago
reger	ff27824964	fix swfParser reading file signature ... before passing to library (current version expects data w/o signature)	9 years ago
reger	c91e712178	further refactor using standard java / (one) utf-8 charset variable ... extending initiative of commit 9a25751850	9 years ago
luc	571bc55937	Refactoring : use StandardCharsets constants instead of hard-coded ... charset names.	9 years ago
reger	1af0e9ef74	remove workaround for Solr bug regarding multivalued date fields ... fixed in 5.4.0 http://issues.apache.org/jira/browse/SOLR-8050	9 years ago
sixcooler	5a35f9383a	bump to solr/lucene 5.4.0	9 years ago
reger	a58d34a4e8	check error URL cache before adding errorDoc to index ... - del obsolete related switchboardconstant	9 years ago
reger	e9539b1086	reintroduce special handling of file upload multipart/form-data from HTTPDemon.parseMultipart ... - add filename to parameter fieldname - add filecontent to special parameter fieldname$file (some servlets use this $file parameter) fix for http://mantis.tokeek.de/view.php?id=542	9 years ago
reger	cd26717ba2	fix low memory status hint (dht-in disabled) ... http://mantis.tokeek.de/view.php?id=619	9 years ago
reger	a5faf73afa	remove obsolete yacy.init entries interaction.* ... (related to removed triplestore)	9 years ago
sixcooler	dce1cb65c4	Merge remote-tracking branch 'choose_remote_name/master'	9 years ago
reger	46ac0867ff	fix poison mediawikiimporter output queue also after ExecutionException ... in worker thread. Writer of importer keeps needs a poison to close the file. On exception (e.g. OOM) add a poison marker in outer most try/catch to assure output queue will terminate in this condition too (and closes+renames the surrogate/in/xxx.prt file)	9 years ago
reger	a7591d3ed0	fix mediawikiimporter number format exception on coordinate parsing ... handle uncomplete metadata like "NS=43/50//N". For other {expr ... } type entries a try catch added	9 years ago
reger	9da1712a31	increase http header EXPIRES for css and images in DefaultServlet ... to increase browser cache hits for not changing content	9 years ago
reger	6d54eb3d36	skip loading document on crawl start for YMark bookmarks ... by adding a constructor giving the already loaded document as parameter.	9 years ago
reger	80e2c82249	fix NPE on empty blog importfile parameter	9 years ago
reger	e84d94f8ca	fix mime table for ms office / open office documents ... (causing wrong parser detect in intranet mode)	9 years ago
reger	45b9bd8403	adjust MultiProtocolURL.protocol detection to handle mailto with "://" in parameters, ... and feeding hyperlinks to webgraph processing.	9 years ago
reger	d5fd031449	fix reading of ippattern config array in URLProxy	9 years ago
reger	b7e8358645	make use of header.getContentType where possible (mime is normalized afterwards) ... otherwise use header.mime() differentiated in prev. commit.	9 years ago
reger	7a8c077838	fix HeaderFramework.mime() to strip charset parameter. ... Differentiate mime() and getContentType() which gives the raw header field. This improves parser detection if charsets are included in http content-type field.	9 years ago
reger	b4b6910d60	fix (todo): correct doc.id of remote search result if no match with newly ... calculated doc hash if different. Testing showed that in some cases delivered url doesn't match the local calculated hash. In this case replace doc.id (and host_id_s) with calculation from url.	9 years ago
reger	dec3e6ad96	fix: adjust urlstub for mailto links ... (skip protocol)	9 years ago
reger	cb83e65f89	drop returning document language "en" if unknown (fix todo) ... which also harmonizes handling of query.modifier for rwi and solr results (to result must match a given language filter)	9 years ago
reger	0c5548a7ff	fix (todo) remove redundant holding of email link nameproperty in parser document	9 years ago
reger	71c416f383	show mailto links in ViewFile.html linklist	9 years ago
reger	6b7c10cef8	fix dc:date in mediawikiimporter/document.writexml to use lastmodified	9 years ago
reger	14803d58cd	let html scraper accept html5 <link rel="icon"> for favicon links	9 years ago
luc	b4cdacee76	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
luc	ba0a293f5c	Corrected another case of ... org.apache.lucene.store.AlreadyClosedException" occuring when SearchEvent.cleanup() was called while committing local solr index.	9 years ago
reger	4d2b934487	prevent mailto links getting into parser result document's in/outbound link collection ... by checking mailto scheme early. - fix upper case mailto protocol assignment - add test case for getProtocol	9 years ago
luc	8c4ab9c76b	Added an option to eventually limit size of remote solr documents put to ... local index. See mantis #626.	9 years ago
luc	a2c08402af	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
luc	70595d05d0	Modified MemoryControl.main() test to properly end for better results ... displaying.	9 years ago
sixcooler	1be67d9ab6	CachedSolrConnector was replaced by ConcurrentUpdateSolrConnector years ... ago - time to let it go Commented out unused table of cache-objects	9 years ago
reger	28b8bc290a	fix use of NETWORK_SEARCHVERIFY for rwi verification ... was not used to set the searchevent parameter (done in SearchEventCache.getEvent) - remove unused corresponding QueryParams.filterfailurls param.	9 years ago
reger	020630efd8	remove unused network scanner parameter from queryparameter ... Search event is not using networkscanner (removed filterscannerfail param always init to false)	9 years ago
luc	ad5586f8f6	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
luc	8ebefa4233	Fixed MediaWiki import : DCEntry conversion to SolrInputDocument was ... failing. Looks like it was broken since Commit b43811d38c	9 years ago
luc	7736ee5a42	Updated MediaWimporter main() : display usage in console and stop ... properly without calling System.exit	9 years ago
reger	cdb8f3b10d	make current ranking score value avail. to search interface / api ... Update the result score result field with the result queue ranking value to reflect the actual calculated/used score, for rwi & solr stack results. (calc. etc. is unchanged, it's just that result entry carries the latest val as api retrieves the number from it)	9 years ago
luc	27d11f8671	Fixed isSolrDump function : PushBackInputStream was not unread when ... returning false (for example with a WikiMedia dump).	9 years ago
Michael Peter Christen	135a123a77	less logging in new language detection	9 years ago
Michael Peter Christen	ef8cd80593	fix for npe	9 years ago
reger	0347bfa71f	Apply collection query constraint/modifiert to rwi result stack. ... Collection is not available in pure rwi entries (but in local solr metadata) But if user wishes to filter by query constraint also rwi shall adhere to this (even if only rwi entries with parsed or solr received metadata may fit)	9 years ago
luc	2a67d2ba6f	Corrected error management for unsupported image formats, parsing ... errors, and unavailable resources : avoid logging to much Exceptions as these errors easily occur when searching images.	9 years ago
Michael Peter Christen	d6e9834040	Merge branch 'master' of ... https://github.com/Scarfmonster/yacy_search_server # Conflicts: # .classpath # build.xml	9 years ago
Michael Peter Christen	d82d311995	Merge branch 'master' of https://github.com/luccioman/yacy_search_server ... # Conflicts: # .classpath	9 years ago
reger	b5371ea8c1	read/init crawl queue in a thread ... to speed-up YaCy start on large existing crawler queues	9 years ago
reger	1160b13172	remove unused md5 from ViewFile servlet params	9 years ago
reger	e163ea88f6	fix vsdParser (Visio) parser return statement ... (final block un-necessary throw)	9 years ago
reger	b2c8bc0ae6	remove md5_s from default index fields ... it is not assigned a value / not used Due to above also excluded from transfer protocol.	9 years ago
luc	e40ae0943b	- No max dimensions specified : render raw image data when source and ... target image format are the same. - Corrected scaling condition.	9 years ago
reger	90686a75a2	fix flux factor (additional crawl delay by access count) calculation	9 years ago
luc	4af27289e5	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	297fdb60d3	throw exception if crawler hostqueue can't create hostpath directory. ... In rare cases hostname may not be a valid filesystem directory name, which can't be created (e.g. containing '*' char). To prevent crawl queue looping on this invalid entry by throwing a malformedurlexception.	9 years ago
luc	755efac17d	Use same max file size when loading all resource bytes or opening stream ... content	9 years ago
luc	bc6c79fc12	Corrected scaling function for non RGB images.	9 years ago
luc	1565559df8	Refactoring : extracted write InputStream method.	9 years ago
luc	f0478bb14d	BMP and ICO image formats support : integrated /haraldk/TwelveMonkeys ... imageio-bmp-3.2 library. - better BMP format flavours support - handle PNG encoded icons - handle transparency Added some javadoc url references to .classpath	9 years ago
luc	07437986e7	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	97cc03ef6a	start using a template for urlproxy header ... It is included as iframe /proxmsg/urlproxyheader.html to allow full servlet functionallity and flexibility to display some index/meta data in future.	9 years ago
luc	f01d49c37a	Process large or local file images dealing directly with content ... InputStream.	9 years ago
luc	3c4c77099d	If available, check content length before downloading. Check also ... content length is not over Integer.MAX_VALUE.	9 years ago
luc	5bbb2e1730	Ensure resource is closed when reading a full file InputStream	9 years ago
luc	6291a57300	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	0d3c5b223e	have psParser cleanup temp file	9 years ago
reger	7d0d19cb8e	avoid File.deleteOnExit() on temp files ... JVM registers each file in a list regardless of already deleted and never cleans up the list during runtime. This accumulates to a considerable amount of mem during large crawls and/or long uptime. To tackle this, all temp files are now created in a subdir of java.io.tmpdir and the jvm tmpdir property is set to this subdir, which is deleted by code on shutdown. Additionally let pdfParser use this tmp subdir too.	9 years ago
luc	bfe51001e3	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	02e4489a23	set tmpfile.deleteOnExit by default, ... to make sure files are removed on shutdown.	9 years ago
reger	2985baaa01	Exclude repetitive protocol part in tokenized url ... used as description if none is avail. from parser.	9 years ago
reger	ca3d26a401	harmonize wordsintitle & CollectionSchema.title_words_val calculation, ... remove obsolete partial init of wordreference from urimetadata	9 years ago
reger	52a9040ae6	Sort out double keywords (dc_subject) early in parsed documents ... - by direct using Set vs. List - remove not neede String[] getter	9 years ago
luc	49331dc523	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	47d70732f6	improve locale translator ... - skip empty line - robustness file section detection (space independant)	9 years ago
sixcooler	646afe9183	do not store subfield *_coordinate + make all num-fields being docvalues	9 years ago
sixcooler	194df613de	not using 'location' as defaultfacetfield - since we removed it being ... default.	9 years ago
sixcooler	d3b9349b6f	simplification / speedup of GenerationMemoryStrategy	9 years ago
sixcooler	4a905ec134	fix to not let the AccessTracker-Log grow to much, but have enough data ... to monitor. (+gitignore-correction)	9 years ago
reger	20e18d79f8	harmonize document title for archive parsers	9 years ago
luc	f11b5e8309	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	112ae013f4	update bzip and bzip parser process, ... to return one document for the file with combined parser results of the containing file and registers it with supplied url and mime of the archive.	9 years ago
reger	e76a90837b	update zip and tar parser process, ... to return one document for the file with combined parser results of the containing files.	9 years ago
luc	4e673ffc9a	Ensure closing of InputStream even when an exception occurs.	9 years ago
luc	10696b53f7	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	8532565c7d	optimize order of parsers to try ... - start with a parser matching the remote supplied mime	9 years ago
reger	681889ae64	use current tar library for untar files ... - remove old source copy	9 years ago
reger	5d71fc70e3	fix tarParser early exit on looping content ... - adjust check of data available according to doc - return null on no recognized content (to not exit TextParser next parser try) - use commons.compress directly	9 years ago
luc	bcc2e7cb5b	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	2fcf6f104c	fix bzipParser recognition ... - Bzip2Inputstream checks magic byte itself to identify bz2 (leave it in input) - try to suppy fitting mime for parsing bz2 content	9 years ago
luc	745e97a575	Merge branch 'master' of https://github.com/yacy/yacy_search_server	9 years ago
reger	a60b1fb6c2	differentiate api call getLocalPort() from getConfigInt()	9 years ago