6077 Commits (d456f693814928ef741745fa241ee5fe16fec1bf)

Author	SHA1	Message	Date
reger	d456f69381	SeedUpload url : check to reject localhost url included in saveSeedList (same check as in / copied from Seed.isProper() ), to prevent identity change on next startup (due to rejected seeduploadurl).	12 years ago
reger	4987caf1c9	- apply fix for localhost handling (from yacy2solr) also to metadata2solr	12 years ago
reger	0148f1bb8c	fix: exception if default work files don't exist	12 years ago
Michael Peter Christen	9e4033f229	fix for event starter: delete start time when event is removed	12 years ago
Michael Peter Christen	99271ffd13	copy work tables from defaults/data/work if exist there and not in ... DATA/WORK This can be used to create start-up behavior work scripts in the api.bheap table	12 years ago
Michael Peter Christen	24c9bb35f7	extended the Scheduler: introduced scheduled events ... - an event type (once, regular) can be selected - for this event type, a fixed time can be selected. This may be either directly after startup or at one of the full hours at a day (==25 options) The main point about this feature is the opportunity to start an action directly after startup. That makes it possible to create YaCy distributions which, after started at the first time, start to index parts of the intranet/internet by itself.	12 years ago
Michael Peter Christen	433143ba40	removed protocol, tld, ext from the urlmask and created specific ... navigation field for these	12 years ago
Michael Peter Christen	84f82541e8	search process enhancements	12 years ago
Michael Peter Christen	02020b590b	- removed all extension types from extension navigation which are not ... proper/known - automatically show the protocol navigation if there is more than http and https - automatically show the extension navigation if there is some media content	12 years ago
Michael Peter Christen	01200f06cc	using the author field as solr-native facet. this makes it necessary to ... introduce a copy-field for the author field to be copied to a string field. This field is then used to generate facets. Without this field, the facet would consist only of the words of the author names, not of the full author string.	12 years ago
Michael Peter Christen	2a4c064c89	using the publisher information for the author field if no author is ... given. This applies to cases where only the copyright field in the html header is filled but not the author field	12 years ago
Michael Peter Christen	bab573361f	- using a filter query for facet restriction ... - calculating the whole search result in at most two sub-queries from solr	12 years ago
Michael Peter Christen	eac9650b31	added another solr field clickdepth_i which reflects the number of ... clicks which are necessary to get from the portal of a host to a specific document. At this time, only the start document is flagged with clickdepth '0', all other with '-1'. To get the actual clickdepth, a process must use crawled information to collect the actual number of clicks. This will be added in another/next step.	12 years ago
Michael Peter Christen	1052263af3	- added a new solr field references_i which stores the number of ... INCOMING links to the corresponding web page. This information is taken from the reverse link index (a 'little sister' of the RWI index). - this field can be of use to enhance the ranking because a web page with more incoming links can be more more important than others. But this is not true for typical link pages like menues. Therefore the number of outgoing links is needed. - added a new solr attribute 'bf' to solr queries which is a boost function extension. this field can contain a formula which comuptes the boost according to given field values. After some experiments the following forumla is now default: div(add(1,references_i),pow(add(1,inboundlinkscount_i),1.6))^0.4 This takes the number of references and the inbound links. Further experiments are needed to enhance that forumula.	12 years ago
Michael Peter Christen	7c3de8b4cd	- fix for localhost detection ... - added IPv6 patterns for localhost detection	12 years ago
Michael Peter Christen	34f8786508	removed dependency of vocabulary navigation from Jena and it's ... triplestore; the vocabulary search is now done using generic solr fields which are created on-the-fly during runtime.	12 years ago
reger	ad71747525	fix: set defaul language to "en"	12 years ago
Michael Peter Christen	9319b90d8a	- fixes for host navigation ... - fixes for filetype navigation - removed unused code	12 years ago
Michael Peter Christen	cb5cbec14d	distinguishing modified query string and original query string	12 years ago
Michael Peter Christen	fb0fa9a102	- fixed 'delete from subpath' during crawl start which deleted nothing; ... now works; - changed some crawl start html design details	12 years ago
orbiter	712cc37c40	if maxFileSize < 0 then the file size limit is without limit.	12 years ago
orbiter	1f33c30d7b	re-integrating useForHost method (lost sometime?) to get the noProxy ... pattern working again. Without using this method all remote urls including the localhost had been accessed through the configured proxy	12 years ago
reger	f1a9c2e604	fix Servlet template on conditional file include with use of conditional template pattern in included template file (example IndexCreateQueues_p.html) ... see bug http://bugs.yacy.net/view.php?id=215	12 years ago
orbiter	a4a780b871	- fix for bad url conversion in bookmarks when using smb urls ... - fix for localhost hosts in solr schema host handling	12 years ago
reger	e80dfeca23	- making blacklist path part case insensitive (solving http://bugs.yacy.net/view.php?id=171) ... - blacklist test adding explicite response text "not blocked" if no blacklist match	12 years ago
reger	e2d499be9e	remove NOT NEEDED reference to solr.YaCySchema from ConfigurationSet to be able to use ConfigurationSet for other conf files (than solr.keys.default.list).	12 years ago
Michael Peter Christen	a3cd3852ab	introduced a better place to update the lastacc time value in latency	12 years ago
Michael Peter Christen	864abcd33d	removed Latency update after URL selection because that causes ... a completely wrong behaviour when cache fresh cases appear. Makes re-crawling MUCH faster!	12 years ago
Michael Peter Christen	dd241d03bb	latency fix: only set last-visit time if access was actually by the ... robot	12 years ago
Michael Peter Christen	118233a7e6	fix for bad xml in gsa result when doing a query with quotes	12 years ago
Michael Peter Christen	1e002ab18e	added another blacklist-cleaner into balancer	12 years ago
Michael Peter Christen	10527e28ae	fix for wrong display of error urls in HostBrowser	12 years ago
Michael Peter Christen	756772fbd3	fix for waitingtime computation for intranet configuration	12 years ago
Michael Peter Christen	fa27e5820f	- check blacklist (again) when taking urls from the crawl stack because ... the blacklist may get extended during crawling - removed debug output	12 years ago
Michael Peter Christen	adfecc6ba8	more robustness during shutdown	12 years ago
Michael Peter Christen	d4bfe9339e	Brute-force attempt to start solr in case of a memory problem. ... I don't actually know if this is correct. It is a desperate try to get YaCy running on production servers which must get alive even with strange hacks like this. This is also related to a forum posting in http://forum.yacy-websuche.de/viewtopic.php?t=4528&p=27135#p27135	12 years ago
Michael Peter Christen	8aa08261a7	update to Solr Boost handling	12 years ago
Michael Peter Christen	908ad2f174	Added a new servlet to configure the solr ranking using field boosts	12 years ago
Michael Peter Christen	a01e47b992	enhanced exists()-method for solr; should reduce a lot of IO during DHT ... target selection	12 years ago
Michael Peter Christen	72f165d58b	added a Boost class which stores solr query boost values. The class can ... be configured using the yacy.init file. The boost information is taken from the configuration each time when a query to solr is done.	12 years ago
Michael Peter Christen	b5ee88c6af	added more logging to get info which url causes performance problems	12 years ago
reger	1faa045dc1	fix: prevent regex pattern compile error for blacklist import for path '*' (extend it to '.*')	12 years ago
reger	6cf33f899c	prevent Solr "version conflict" on update by set Solr "_version_" field to 0 (=no version check)	12 years ago
Michael Peter Christen	acd98bebb7	improvements in GSA result writer	12 years ago
Michael Peter Christen	3de784c8dd	replaced more split and replaceAll missing pattern pre-compilation with ... pre-compiled pattern	12 years ago
Michael Peter Christen	8fc3679c66	using more pre-compile pattern for split methods	12 years ago
Michael Peter Christen	d48e9788d2	enhanced search result processing behavior ... - query less at one time; query more often - in between the small queries, evaluate results - remove fields from search results which are not needed	12 years ago
Michael Peter Christen	bf512e6350	Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1	12 years ago
reger	469efcdb9d	fix: display and calculate authors and namespace search navigator if configured (otherwise skip overhead) ... (leave hosts, topics and not in ConfigPortal included filetype, protocoll navigator untouched)	12 years ago
Michael Peter Christen	eca68fa197	added debug code to crawler monitor	12 years ago