28b30231c3
fix for url matcher of multiple amp& in an url, see:
...
http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4439&p=26650#p26650
13 years ago
aef9dd0350
- removed cleaning of blacklist cache on startup
...
- added cleaning of blacklist cache if cache is modified in interface
- extended cache saving to all cache types
- moved cache location to DATA/LISTS
- fixed static file path which was relative to the application path but
should be relative to data path - which is different in debian and mac
implementations
13 years ago
c7afa8bc48
using SwitchboardConstants for solr attributes
13 years ago
c6d8950651
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
5f3b8dc040
fix for RSS reader
13 years ago
62202e2d71
refactoring of query attribute variable names for better consistency
...
with (next) stored query words
13 years ago
1addbc792c
use less memory for md5 cache
13 years ago
f32de94723
more logging
13 years ago
d09d9f2364
filter old peers from bootstrap (now stronger: 60 minutes instead of
...
240).
13 years ago
434ee90c59
added classification for control file types which shall not be loaded
...
but placed onto the noload-queue
13 years ago
a90bcb48f6
added webm
13 years ago
801972fe6f
fix for url camel case parser and sentence reader
13 years ago
fbc1a2030d
fix for sitemap importer: can now also import very large sitemaps within
...
small memory configurations
13 years ago
92731e5287
fix for sevenzip parser
13 years ago
45641b0c23
catch and log a warning in RasterPlotter
13 years ago
8efc1c1078
- fixed a memory leak (or bad usage) during parsing/snippet fetch
...
- more logging for errors
13 years ago
c3db015410
prevent loading of content from the cache when retrieval with IFFRESH is
...
used and cache is stale. Should speed up snippet generation when cache
strategy is IFFRESH.
13 years ago
b1e7c11fba
fix for pattern matcher in html parser
13 years ago
8a6edc0031
fix for solr shutdown
13 years ago
b8bcc06283
fix for urls beginning with "//"
13 years ago
b0c408788b
made class methods static where possible
13 years ago
5bd3c90907
- removed unnecessary semicolons
...
- added default case for switch
13 years ago
132afaf687
removed unaccessible code
13 years ago
7c1ba99755
removed more unused method parameters
13 years ago
83701a1b4c
removed unused ImageReference package
13 years ago
0301aba1e9
removed unused method parameters
13 years ago
241dd8410a
removed snippet pattern filter - it was not used
13 years ago
d3964253ae
- added @SuppressWarnings to unused servlet method parameters
...
- removed unnecessary casts
- removed unnecessary throw statements
13 years ago
ea10766bfd
cleaned unnecessary nested code
13 years ago
1481037820
replaced non-generic array with collection
13 years ago
fc0f9543fe
More SentenceReader cleanup
13 years ago
586bb0eb6a
Simplified SentenceReader (no more Reader inside..)
13 years ago
7f851d62a7
replaced HashARC with SizeLimited Objects which are less costly
13 years ago
d4291ac1f3
more tolerance when creating solar document
13 years ago
78fc3cf8f8
refactoring and new usage of SentenceReader: this class appeared as one
...
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
13 years ago
bb8dcb4911
automatically adopt size of word cache to available memory
13 years ago
ad09b786bf
clean up parser data
13 years ago
276a66a793
Adding a limit of 1000 links that a parser shall store during indexing.
...
A limit was necessary because some web pages have such huge numbers of
links that it can easily cause a OOM just by the number of links.
The quesion if the number of 1000 links is sufficient or too weak must
be answered with the result of testing this feature.
13 years ago
613b45f604
- better data structures in secondary search
...
- fixed a big memory leak in secondary search
13 years ago
de903a53a0
parser refactoring & hacks
13 years ago
8a82609360
- smaller caches to save memory
...
- close cloneable iterators to free memory
13 years ago
7249d9c9de
bugfix for concurrent seed loader
13 years ago
c72d3b12cd
concurrently initialize the seed list during p2p network bootstrap
13 years ago
1825f165b8
better integration of blacklist according to use case
13 years ago
c18fa9fa75
Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1
13 years ago
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
13 years ago
0c345d1559
giving threads name so its easier to see whats happening during
...
debugging and within a thread dump
13 years ago
067728bccc
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
13 years ago
03280fb161
removed segments-concept and the Segments class:
...
the segments had been there to create a tenant-infrastructure but were
never be used since that was all much too complex. There will be a
replacement using a solr navigation using a segment field in the search
index.
13 years ago
508a81b86c
added solr field 'refresh_s' which stores the refresh url contained in
...
the meta-refresh html header field.
13 years ago
orbiter