orbiter
c6525ab75f
fix for NPE in seed handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fea82b54ef
more contrast on search snippets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5370 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1951d30a62
addendum to last commit
...
handle words with length < 3 correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
325ba7bfb8
only query words with length > 2
...
this is not complete, yet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
489edb4473
improved pattern selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5367 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
e423fa9846
*) added method to only get file names in directory listing which match a filter
...
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
577b53aee6
added more search engines
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5365 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
7f4d411c0d
npe-fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5364 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
513179f404
changed interface to colletctionIndex and adopted all implementing classes:
...
do not return a result of a double-check when adding entries with addUnique
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5363 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9d64693cfb
reverting again the changes to new concurrent chunkIterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5362 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
45ad1c3dd5
- re-activated concurrent iterator for EcoFiles
...
- added javadoc for new concurrent intialization in kelondroBytesLongMap
- switched default value for commons storage to false
- version step
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5361 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2e2120046f
speed enhancement for BLOBHeap opening process
...
using concurrency of FileIO and content processing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5360 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1545e5440a
* index deletion: checkbox-confirmation
...
* watch crawler: less load on exhausted peers; wait for data before reloading again
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5359 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fa26a8f25a
fix for deadlock-like behavior in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5358 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1918a0173e
added more exception handling during crawling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5357 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
10f5ec1040
reverted last commit (more testing needed)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5356 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
5af8923f37
* distribute forgotten jar-file in parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5355 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b0f2003792
fast database initialization and fast start.up of yacy:
...
- applied knowledge about concurrent files stream reading and index processing from the wikimedia reader
to the EcoTable initialization process: the file reader is now concurrent to the index generation
- changed also some initialization processes to avoid some pauses during initialization
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5354 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
daburna
ba5b274b8c
#translation update:
...
-blacklist
-crawlstart
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5353 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0ca4bc7b79
- added reader and visualization for mediawiki-export files:
...
files exported from mediawiki using the xml schema according to
http://www.mediawiki.org/xml/export-0.3/
can be processed to be viewed in a YaCy servlet.
To acces such a file, place it into
DATA/HTCACHE/mediawiki/
i.e. the export from german wikipedia would be:
DATA/HTCACHE/mediawiki/wikipedia.de.xml
This file can then be accessed using the URL
http://localhost:8080/mediawiki_p.html?dump=wikipedia.de.xml&title=YaCy
if this is done the first time, an index file is created
(for this case: more than 4 million lines must be written, this takes about 15 minutes)
Then try the same url again.
- enhanced also the md5 computation speed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5352 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
2e63f03ca5
copy&paste vergessen :/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5351 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
cd8082b4e3
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111#p11166
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5350 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
4f996a7651
fix for logparser pattern
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5349 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d18c18971e
* dirlisting in UTF-8 encoding
...
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1550&hilit=#p11108
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5348 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
bb570716e6
added more testfiles
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5347 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
867d0f2f56
removed some unnecessary pause delays
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5346 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d49ffcd818
* files distributed by yacy are utf-8, files from repository use the system default charset
...
* fixes http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1564#p11092
and http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1550
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5345 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8c96bc2ac1
do not use proxy caching rules for crawling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5344 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fd83e59f8e
new remote search average
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5343 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dba7ef5144
extended crawling constraints:
...
- removed never-used secondary crawl depth
- added a must-not-match filter that can be used to exclude urls from a crawl
- added stub for crawl tags which will be used to identify search results that had been produced from specific crawls
please update the yacybar: replace property name 'crawlFilter' with 'mustmatch'.
Additionally, a new parameter named 'mustnotmatch' can be used, which should be by default the empty sring (match-never)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5342 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
96174b2b56
more debugging / better result status logging for parser/caching errors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5341 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
84185baa81
added more test files for windows from lulabad
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5340 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
73c44573e8
revert; used to hide memory tables and thread timings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5339 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
8e1636e6d0
removed unused config-option
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5338 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
90e78b2cf6
* improve encoding detection of http service
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5337 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3246358485
mistake -> rename
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5336 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
55ec57d27f
added linux umlute test files from low012
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5335 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0ae84f4f8e
set some default values for a crawl start that should cause less confusion and mistakes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5334 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e9262b3890
re-named old test files
...
added more mac test files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5333 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ff2a54da68
added more umlaute test files: mac
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5332 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
4745e89451
auto-choose crawl type
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5331 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
421d056550
*) changed layout of blacklist adminstration (less cluttered)
...
*) it is possible to move/edit/delete more than one entry at a time now
*) it is easier to choose a target for blacklist import now
*) fixed several bugs
*) to be continued...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5330 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ef66438662
- more space in error db to store larger error messages
...
- added hash to HTCACHE storage files which will make it possible to join separate caches by just copying files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5329 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
674ad2d55b
different handling of error cases that occur during loading files with http or ftp:
...
methods throw exception instead of returning an error string
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5328 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
538359a0ff
simple fix to get DHT working again (maybe something more has to be done ;)
...
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1578
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5327 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
8ba0c9d1e9
cosmetics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5326 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
a94b3be80b
- modern ui for Windows installer
...
- installer has native system language now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5325 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
ae80f3e6a5
* extend opensearchdescription to support compare_yacy.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5324 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
7e1fe05e3c
* added utf8-encoding to many getBytes-calls
...
* utf8 should work now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5323 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fad044fb54
update to snippet marker:
...
- do not display indexed html (solves xss issues)
the single words are analyzed for already marked parts. this is needed to avoid false encoding of the marker (<b>) tags.
- improved speed for existing routine
heavy used regex pattern are precompiled now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5322 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago