low012
00e27e5050
*) fixed bug which made it possible to write files outside of the DATA/LIST directory when creating a new blacklist
...
*) a blacklist will only be created if no blacklist with same name exists (some refactoring has been necessary for this)
*) further minor fixes
*) to be continued...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5301 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
0f9c0bd0d5
fix for ConcurrentModificationException at de.anomic.index.indexContainerHeap$heapCacheIterator.next(indexContainerHeap.java:324)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5300 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
103ad2a437
some javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5299 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b098522977
some very small advances to index utf-8 (not working yet), inserted also debugging code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5298 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2f49666908
integrated the character decoding into the parser, removed old code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5297 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
49293c1358
fix for deadlock in new encoder :-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5296 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0edec2b760
FULL redesign of algorithms in htmlTools to encode/decode strings from/to unicode and html.
...
The old process used a not really efficient way to detect html encoding strings in texts.
All calling methods had been adoped to call the new class in an enhanced way with less parameters.
Many classes in interfaces used a XML encoding only (instead of full html conversion from unicode to html); this behavior was not changed with this commit but should be controlled again since it points out possible XSS leaks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5295 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
958ec20cd0
removed specialized umlaute-handling in html parser. This has to be replaced by something that is able to transfer all possible html encodings into utf-8. Please see SVN 5293 for test cases.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5294 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
204220ecd5
added test files for UTF-8 / Umlaute - Testing:
...
These 3 files contain the same text in different HTML encodings. We use this documents to test if the parser and indexer creates the same set of word hashes for all three texts.
To use these files, run a indexing/crawling on them. To get the files inside the localhost-path, do the following:
cd <yacy-home>
rmdir DATA/HTDOCS/repository
ln -s test/parsertest DATA/HTDOCS/repository
you have then linked the test directory as repository directory which you can reach in yacy if you switch to intranet indexing mode. So the next step is to start yacy, then
- switch to intranet use case
- go to the crawl start page
- the repository directory should be the default path as crawl start
- start the crawl
- search for any word that appears in the demo texts
- search not only for words with umlautel but also for words without umlaute to ensure that you find _all_ three documents
- see how yacy presents the snippet with the text containing umlaute
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5293 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2e53cbc66a
should compile now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5292 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
f3bf2e379e
should compile again
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5291 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
dd8441f102
fix bug: data from plasmaParser is allready converted to UTF-8
...
After removing the restrictions in the code, YaCy should be able to index Unicode-charaters!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5290 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
47f0c3b002
replaced the cacheAdmin with the ViewFile servlet, because the cacheAdmin was an interface to the old HTCACHE data structure which does not exist any more. Changed links to point to the ViewFile servlets.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5289 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6941bf42b1
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5288 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9b0c4b1063
redesign of parts of the new BLOB buffer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5287 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1778fb420d
- added some performance tweaks to the new BLOB buffer
...
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
77e41da7d2
*) further propagation of display value (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536 )
...
*) removed another depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5285 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9663e61449
added another class to handle BLOB writings to the new HTCACHE data storage:
...
- entries are buffered and written as stream with many entries at once (saves many IO accesses)
- entries are compressed with gzip: increases capacity of cache
- concurrency for stream-writing and compression: all writings to the cache are non-blocking
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5284 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
ff46ce8520
*) fixed display=2 (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5283 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
382226da94
fix for bug introduced in SVN 5281: parameters were switched
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5282 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
f2fd043797
refactoring (moved duplicate code into methods)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5281 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
c612046e5e
r5278 java 1.5 compatible
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5280 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
af71ec93bf
ops, forgot to import something
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5279 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
9e65e9141c
* always use UTF-8 for encoding hashes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5278 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
03d2b323c2
* remove bad mimetype tag so I see my changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5277 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
826ca79735
refactoring and new architecture to store the files of the web cache:
...
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods
this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
2b18a9b2c4
*) removed depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5275 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
f095137238
- respecting httpdMaxBusySessions (refusing new connections if limit is hit)
...
- comments in serverBusyThread converted to JavaDoc
- better debug output for npe-case in diskUsage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5274 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
94e43ece41
unter debian yacy als (fast) letztes starten und erstes beenden
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5271 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
9d69964d3d
start daemon in different runlevels on debian and fedora/openSUSE-Systems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5270 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8ba33f104e
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5269 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
998861acfd
- some refactoring in BLOBHeap to enable more gap processing functions
...
- better gap merging in BLOBHeap
- shrinking of heap file if gap is at end of file when file is closed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5268 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
9d50bfd0b3
fix for npe: http://forum.yacy-websuche.de/viewtopic.php?p=10562
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5267 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
766cad6e93
enhancement in memory management of BLOB Heap files / merging of deleted entries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5266 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7860d5d632
fix for bug in seed list management (cause was bad class overloading, only visual effects!)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
603282bcf4
fix for out of bounds exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5264 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ffed5fc415
fixed problem with lost peers in database
...
migrated seedDB from BLOBTree to BLOBHeap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6fb865fbdc
- fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing
...
- some refactoring of classes that use kelondroMap (Map instead of HashMap)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2d65887723
- fix for bug in new profile handling
...
- added a new feature in ymageChart (cannot be seen yet, just wait... will be used in profiling chart)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5261 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
4df63626f5
sorry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5260 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
736dd86193
- option enableSimpleConfig can disable hidden tables
...
- corrected some Xmx values
- friendlier welcome message format
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5259 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ff68f394dd
fix for problem with balancer and lost crawl profiles:
...
if crawl profile ist lost, no robots.txt is loaded any more
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d197a62faf
* wrong default parameter in initscript
...
* should fix http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1500
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5257 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
f0c7166a48
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1505
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5256 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
88fdd4c4f2
* remove unix wrappers from installonlinux-target, because they are useless on systemwide installations
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5255 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
daburna
9286afad4e
new development cycle - nun aber wirklich :-P
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5254 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
ed51ccf725
* correct error message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5253 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
fe6142a6ab
*) ...and even more languages.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5252 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
3717d2057a
YaCy-UI: fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1483
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5251 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
9f0cc3afdd
*) more languages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5250 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago