orbiter
bb935fdbb0
less organization overhead for DNS caching and prefetching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
4907697cfa
* make fileuploads through proxy bigger than 65500 bytes possible
...
* remove gzip-encoding for files from cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc8189f3fb
better self-healing of corrupted databases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
963da8c3f9
* updated tm-extractors to new version 1.0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
51f1a1927c
* remove saaj.jar and axis.jar and references to it (was for soap-stuff?)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5404 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
5a89266598
*) new parameters for future use (better blacklist handling for im- and export)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5403 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e34ac22fbd
- added new monitoring servlet at
...
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
449e697436
fix for null-seed in seedfile
...
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d376d81fc4
replaced busy thread control of crawl stacker by blocking threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f29b48d9ff
patch for IndexOutOfBoundsException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
0881190b19
* Robots.txt: don't interpret Crawl-Delays for other robots
...
fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
243e73f53b
removed unnecessary usage of kelondroBLOBTree
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8cb7170b75
- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated
...
- removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7535fd7447
- refactoring of CrawlEntry and CrawlStacker
...
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
6569cbbec1
npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
...
(break to avoid bad side effects)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5394 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
18513e2ee2
npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2802138787
- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)
...
- fix of shallBeOwnWord target computation heuristic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
b1e211b258
no error-alert: http://forum.yacy-websuche.de/viewtopic.php?t=1639
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5391 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
13cb0916ee
changes to statistics and content of thread dump servlet
...
(points now more directly to performance leaks without mentioning class calls inside of sun/java calls that cannot be changed anyway)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5390 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
db6b3bf5a3
speed enhancement for integrated http server:
...
- tuning hacks in template engine
- bypassing the template engine if no servlet present
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7cd08bd5fb
fix for NPE in BLOBCompressor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5b94498643
fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1779c3c507
- added a read cache to the RAFile interface to RandomAccessFile
...
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e1acdb952c
fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
2c682d649b
- no stop shortcut (-> stop via tray)
...
- store registry keys on current profile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5384 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
e918d64c23
show hand-cursor an labels
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5383 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4a2dac659e
more speed hacks:
...
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
07d7653de1
update to JRE 6u11
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5381 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1fb518a5b4
display <String> etc.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5380 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
47292e696a
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
759cef23dd
fix for bug in kelondroAbstractRA.readFully
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5378 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bd1dc9cd5d
thread dump with statistics, a little bit of profiling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5377 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d39d420b39
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
5280ad638d
added basic performance page
...
other performance settings can be found on advanced settings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5375 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1a51d9fcfd
display proper values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5374 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0b4808ba3d
added new interactive search feature:
...
- during the user types search queries, the local database is searched
- results are presented interactively
This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
74a3d86114
fixed a error response that might present classified information
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5372 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c6525ab75f
fix for NPE in seed handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fea82b54ef
more contrast on search snippets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5370 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1951d30a62
addendum to last commit
...
handle words with length < 3 correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
325ba7bfb8
only query words with length > 2
...
this is not complete, yet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
489edb4473
improved pattern selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5367 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
e423fa9846
*) added method to only get file names in directory listing which match a filter
...
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
577b53aee6
added more search engines
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5365 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
7f4d411c0d
npe-fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5364 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
513179f404
changed interface to colletctionIndex and adopted all implementing classes:
...
do not return a result of a double-check when adding entries with addUnique
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5363 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9d64693cfb
reverting again the changes to new concurrent chunkIterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5362 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
45ad1c3dd5
- re-activated concurrent iterator for EcoFiles
...
- added javadoc for new concurrent intialization in kelondroBytesLongMap
- switched default value for commons storage to false
- version step
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5361 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2e2120046f
speed enhancement for BLOBHeap opening process
...
using concurrency of FileIO and content processing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5360 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1545e5440a
* index deletion: checkbox-confirmation
...
* watch crawler: less load on exhausted peers; wait for data before reloading again
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5359 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago