orbiter
6be57983a8
another update to the crawl balancer
...
can now alternate between top and bottom of the crawl stack
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3487 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
91cdc1493f
removed query to NAT or responder in case that no other peer is there.
...
this is not needed any more, there are enough peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3486 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4783a30910
- fixed a flush problem in balancer
...
- return to idle divisor in RWI RAM cache flush
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3485 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
91c2a042a7
*) bugfix for wrong proxy traffic accounting
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3484 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
861f41e67e
redesigned NURL-handling:
...
- the general NURL-index for all crawl stack types was splitted into separate indexes for these stacks
- the new NURL-index is managed by the crawl balancer
- the crawl balancer does not need an internal index any more, it is replaced by the NURL-index
- the NURL.Entry was generalized and is now a new class plasmaCrawlEntry
- the new class plasmaCrawlEntry replaces also the preNURL.Entry class, and will also replace the switchboardEntry class in the future
- the new class plasmaCrawlEntry is more accurate for date entries (holds milliseconds) and can contain larger 'name' entries (anchor tag names)
- the EURL object was replaced by a new ZURL object, which is a container for the plasmaCrawlEntry and some tracking information
- the EURL index is now filled with ZURL objects
- a new index delegatedURL holds ZURL objects about plasmaCrawlEntry obects to track which url is handed over to other peers
- redesigned handling of plasmaCrawlEntry - handover, because there is no need any more to convert one entry object into another
- found and fixed numerous bugs in the context of crawl state handling
- fixed a serious bug in kelondroCache which caused that entries could not be removed
- fixed some bugs in online interface and adopted monitor output to new entry objects
- adopted yacy protocol to handle new delegatedURL entries
all old crawl queues will disappear after this update!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3483 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
094a1482f4
*) removing yacy.exe on ant clean
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3482 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
832662ccd2
*) removing yacy.jar on ant clean
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3481 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
9b5fb3908d
*) a peer-message are now created when a blog-comment is written
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3480 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
581db87237
more debug code for
...
http://www.yacy-forum.de/viewtopic.php?p=33009#33009
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3479 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
81c4cc6bf7
better debugging of balancer failure
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3478 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dd06d4cada
more logging to better trace bug
...
http://www.yacy-forum.de/viewtopic.php?p=33001#33001
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3477 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
96b79bf86d
redesigned remove method in kelondroRowSet
...
This should fix also numerous bugs like
http://www.yacy-forum.de/viewtopic.php?p=31077#31077
(java.lang.ArrayIndexOutOfBoundsException in kelondroRowCollection.removeShift)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3476 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9f929b5438
better snippet handling in case of snippet load fail
...
see also http://www.yacy-forum.de/viewtopic.php?p=31096#31096
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3475 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
d451ad48d3
*) improved peerloadgraphic:
...
- unnecessary (0 %) pieces are removed
- percent-values of each thread displayed in legend
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3474 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a5d668c0c6
added speed-buttons for easy performance setting
...
appears in crawl start and on indexing monitor page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3473 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5b0a84ce09
fix for synchronization deadlock with flushMissNameCache.
...
see also: http://www.yacy-forum.de/viewtopic.php?p=32939#32939
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3472 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
e2ac5f62bd
- Code hübscher machen [von NNs TODO]
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3471 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
f04097c3dd
integrated tor-patch for crawling, if yacyDebugMode is set.
...
(replaces: http://yacy.deruwe.de/overlay/net-misc/yacy-tor/files/disable_dns_checks-svn3132.patch )
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3470 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
22fe14f292
*) first version of Peerload-graphic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3469 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
432d7d4e9c
better catch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3468 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8f7e8b6ee2
auto-delete for not-fixable db error in crawl stacker.
...
see also http://www.yacy-forum.de/viewtopic.php?p=32906#32906
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3467 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7a52b07fcc
better memory protection during freemen cycle
...
see also http://www.yacy-forum.de/viewtopic.php?p=32903#32903
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3466 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6faa262259
fix for NURL-fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3465 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
909d7a8ae9
fixed wrong implemented row iterator in kelomdroFlexSplitTables
...
this has no effect, until now this iterator was only used on
the Index Administration page.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3464 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a1fb8358b2
lets make a well-formed http link so that other crawlers don't have a problem to follow this link :-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3463 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4edb70f68b
added yacybot info-page from Roland
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3462 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3ef77d2030
fix for http://www.yacy-forum.de/viewtopic.php?p=29878#29878
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3461 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3bb3df3fc0
fix for http://www.yacy-forum.de/viewtopic.php?p=32298#32298
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3460 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b3ca177a5d
fix for http://www.yacy-forum.de/viewtopic.php?p=32797#32797
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3459 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
243a2f831b
fixed problem with not found NURL-hashes
...
The cause for this problem could still not be found, but the effect
is handled much better. The NURL-pop will continue automatically until
it found a hash that can be found.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3458 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6ad39bae1e
fixed shutdown problem
...
this fixes the 'inconsistency' messages during start-up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3457 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
38b93f8cb8
bugfix for my last commit:
...
iterator did not consider secondary start point in case of rotation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3456 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
264a82eec8
- fix for http://www.yacy-forum.de/viewtopic.php?t=3657
...
- fix for http://www.yacy-forum.de/viewtopic.php?p=32758#32758
- Diff takes any objects now, not only strings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3455 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
045d758537
Avoid stopwords as topwords, configurable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3454 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d755a8026d
- better OOM protection
...
- better memory allocation for FlexTable indexes
- splitting between static index and dynamic index (only the dynamic part must grow)
- to enable a merge-iteration of new splittet index, a huge number of classes needed to be adopted for new iterator classes
- added new iterator classes that support cloneable iterators
- adopted all iterator classes to implement cloneable itarators
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3453 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
2be405e1e1
- fix for last two commits
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3452 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
de1b4a1731
- don't publish news if empty or equal page is submitted in wiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3451 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
dcc13abd59
- fixed small bug at home page, button "peer's console"
...
- fixed <fieldset><dl> for safari on many pages
- added Blog-link to Network page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3450 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
6596167277
*) bugfix for wrong RSS feed pubDate formats
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3449 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
0d178d00a5
*) adding RSS feed for peer messages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3448 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
23338d2070
small fix for RAM computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3447 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
33f97cff7a
changed startup initialization sequence slightly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3446 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
4f2e6ef47b
- WatchCrawler_p shows max. 80 characters of URLs now (maybe dynamically adjustable based on browser width?)
...
- typo in BlacklistCleaner
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3445 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
70cd391ea1
fix for dl/fieldset problem in Safari
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3444 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5741701b59
moved crawl start up, personal web pages down in main menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3443 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b627c77df6
- workaround for safari bug with definition lists inside fieldsets in ConfigBasic
...
- alternative can be seen in PerformanceMemory, where a dl is simulated with a table layout
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3442 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4e8eb1dbe3
some minor changes here and there
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3441 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
03c5906ae7
- minor bugfixes for url-fetcher & http://www.yacy-forum.de/viewtopic.php?t=3646
...
- PerformanceMemory_p.html is valid XHTML again
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3440 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3499a364ef
a little bit better memory protection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3439 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
313f6a7680
fix for http://www.yacy-forum.de/viewtopic.php?p=31553#31553
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3438 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago