orbiter
3e9d509c39
some small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2425 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
276225d79e
fix for URL class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2423 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
eb633c0a4f
server threads must now supply a method that can be called in case
...
of short memory. This has been realized for the indexing thread.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2421 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f5720cb2fa
removed most synchronization in wordIndex (for testing)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2420 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0187c60010
because of a bug in the JRE 1.4.2 there was no memory protection
...
see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4686462
this commit fixes the bug by using a memory-computation patch.
All uses of Runtime.maxMemory had been replaced by serverMemory.max
The bug is not present any more in Java 1.5
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2419 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
auron_x
4eca0f8830
*) fixed PPM calculation for multiple indexer-threads
...
*) fixed totalPPM calculation and added total PPM to Network.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2418 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cfb51fdef1
less synchronization in plasmaWordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2416 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d6a928c2da
quickfix for http://www.yacy-forum.de/viewtopic.php?t=2705
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2415 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6ad471ef96
* applied many compiler warning recommendations
...
* cleaned up code
* added unit test code
* migrated ranking RCI computation to kelondroFlex and kelondroCollectionIndex
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2414 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
cf1186597b
utf fix from theli
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2412 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
9da3aa74d3
silly me, fix for the fix as advised by theli
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2408 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
bb3d9a5582
*) e.getMessage().indexOf() can only be used if there is actually an ExceptionMessage.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2407 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
7a54010a9c
*) Iterators can't be casted to IndexContainer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2406 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
5e0b6f8f83
*) sorting peer name list on Blacklist_p.html
...
*) restructuring of sharedBlacklist_p.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2405 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cd5f7e137c
fixed problem with NURL-generation upon first startup
...
(a new kelondroFlexTable was generated, which should not)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2402 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8418af141a
added several consistency checks and small changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2400 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9d13aeca13
*) removing class. does not work so far
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2399 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
95a84ae469
*) adding missing classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2398 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
eee44be602
*) adding an interface for customized blacklist classes
...
- now it's possible to use a customized blacklist engine
instead of the default one
- this can be done by configuring the property BlackLists.class
See: http://www.yacy-forum.de/viewtopic.php?t=2108
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2397 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6d2f15971a
there is a very strange error that causes that the kelondroRecords structure
...
is corrupted. The cause is, that the deleted-records-chain has wrong entries,
and one of the pointers in that chain points to a place behind the file end.
This causes an IndexOutOfBoundsException within an IO operation.
I currently don't know the reason that the deleted-records-chain is
corrupted, but the error can be catched. If this now happens with the
assortment database, the database is deleted.
See also:
http://www.yacy-forum.de/viewtopic.php?p=24586#24586
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2396 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d2e8e76218
*) now it's possible to configure the yacy blacklist separately for dht, search, proxy, crawler
...
See: http://www.yacy-forum.de/viewtopic.php?t=2541
http://www.yacy-forum.de/viewtopic.php?p=24516
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2389 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9ae9062bd3
* disabled new kelondroFlex table for NURLs
...
* added new RAM index Class
* fixed possible synchronization problem in kelondroRecords
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2388 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
689bbcf9cd
replaced kelondroTree db for NURLs by new kelondroFlexTable
...
The new database is only created if the old is deleted or does not exist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2387 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7fbba41962
synchronization fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2386 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
328f9859a5
more synchronization in plasmaWordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2385 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f43c90fa98
fixed handling of null referer in crawlOrder
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2384 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
130e6d4719
generalized index object for eurl, nurl and lurl to prepare move
...
of these tables to new kelondroFlexTable Object
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2382 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
acdf24877f
more synchronization against outOfMemoryError in wordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2381 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
95160d7f2c
fixed size computation of index elements from the collection index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2380 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
26116cabde
added missing rowdef assignment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2379 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cfbacbbf08
reverted change in robotsParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2378 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
abf22f6e60
removed url normalform computation from htmlFilterContentScraper.
...
This method was implemented in de.anomic.net.URL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2377 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
740d49751d
* strict type and size check in kelondroRow handling
...
* adopted all code to use the declaration form of kelondroRow
* fixed a bug in kelondroRow which caused wrong parsing of encoding type
* the bug caused bad database behaviour in new indexCollection data structure.
because of this bug, all test databases are now already void. A new database is created
* the kelondroFlexTable and indexCollection data structures now store a declaration of the row definition
into a properties file along the database files.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2375 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
314021453f
* more logging
...
* option in yacy.init to set useCollectionIndex usage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2374 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
a52f36787f
better templatedebugging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2371 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
3480d36417
added some debug code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2369 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
61b151b083
* added another auto-fix for collection index inconsitency check
...
* fixed words size computation for collection index
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2368 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0bbbd129ef
small fix for exception message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2367 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
718fbc2dae
enhancements in kelondroCollectionIndex:
...
* synchronized array and index objects
* auto-fix function for slightly corrupted index entries
* generalized internal access methods
also extended kelondroIndex interface to support ordering access
which is used in kelondroCollectionIndex for string comparisments
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2366 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f58283def2
better control of index flush
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2364 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4be21a3cab
ups
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2363 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
80b6c90d54
enhancements to prevent blocking during dht transfer receive
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2362 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9f298083cd
*) adding more urls to the error url
...
- old error strings where replaced with there corresponding constants
See: http://www.yacy-forum.de/viewtopic.php?t=2638
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2360 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
d56f06401e
- Cache known URLs during indexReceive to avoid getting blocked during loadedURL.exists() whenever possible
...
- Small logging updates
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2359 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c09f734d06
*) offer router configuration on ConfigBasic.html
...
- checkbox to allow router configuration is shown if
- a) the UPnP forwarder is installed
- b) a UPnP enabled router was found
- c) no other forwarder was configured
See: http://www.yacy-forum.de/viewtopic.php?p=24264
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2358 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
dcbb4d0a6b
Display the size of HashBlacklistedCache on PerformanceMemory page.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2357 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d799622da1
better flush limit for index collections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2354 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d468d665c9
some changes that may help to prevent deadlocks that cause an OutOfMemoryError
...
as described in
http://www.yacy-forum.de/viewtopic.php?p=24359
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2353 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d54767f634
*) last step of removing embedded html from dir class
...
- migration finished
*) dir list now sorts the dirlist entries.
- directories are listed before files
- files are sorted alphabetically, case insensitive
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2351 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
279b1d969d
Integrated new indexing data structure 'collections' into the main class
...
for indexing, the plasmaWordIndex.
The new data structure is ready-to-use, but currently disabled.
It can be activated by setting the static
plasmaWordIndex.useCollectionIndex
to true. This shall be done for testing purpose.
The new index is stored to
DATA/INDEX/PUBLIC/TEXT
The directory PLASMA shall be used only for crawler in the future.
Attention: during testing the data structure in INDEX may change,
and created indexes with the new data structure may get useless.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2348 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago