orbiter
92f4cb4d73
added option to configure the start-up delay time for kelondro database files.
...
the start-up delay is used to pre-load the database node cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2276 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ce9dd3e76d
some work in the index construction zone (no effect yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2275 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fe617d7e54
*) adding function to return the protocol type of a ssl connection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2274 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
018b3e0832
added pause option to server threads.
...
The pause is started by calling intermission(Long.MAX_VALUE)
and can be stopped by calling intermission(0)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2272 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e1a52bea22
added a class stub for the new database structure:
...
a reverse word index based on a a collection index,
which is an index for a set of array files containing
row collections.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2271 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3b69b35bf2
added pre-load of node cache entries to kelondroRecords
...
this gives the kelondroTree data structure a similar start-up
behaviour like the kelondroFlexTable: the cache is filled with
routing data in such a way that is more performant than
reading node records during normal operation.
The pre-load phase stops automatically after a time-out of 500 milliseconds
of if the cache is full.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2270 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
85d575e928
enhancements to kelondroRow and kelondroColumn
...
these are changes towards a better indexURLEntry implementation
which are needed for the new database structures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2268 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ab1ed053f5
another small correction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2267 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b92561fb67
removed unused code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2266 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
eadbd56fc5
small adjustment to last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2265 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e9765ac4e6
introduced bulk read for node iterator in kelondroRecords
...
this speeds up the iterator by factor 2
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2264 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6643da3fbd
bugfix for http://www.yacy-forum.de/viewtopic.php?p=23463#23463
...
(affected URL DB Cleaner)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2263 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
866d53ed70
fix for DNS block bug
...
see http://www.yacy-forum.de/viewtopic.php?p=23458#23458
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2262 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6af70febef
- added kelondroTree index option to kelondroFlexTable
...
- automatic generation of index file when index is too large for RAM
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2261 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dd2865178a
major bugfix (searched a whole week for the bug) for
...
the kelondroRowBuffer, which has effect mostly to the
kelondroFlexTable but also to all other database functions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2260 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f9b9d085c4
just changed testing code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2259 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b594ee9a5a
*) Adding possibility to configure if the http proxy should send the
...
X-forwarded-for header (requested by TeeSee)
See: http://www.yacy-forum.de/viewtopic.php?t=2577
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2257 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ef84fc4956
added IOException to size() and row()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2256 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
84dfd76a6a
kelondroFlex bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2254 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
8ba8e2b7d9
*) added cache for blacklists urlhashs recieved by DHT. DHT does not request URLs listed in this cache.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2251 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
53cbcc6d6e
Implement emergency break in index receive when the limit of the ramCache is exceeded by more than cacheLimit
...
See: http://www.yacy-forum.de/viewtopic.php?p=22911#22911
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2248 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e40987ecab
removed default memory reservation for testing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2247 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4cc6e6551f
bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2245 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
66964dc015
removed high/med/low from kelondroRecords cache control.
...
this was done because testing showed that cache-delete operations
slowed down record access most, even more that actual IO operations.
Cache-delete operations appeared when entries were shifted from low-priority
positions to high-priority positions. During a fill of x entries to a database,
x/2 delete situation happen which caused two or more delete operations.
removing the cache control means that these delete operations are not
necessary any more, but it is more difficult to decide which cache elements
shall be removed in case that the cache is full. There is not yet a stable
solution for this case, but the advantage of a faster cache is more important
that the flush problem.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2244 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6866bc2758
be quiet!
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2243 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
4c6083b264
network picture;
...
back to old version
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2242 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
955915385a
network picture;
...
small changes;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2241 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
027fa8ab1c
network picture;
...
bigger;
more dot steps;
small other;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2240 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
41c4641612
added some profiling to kelondro caching classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2239 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dd560e4b2f
finetuning
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2238 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5b1d77cd4b
some enhancements to caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2236 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b20496e42b
*) make DHT DoS check configurable (requested by KoH)
...
- check can be disabled via property indexDistribution.dhtReceiptLimitEnabled
- upper bound can be configured via indexDistribution.dhtReceiptLimit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2234 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
650c7e9e55
some enhancements to caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2233 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
d4645062bc
Correct usage of vhost in wget/wput requests:
...
- yacyClient: don't use own .yacyh domain in requests, instead use .yacyh domain of target peer for everything but ranking distribution
- natLib: use full hostname instead of just SLD.TLD
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2232 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bde748f68f
- integrated buffer in RowCollectionSet which speeds up the new
...
cache by factor 3. (but the new cache was 5 times slower than the
old cache...)
- removed old unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2231 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d083b15c4b
tests/fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2226 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ce9a98ee25
experiments with buffer for collection class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2225 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6aa0f6eb0f
better abstraction, less synchronization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2224 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1ec07fe351
better structure of the kelondro collection classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2223 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
c264cda678
support to parse gettext files and to generate updated files, where only the new strings are empty.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2222 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
761d3bba54
integrated kelondroRowCollection in kelondroCollectionIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2221 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8f0c1a240a
added kelondroRowCollection and kelondroRowSet which
...
shall replace kelondroCollection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2220 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
ed2cb040d1
*) Bugfix for http connection header validation
...
- Connection header was not handled correctly if it contains
multiple values, e.g. Connection: TE, close
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2219 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f95c006dc2
generation of empty gettext files from yacy html-files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2218 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4fa3271754
adopted memory calculation to new memory amount used by kelondroCollectionObjectMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2215 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
0621106ef3
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2214 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
11951aed41
replaced node cache Object type in kelondroTree:
...
kelondroCollectionObjectMap replaces HashMap.
the kelondroCollectionObjectMap uses only 1/4 of the memory of HashMap,
but is slightly slower than the HashMap.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2213 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
12af69dd86
cosmetics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2212 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ea72a7c580
added new objectMap (can replace HashMap and should use less memory, but is slower)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2211 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
67a8c74be3
Fix for dynamic login with static password.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2210 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago