orbiter
1db9cdd4e4
fixed bug in writing of robots.txt entries in case that host names exceeded 64 characters and some other problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5918 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
bde88b684a
* splitt off yacyRelease from yacyVersion
...
* added some gui infos about signatures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5916 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1351d903a1
don't follow links like mailto:
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5909 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
eacf95213a
fix for crawling of mailto-links
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5906 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d3f8aa5a2a
set of small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5903 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d31e6f9c14
fix for http://forum.yacy-websuche.de/viewtopic.php?p=14457#p14457
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5899 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
58802e4201
added missing success test in storeDocumentIndex,
...
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1922&hilit=
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5862 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
b08991e278
moved some constants, rename of Tray class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5858 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
138422990a
- removed useCell option: the indexCell data structure is now the default index structure; old collection data is still migrated
...
- added some debugging output to balancer to find a bug
- removed unused classes for index collection handling
- changed some default values for the process handling: more memory needed to prevent OOM
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5856 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dfb96ecb72
more fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5844 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
0b0a46d35a
* fix transferRWI as suggested by celle (thanks!)
...
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2000#p14023
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5842 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
635b0a9da7
code-split
...
allow cgi indexing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5839 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2f860a2564
* convert byte[] hashes to string for log output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5830 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d93a2a6552
* ignore whitespaces so you can copy&paste signatures better
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5828 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
44daec7936
* introduce signatures to autoupdate
...
as long as there aren't publickeys for the updatelocations set,
no signatures are checked
* wiki-article follows...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5822 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c8624903c6
full redesign of index access data model:
...
terms (words) are not any more retrieved by their word hash string, but by a byte[] containing the word hash.
this has strong advantages when RWIs are sorted in the ReferenceContainer Cache and compared with the sun.java TreeMap method, which needed getBytes() and new String() transformations before.
Many thousands of such conversions are now omitted every second, which increases the indexing speed by a factor of two.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5812 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
89ec3acb3e
- full abstraction of index content type: the kelondro full text index may now also contain indexes about other content than text, i.e. navigation indexes or reverse linking indexes.
...
- during index joins all word positions are maintained: better ranking for word distance possible; exact phrase match can be implemented soundly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5804 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
7a48090fcf
- fix for "uk" language
...
- svn attributes added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5803 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c3aff2521e
fix for NPE
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5789 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
44e01afa5b
- refactoring
...
- a little bit more abstraction
- new interfaces for index abstraction
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5783 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c2359f20dd
refactoring: better abstraction of reference and metadata prototypes.
...
This is a preparation to introduce other index tables as used now only for reverse text indexes. Next application of the reverse index is a citation index.
Moved to version 0.74
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5777 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5b138ada16
fixes to web structure reference collection and url construction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5775 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7ba078daa1
- added fast site-operator
...
- refactoring merge into BLOBArray
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5770 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
587838bd09
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5758 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
96eaecda3e
- added migration class to go from index collections to the index cell data structure.
...
- added better control over file deletion, because this sometimes fails, especially on windows
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5756 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lulabad
df87e4dbf6
missing count of send Index and URLs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5747 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9a90ea05e0
added a merge operation for IndexCell data structures
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5727 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
83792d9233
more refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5722 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
cdbdc731c5
small updates: unescape, isCGI
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5720 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
209f25f5f5
refactoring to integrate indexCell data structures
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5718 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
359a238acf
faster isCGI()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5717 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
f75628e53b
some corrections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5716 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7dff1cba62
removed option to use different primary keys in kelondro tables
...
this option was never used and there is also no use to set other columns but the first as the primary key. as a result, access methods to the key do not need to compute key positions, and they work faster.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5711 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7f67238f8b
refactoring of plasmaWordIndex: less methods in the class, separated the index to CachedIndexCollection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5710 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
14a1c33823
refactoring of wordIndex class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5709 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
13c666adef
performance hack to ObjectIndex put() method:
...
Java standard classes provide a Map Interface, that has a put() method that returns the object that was replaced by the object that was the argument of the put call. The kelondro ObjectIndex defined a put method in the same way, that means it also returned the previous value of the Entry object before the put call. However, this value was not used by the calling code in the most cases. Omitting a return of the previous value would cause some performance benefit. This change implements a put method that does not return the previous value to reflect the common use. Omitting the return of previous values will cause some benefit in performance. The functionality to get the previous value is still maintained, and provided with a new 'replace' method.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5700 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
hermens
8c60d6d117
In DHT selection delete only those references that were actually selected
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5693 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3b28daab40
code-beautification (to be consistent with external documentation paper)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5686 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
efcd95dc37
simplification of (internal) query process / refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5671 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
aa44d9bad9
more refactoring of kelondro.text / deleted de.anomic.index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5664 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6ffc6e3389
more refactoring of indexer and kelondro classes;
...
- integrating the indexer into kelondro as package 'text'
- renaming of classes in kelondro.index
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5663 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
76ef5f0f14
refactoring of index package: better names for the classes (to be continued)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5661 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c728879ab8
fixes to yacyURL - more exceptions in case that urls are strange
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5657 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7542336ae5
performance enhancement to yacyURL: omit second processing of resolveBackpath. This method is already applied during initialization of the object and was called a second time when the url was exportet.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5656 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e521e81148
bugfix in yacyURL (for latest performance hack)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5654 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
54625360f7
performance update
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5653 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
46632f4385
performance update to yacyURL
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5651 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8444357291
added new row interator in kelondro tables files that enumerates rows
...
without an order by the primary key. The result is a very fast enumeration of the Eco table data structure. Other table data types are not affected.
The new enumerator is used for the url export function that can be accessed from the online interface (Index Administration -> URL References -> Export). This export should now be much faster, if all url database files are from type Eco
The new enumeration is also used at other functions in YaCy, i.e. the initialization of the crawl balancer and the initialization of YaCy News.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5647 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
6117e083e5
option to customize tray label (tooltip) with tray.label
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5642 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
9519d84372
changed "dooble" variable to "browserintegration" to be less specific
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5636 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
8429083972
adjusted tray for dooble:
...
you can now set dooble=true in yacy.init to disable the menu and browser popups by default
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5633 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c852d2d70e
- reject too old seeds
...
- do not store the complete seed in the reverse name cache, only the hash of the peer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5628 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4f9dae2571
remove reference in crawl entries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5623 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c12bb8a6d0
- refactoring of the http client
...
- added a protection against memory leaks for the access tracker
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5621 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
62505bb3cb
more bugfixes as recommendet by findbugs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5619 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
94c42691d8
- reject less transmissions as transmission receiver
...
- do not flag too much receiver when something goes wrong during transmission as sender
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5616 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
26978b2a25
- better memory protection in kelondro caches: computation of needed memory for cache grow
...
- removed excessive gc calls
- step to 16 vertical DHT partitions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5611 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
be0c492ae5
fix for memory leak bug in new dht transmissions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5606 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
40d9849aa4
- better control of chunk size in dht selection
...
- more restrict values in selection
- step to 4 vertical partitions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5603 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
411f2212f2
more memory leak fixing hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5599 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
985d421f91
found and fixed some memory leaks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5596 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6c627dbdff
update to the server core
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5591 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5393f356aa
fix for termination problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5589 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6a876ecb88
first fixes to the DHT transmission process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5588 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c25c334b75
replaced old DHT transmission method with new method. Many things have changed! some of them:
...
- after a index selection is made, the index is splitted into its vertical components
- from differrent index selctions the splitted components can be accumulated before they are placed into the transmission queue
- each splitted chunk gets its own transmission thread
- multiple transmission threads are started concurrently
- the process can be monitored with the blocking queue servlet
To implement that, a new package de.anomic.yacy.dht was created. Some old files have been removed.
The new index distribution model using a vertical DHT was implemented. An abstraction of this model
is implemented in the new dht package as interface. The freeworld network has now a configuration
of two vertial partitions; sixteen partitions are planned and will be configured if the process is bug-free.
This modification has three main targets:
- enhance the DHT transmission speed
- with a vertical DHT, a search will speed up. With two partitions, two times. With sixteen, sixteen times.
- the vertical DHT will apply a semi-dht for URLs, and peers will receive a fraction of the overall URLs they received before.
with two partitions, the fractions will be halve. With sixteen partitions, a 1/16 of the previous number of URLs.
BE CAREFULL, THIS IS A MAJOR CODE CHANGE, POSSIBLY FULL OF BUGS AND HARMFUL THINGS.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5586 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
65a1de6c05
longer timeout for remote crawl queries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5573 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
78b7361937
fixed problem with logging
...
YOU MUST DELETE DATA/LOG TO MAKE THIS WORK! (sorry..)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5552 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
94110df85a
moved logging partially to kelondro
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5545 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
024da2916b
refactoring of logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5544 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
83ce65707a
(almost) completed partition of classes in kelondro
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5543 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7ee494fde5
more refactoring of kelondro:
...
- seperated BLOB from table classes
- renamed 'coding' package to 'order'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5542 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bf93767ec6
refactoring of kelondro database classes
...
(to be continued)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5540 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc27bf8c4c
refactoring of kelondro classes:
...
kelondro shall become independent from other packages.
moved bytebuffer, date and memory to kelondro
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5539 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
aaafe05c02
* revert debug change
...
* contains instead of startsWith, because there might me localizied strings
* decode punycode for every domainpart seperately (see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1749 )
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5516 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
419469ac27
added more methods to control the vertical DHT (not yet active .. )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5514 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dedfc7df7f
removed distinction between DHT-in and DHT-out. This is necessary to make room for the new cell data structure, which cannot use this this distinction in the first place, but will enable the same meaning with different mechanisms (segments, later)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5511 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5080fc33bf
fix for http://forum.yacy-websuche.de/viewtopic.php?p=12247#p12247
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5506 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
78778df464
*) this should adjust the Dev/Main detection of the updater to the new version numbers (0.7x is Dev, if x != 0)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5504 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9d119c6b61
migration of auto-update rules to new release strategy:
...
next stable will be 0.7, development releases are 0.*x, experimental will be if x = 1, 2, 3
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5458 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
c8451614f3
fix for overflow
...
http://forum.yacy-websuche.de/viewtopic.php?p=11696#p11696
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5440 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c4c4c223b9
fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6072831235
no cr transmission for robinson peers
...
see also: http://forum.yacy-websuche.de/viewtopic.php?p=10290#p10290
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5436 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e004da48d3
- added fast fingerprint computation for files (any). Will be used in new index dump method
...
- refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2d2ce24011
* remove all encoding-stuff from proxy
...
encoding is handled by parsers or browser, proxy only passes through
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
449e697436
fix for null-seed in seedfile
...
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2802138787
- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)
...
- fix of shallBeOwnWord target computation heuristic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1779c3c507
- added a read cache to the RAFile interface to RandomAccessFile
...
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
47292e696a
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d39d420b39
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c6525ab75f
fix for NPE in seed handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
538359a0ff
simple fix to get DHT working again (maybe something more has to be done ;)
...
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1578
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5327 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
7e1fe05e3c
* added utf8-encoding to many getBytes-calls
...
* utf8 should work now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5323 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3f746be5d4
- consolidation and refactoring of many DHT target - computing methods
...
- implemented vertical DHT acceptance ("my own DHT") to accept new targets
- added new target computation for global search: addresses vertical targets also
- enhanced remote crawling: collection of remote crawl urls if queue has less than 100 entries (was: 0 entries)
- better performance value computations for PPM selection in network configuration
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5319 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d014b2728a
Design-check, Extension and Refactoring of DHT target position computation:
...
- two different computations (but mathematical equivalent) of the DHT distance had been consolidated
- moved from 0.0 .. 1.0 double-range position computation to 0 .. Long.Max range for DHT targets
- added fast Long - to - hash computation
- high-precision target computation of gaps for new peers
- added new target computation for horizontal and vertical DHT targets (not yet in use)
- old horizontal-only DHT targets will be upwards compatible to new horizontal and vertical DHT positions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5318 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ffed5fc415
fixed problem with lost peers in database
...
migrated seedDB from BLOBTree to BLOBHeap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
6fb865fbdc
- fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing
...
- some refactoring of classes that use kelondroMap (Map instead of HashMap)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
820a03f9d6
- removed some warnings
...
- used fix in SVN 5233 for ysearch.java and search.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
dda771db9d
- search result layout
...
- tray only for windows
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
31c31e54e4
new tray icon image for different icon sizes (e.g. linux)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5216 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
9589dfe080
* removed trayicon popupmenu title
...
* added some menu items to trayicon
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5213 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago