borg-0300
4191b21e73
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@858 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
f6cf3967de
fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a2fa75e688
*) Asynchronous queuing of crawl job URLs (stackCrawl)
...
various checks like the blacklist check or the robots.txt disallow check are now
done by a separate thread to unburden the indexer thread(s)
TODO: maybe we have to introduce a threadpool here if it turn out that this single
thread is a bottleneck because of the time consuming robots.txt downloads
*) improved index transfer
The index selection and transmission is done in parallel now to improve index
transfer performance.
TODO: maybe we could speed up performance by unsing multiple transmission threads in
parallel instead of only a single one.
*) gzip encoded post requests
it is now configureable if a gzip encoded post request should be send on
intex transfer/distribution
*) storage Peer (very experimentell and not optimized yet)
Now it's possible to send the result of the yacy indexer thread to a remote peer
istead of storing the indexed words locally.
This could be done by setting the property "storagePeerHash" in the yacy config file
- Please note that if the index transfer fails, the index ist stored locally.
- TODO: currently this index transfer is done by the indexer thread.
To seedup the indexer
a) this transmission should be done in parallel and
b) multiple chunks should be bundled and transfered together
*) general performance improvements
- better memory cleanup after http request processing has finished
- replacing some string concatenations with stringBuffers
- replacing BufferedInputStreams with serverByteBuffer
- replacing vectors with arraylists wherever possible
- replacing hashtables with hashmaps wherever possible
This was done because function calls to verctor or hashtable functions
take 3 time longer than calls to functions of arraylists or hashmaps.
TODO: we should take a look on the class serverObject which is inherited from hashmap
Do we realy need a synchronization for this class?
TODO: replace arraylists with linkedLists if random access to the list elements is not needed
*) Robots Parser supports if-modified-since downloads now
If the downloaded robots.txt file is older than 7 days the robots parser tries to
download the robots.txt with the if-modified-since header to avoid unnecessary downloads
if the file was not changed. Additionally the ETag header is used to detect changes.
*) Crawler: better handling of unsupported mimeTypes + FileExtension
*) Bugfix: plasmaWordIndexEntity was not closed correctly in
- query.java
- plasmaswitchboard.java
*) function minimizeUrlDB added to yacy.java
this function tests the current urlHashDB for unused urls
ATTENTION: please don't use this function at the moment because
it causes the wordIndexDB to flush all words into the
word directory!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
01db66dc69
implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
f6a0e0f162
small bugfix to readFully
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@851 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
6d5d0ac801
bugfix for startup problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@850 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
5dc0d41900
bugfix in kelondroRA (hint by Martin)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@847 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0c3a20d44f
more + changed log for better understanding of outOfMemory bug and others
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e5f8163203
fixed a bug with news; news moving could lead to shurtcut loop / 100% CPU; appeared when clicked on a 'Profile' news in Network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@845 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
fbb5e36b80
documentation update
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@843 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
cd77078aa0
old Version restored before Release
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@842 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
a4b747fe97
ProxyAccounts based on userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@841 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
0fd9aa6c6e
*) Bugfix: supportedFileExt Function didn't detect the file extension correctly because of missing conversion to lower case
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@837 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
8a33c9b309
*) Bugfix: supportedFileExt Function didn't detect the file extension correctly if there was a dot
...
in one of the parent directories of the file.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@836 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
28c5687ff9
*) Bugfix for "download of non supported file content" via crawler
...
See: http://www.yacy-forum.de/viewtopic.php?p=10724#10724
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@835 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2b3f964037
*) Bugfix: supportedFileExt Function didn't chop http parameters before trying to detect the file extension
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@834 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
d388292f24
*) adding function for user accounting which is called after each http request
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@827 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
c7b7c69484
*) adding some functions to userDB.Entry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@826 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
d9c08a47f9
*) Minor changes to userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@824 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
af7829df25
*) adding some functions to userDB.Entry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@823 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
ff1d3d0680
Init of userDB
...
Pagelayout of User_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
19648702cc
*) adding iterator to userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@819 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
dd59c1ace9
*) Adding first version of yacy user DB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@817 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
595e0c7e56
*) Bugfix for ProxyErrormsg: Wrong base URL
...
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@815 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
9e1485c13b
new Class for UserAccounts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@813 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
9c4306e41e
fixed problem with htcache path
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
25a59a51ad
fixed problem created with last svn commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0054d3b1a6
added age in network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3dd7e90cdd
kbytes instead of bytes in performance settings; new default values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
1669eaaa1a
fixed svn 805
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@807 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
ca82d690a9
changed in SVN 805 one line too much
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@806 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
4bb1f849a0
Bugfix for http://www.yacy-forum.de/viewtopic.php?t=1233
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@805 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2c7b490e30
memory-logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
7fc822a59b
changed handling of time-zones
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
9b7f37fc37
*) Minor changes
...
- more debugging output: storageTime for indexed document is logged now
- saving memory in plasmaParserDocument.java, plasmaWordIndexEntryContainer.java (not a big deal)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@798 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3c1d968d29
fix-fix for 792 and small changes in ftpc/download/dir experiments
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@797 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
b5a8992d29
*) Setting some object fields to final
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@796 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
5f95a1cf62
*) Bugfix for ProxyErrormsg: Wrong http host header
...
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@795 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
023be89586
*) Bugfix for "Robots.txt wird immer wieder geladen"
...
See: http://www.yacy-forum.de/viewtopic.php?p=10241#10233
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@794 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
35c6c5ead7
*) Bugfix for "Blacklist und Crawlen" Bug.
...
: Crawling continues even if URL is listed in Blacklist
See: http://www.yacy-forum.de/viewtopic.php?p=10279#10279
- missing return statement added. Thanks to allo for the
code review.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@793 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
dc474aa22f
various bug-fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
9e2fc7e5fe
load balancing of crawl target domains
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@791 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3fcc95a82c
integrated crawl-profiles db in memory-performance monitor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@788 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
72ce36baba
cleanup in kelondroRecords
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@787 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
fe6a6abc0b
*) Adding robots.txt db to Performance Settings for Memory menue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@785 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3274ae725e
increased cache size of robots database; however, this should be integrated into new memory control
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@784 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
c6d2f50375
changed order of robots and double-check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@783 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
68d5ff2ef1
added stringbuffer in condenser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@782 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
90f02ea455
*) removing metainfo from serverargs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@780 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
70a5681a4f
*) Bugfix for inactive scp seed uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@779 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago