orbiter
77ae30063d
refactoring of websearch process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@893 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
579b22d8ff
small update to network drawing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@892 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e85989510a
update to network image; added disconneced peers by disconnection time and changed colors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@890 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c83594528c
integrated crawl stacker into thread control
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@887 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d666b61b83
fix for news-deletion, see also http://www.yacy-forum.de/viewtopic.php?p=11000#11000
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@885 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
45f55a6fad
*) Bugfix for wrong index-queue size displayed on status page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@883 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c1c94111b0
added new network picture at Network menu using the new image-servlet method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@880 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
a1777788a5
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@879 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d98ed41f31
update to image painter and test on http://localhost:8080/imagetest.png
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@878 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f65c939a60
userDB Auth
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@874 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5153ec0f3e
update to image painter
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@873 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c6f0e35a32
removed notifier.gif; this file is created automatically after startup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@871 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1b2db0b52a
fix for file-share access; damaged some commits before by me :-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@870 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
f875eda9b7
changed 2, added 5 new picture for more information
...
added runtime to the list
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@869 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1a5d98cd6d
better imagePainter example and fix for typo http://www.yacy-forum.de/viewtopic.php?p=10920#10920
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@868 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
06f4ac494a
image-painter test; try with http://localhost:8080/imagetest.png
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@864 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
fb27428674
added restart to Status.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@863 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
13dd16d1dd
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@861 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
5c4db6968e
sorted URL list, StringBuffers, finals, cleaned, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@860 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
2a18e6486b
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@859 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
64acb46a91
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@857 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
52168fab9b
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@856 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
be60b512b7
fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@855 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f6cf3967de
fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
a2fa75e688
*) Asynchronous queuing of crawl job URLs (stackCrawl)
...
various checks like the blacklist check or the robots.txt disallow check are now
done by a separate thread to unburden the indexer thread(s)
TODO: maybe we have to introduce a threadpool here if it turn out that this single
thread is a bottleneck because of the time consuming robots.txt downloads
*) improved index transfer
The index selection and transmission is done in parallel now to improve index
transfer performance.
TODO: maybe we could speed up performance by unsing multiple transmission threads in
parallel instead of only a single one.
*) gzip encoded post requests
it is now configureable if a gzip encoded post request should be send on
intex transfer/distribution
*) storage Peer (very experimentell and not optimized yet)
Now it's possible to send the result of the yacy indexer thread to a remote peer
istead of storing the indexed words locally.
This could be done by setting the property "storagePeerHash" in the yacy config file
- Please note that if the index transfer fails, the index ist stored locally.
- TODO: currently this index transfer is done by the indexer thread.
To seedup the indexer
a) this transmission should be done in parallel and
b) multiple chunks should be bundled and transfered together
*) general performance improvements
- better memory cleanup after http request processing has finished
- replacing some string concatenations with stringBuffers
- replacing BufferedInputStreams with serverByteBuffer
- replacing vectors with arraylists wherever possible
- replacing hashtables with hashmaps wherever possible
This was done because function calls to verctor or hashtable functions
take 3 time longer than calls to functions of arraylists or hashmaps.
TODO: we should take a look on the class serverObject which is inherited from hashmap
Do we realy need a synchronization for this class?
TODO: replace arraylists with linkedLists if random access to the list elements is not needed
*) Robots Parser supports if-modified-since downloads now
If the downloaded robots.txt file is older than 7 days the robots parser tries to
download the robots.txt with the if-modified-since header to avoid unnecessary downloads
if the file was not changed. Additionally the ETag header is used to detect changes.
*) Crawler: better handling of unsupported mimeTypes + FileExtension
*) Bugfix: plasmaWordIndexEntity was not closed correctly in
- query.java
- plasmaswitchboard.java
*) function minimizeUrlDB added to yacy.java
this function tests the current urlHashDB for unused urls
ATTENTION: please don't use this function at the moment because
it causes the wordIndexDB to flush all words into the
word directory!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
01db66dc69
implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
a9c466ef21
cleaned, finals, StringBuffer, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@849 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0c3a20d44f
more + changed log for better understanding of outOfMemory bug and others
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
d8544e8416
cosmetic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@844 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
0cb9566546
HTML for last Commit.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@840 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
8d871a6a79
delete Function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@839 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
7df4c85506
bugfix & cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@838 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
f58cfbf9a8
passive red to yellow changed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@833 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
252492d031
update User
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@832 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
1b61b674db
changed method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@831 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
2d481e89df
Michael would like it
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@829 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
60fc32ffeb
load values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@828 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
8de3b2dd3c
User create function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@825 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ff1d3d0680
Init of userDB
...
Pagelayout of User_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
8d728d742b
remove my SVN 818 change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@821 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
a3bb20ca88
New column Info replaced Type, Contact and accept Crawl/Index;
...
11 new pictures show the information in 11x32 pixel;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@818 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
1d1b8f699e
*) Extending proxy autoconfig to avoid problems with multiple local network cards
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@816 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
394fb5d484
*) Small changes that make entering values much easier for people who use the TAB-key to navigate through the page or who use textbowsers like lynx.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@814 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
138d218d5b
all big number have now separators (not Last Seen);
...
used dot instead of comma;
used yacySeed.constants;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@812 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9c4306e41e
fixed problem with htcache path
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
25a59a51ad
fixed problem created with last svn commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0054d3b1a6
added age in network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3dd7e90cdd
kbytes instead of bytes in performance settings; new default values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
2c7b490e30
memory-logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ee347364d9
basic xhtml/css layout
...
-to test, replace #[he4ader]# with #[xhtml-header]#, #[footer]# with #[xhtml-footer]# and use the xhtml-style css
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@803 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago