borg-0300
5c4db6968e
sorted URL list, StringBuffers, finals, cleaned, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@860 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
2a18e6486b
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@859 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
64acb46a91
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@857 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
52168fab9b
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@856 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
be60b512b7
fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@855 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
f6cf3967de
fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a2fa75e688
*) Asynchronous queuing of crawl job URLs (stackCrawl)
...
various checks like the blacklist check or the robots.txt disallow check are now
done by a separate thread to unburden the indexer thread(s)
TODO: maybe we have to introduce a threadpool here if it turn out that this single
thread is a bottleneck because of the time consuming robots.txt downloads
*) improved index transfer
The index selection and transmission is done in parallel now to improve index
transfer performance.
TODO: maybe we could speed up performance by unsing multiple transmission threads in
parallel instead of only a single one.
*) gzip encoded post requests
it is now configureable if a gzip encoded post request should be send on
intex transfer/distribution
*) storage Peer (very experimentell and not optimized yet)
Now it's possible to send the result of the yacy indexer thread to a remote peer
istead of storing the indexed words locally.
This could be done by setting the property "storagePeerHash" in the yacy config file
- Please note that if the index transfer fails, the index ist stored locally.
- TODO: currently this index transfer is done by the indexer thread.
To seedup the indexer
a) this transmission should be done in parallel and
b) multiple chunks should be bundled and transfered together
*) general performance improvements
- better memory cleanup after http request processing has finished
- replacing some string concatenations with stringBuffers
- replacing BufferedInputStreams with serverByteBuffer
- replacing vectors with arraylists wherever possible
- replacing hashtables with hashmaps wherever possible
This was done because function calls to verctor or hashtable functions
take 3 time longer than calls to functions of arraylists or hashmaps.
TODO: we should take a look on the class serverObject which is inherited from hashmap
Do we realy need a synchronization for this class?
TODO: replace arraylists with linkedLists if random access to the list elements is not needed
*) Robots Parser supports if-modified-since downloads now
If the downloaded robots.txt file is older than 7 days the robots parser tries to
download the robots.txt with the if-modified-since header to avoid unnecessary downloads
if the file was not changed. Additionally the ETag header is used to detect changes.
*) Crawler: better handling of unsupported mimeTypes + FileExtension
*) Bugfix: plasmaWordIndexEntity was not closed correctly in
- query.java
- plasmaswitchboard.java
*) function minimizeUrlDB added to yacy.java
this function tests the current urlHashDB for unused urls
ATTENTION: please don't use this function at the moment because
it causes the wordIndexDB to flush all words into the
word directory!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
01db66dc69
implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
a9c466ef21
cleaned, finals, StringBuffer, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@849 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0c3a20d44f
more + changed log for better understanding of outOfMemory bug and others
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
d8544e8416
cosmetic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@844 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
0cb9566546
HTML for last Commit.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@840 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
8d871a6a79
delete Function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@839 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
7df4c85506
bugfix & cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@838 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
f58cfbf9a8
passive red to yellow changed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@833 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
252492d031
update User
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@832 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
1b61b674db
changed method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@831 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
2d481e89df
Michael would like it
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@829 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
60fc32ffeb
load values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@828 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
8de3b2dd3c
User create function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@825 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
ff1d3d0680
Init of userDB
...
Pagelayout of User_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
8d728d742b
remove my SVN 818 change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@821 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
a3bb20ca88
New column Info replaced Type, Contact and accept Crawl/Index;
...
11 new pictures show the information in 11x32 pixel;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@818 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
1d1b8f699e
*) Extending proxy autoconfig to avoid problems with multiple local network cards
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@816 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
394fb5d484
*) Small changes that make entering values much easier for people who use the TAB-key to navigate through the page or who use textbowsers like lynx.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@814 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
138d218d5b
all big number have now separators (not Last Seen);
...
used dot instead of comma;
used yacySeed.constants;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@812 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
9c4306e41e
fixed problem with htcache path
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
25a59a51ad
fixed problem created with last svn commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0054d3b1a6
added age in network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3dd7e90cdd
kbytes instead of bytes in performance settings; new default values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2c7b490e30
memory-logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
ee347364d9
basic xhtml/css layout
...
-to test, replace #[he4ader]# with #[xhtml-header]#, #[footer]# with #[xhtml-footer]# and use the xhtml-style css
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@803 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
7fc822a59b
changed handling of time-zones
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3c1d968d29
fix-fix for 792 and small changes in ftpc/download/dir experiments
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@797 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
dc474aa22f
various bug-fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
1aa79f5bb5
cleaned;
...
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@790 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
694226d4b1
typo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@789 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3fcc95a82c
integrated crawl-profiles db in memory-performance monitor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@788 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
fe6a6abc0b
*) Adding robots.txt db to Performance Settings for Memory menue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@785 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
68d5ff2ef1
added stringbuffer in condenser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@782 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
c42a543bc3
*) Adding peername to logmessage when receiving URLs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@781 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
495bc8bec6
removed cache-control from low and medium priority caches which reduces memory use and computation overhead
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@774 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
1dc94e7753
*) Adding support for gzip content-encoding of http post requests
...
used to transferRWIs and transferURLs.
See: http://www.yacy-forum.de/viewtopic.php?t=1167#10020
*) adding yacyVersion.java containing constants defining yacy versions
that support a given feature.
Needed to determine if a remote peer is able to decode gzip
content-encoded http post bodies properly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@772 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e6b9b23290
configuration of startup-memory in webinterface
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@771 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
07f30931ec
various configuration options in memory performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@763 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2f732e32a2
enhancements to memory menue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@762 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
150bd33591
finals;
...
cleaned;
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@761 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e17df64b54
removed IS_ADMIN - feature. This was covered by plasmaSwitchborad.adminAuthenticated
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@760 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
b5337a122c
some more information about available memory in PerformaceMemory menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@759 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
96a5b6e8fb
removed yacy peer types from serverSwitch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@758 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago