allo
4320425a17
ipAuth (this does not work yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@937 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
9ff2bd2f0e
allow surfing, if no timelimit set.
...
thx, theli
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@920 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
141417821e
limitless Accounts (set timelimit = 0)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@914 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6bd9348bf1
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@907 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
be8e95d86a
debug printlns removed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@905 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f1ff33177d
reset Timelimits on Daychange
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@904 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
5605cc8018
TimeLimits
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@901 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3966b38360
*) Bugfix for userDB time limit calculation
...
See: http://www.yacy-forum.de/viewtopic.php?p=10949
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@898 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
1688be8590
*) plasmaSwitchboard.java
...
adding more verbose logging output for db initialization
*) httpdFileHandler.java
adding cache for servlet response methods
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@897 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
959eefbc4f
*) Robots.txt parser/ppt
...
cutting of comments at the line end
*) Adding Threadpool for stackCrawl Thread to speedup robots.txt download
and double url checks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@882 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
fb27428674
added restart to Status.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@863 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
0dcfc25d98
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@862 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
a2fa75e688
*) Asynchronous queuing of crawl job URLs (stackCrawl)
...
various checks like the blacklist check or the robots.txt disallow check are now
done by a separate thread to unburden the indexer thread(s)
TODO: maybe we have to introduce a threadpool here if it turn out that this single
thread is a bottleneck because of the time consuming robots.txt downloads
*) improved index transfer
The index selection and transmission is done in parallel now to improve index
transfer performance.
TODO: maybe we could speed up performance by unsing multiple transmission threads in
parallel instead of only a single one.
*) gzip encoded post requests
it is now configureable if a gzip encoded post request should be send on
intex transfer/distribution
*) storage Peer (very experimentell and not optimized yet)
Now it's possible to send the result of the yacy indexer thread to a remote peer
istead of storing the indexed words locally.
This could be done by setting the property "storagePeerHash" in the yacy config file
- Please note that if the index transfer fails, the index ist stored locally.
- TODO: currently this index transfer is done by the indexer thread.
To seedup the indexer
a) this transmission should be done in parallel and
b) multiple chunks should be bundled and transfered together
*) general performance improvements
- better memory cleanup after http request processing has finished
- replacing some string concatenations with stringBuffers
- replacing BufferedInputStreams with serverByteBuffer
- replacing vectors with arraylists wherever possible
- replacing hashtables with hashmaps wherever possible
This was done because function calls to verctor or hashtable functions
take 3 time longer than calls to functions of arraylists or hashmaps.
TODO: we should take a look on the class serverObject which is inherited from hashmap
Do we realy need a synchronization for this class?
TODO: replace arraylists with linkedLists if random access to the list elements is not needed
*) Robots Parser supports if-modified-since downloads now
If the downloaded robots.txt file is older than 7 days the robots parser tries to
download the robots.txt with the if-modified-since header to avoid unnecessary downloads
if the file was not changed. Additionally the ETag header is used to detect changes.
*) Crawler: better handling of unsupported mimeTypes + FileExtension
*) Bugfix: plasmaWordIndexEntity was not closed correctly in
- query.java
- plasmaswitchboard.java
*) function minimizeUrlDB added to yacy.java
this function tests the current urlHashDB for unused urls
ATTENTION: please don't use this function at the moment because
it causes the wordIndexDB to flush all words into the
word directory!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c7b7c69484
*) adding some functions to userDB.Entry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@826 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d9c08a47f9
*) Minor changes to userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@824 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
af7829df25
*) adding some functions to userDB.Entry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@823 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
19648702cc
*) adding iterator to userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@819 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
dd59c1ace9
*) Adding first version of yacy user DB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@817 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
023be89586
*) Bugfix for "Robots.txt wird immer wieder geladen"
...
See: http://www.yacy-forum.de/viewtopic.php?p=10241#10233
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@794 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dc474aa22f
various bug-fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
495bc8bec6
removed cache-control from low and medium priority caches which reduces memory use and computation overhead
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@774 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
fb52a82008
added new performance page for memory settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@751 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
9dfbd93c7b
Updated german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@748 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
718950c5da
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@679 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
2cd695f376
*) Bugfix path-entries of robots.txt were not decoded correctly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@676 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
f8ad65eae1
*) First trial implementation of robots.txt support
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@674 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
9300689dde
bugfix *gr*
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@662 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ebc39a7b9a
minor fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@659 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
e19ededd66
*) prevents problems due to HTML code in profiles (my posting from 05.Sep.2005 http://www.yacy-forum.de/viewtopic.php?t=964 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@657 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f90f699ab1
missing package line.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@655 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
89d7c6e45e
*) Bugfix: offset of 1 prevented correct replacement of HTML
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@654 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
06a451768f
a simple robotsParser.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@652 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
2ee4f9f4e5
*) added escape characters [= and =] to yacyWiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@622 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
4fd5b95b1f
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logSevere instead of logFailure and logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@615 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6adf8a4bde
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logFailure instead of logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
ecfdc4928a
*) better implementation of definition lists
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@564 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
36dfb9f686
implemented citings (blockquote) and an early version of definitions (not much of a list yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@557 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
5d511d651e
fix for Windows
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@513 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
ee0a9a2d9b
recursive Translations.
...
You can now translate the Menu and other things in subfolders, too
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@508 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
53dcbadfbf
using Wikicode instream of bbCode
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@504 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
8659a689f4
seperate Class for the wikiParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@500 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2d8557cb10
minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@487 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
cc38c5e77b
bugfix for wiki key-length
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@468 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
0a68aa6999
translate the last file, too
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@443 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3470a72d48
fixed div by zero, set default delays, fixed release number format and display
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@435 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
bf7d5d1310
Usage of logging-class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@432 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
fe62a38896
minor fixes to the translation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@429 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
98d22a73ef
"File: foo.html"
...
and
"File:bar.html"
support
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@420 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
e6c381a2e2
Translationfile Division for different files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@411 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
40036ba69c
fixed dht transmission; added url-blacklist blocking also for remote search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@398 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
311e627363
blocking of blacklisted urls in indexReceive and small changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@397 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
858cd94299
replaced indexing ram-queue by file-based stack-queue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@381 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
a31029c5de
== as separator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@374 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
9f0f45bcae
Warning: this is untested.
...
I will need this for recursive Translation, and check it while extending the Translator.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@373 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
bef3aaec38
better Translationsystem.
...
You only need to escape Chars, which have special meanings in regexps.
some you have to escape: ()[]{}
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@348 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
890e3f4d4a
*) adding missing calls for function close() to avoid "too many open file" bug*) adding
...
*) bugfix in plasma/plasmaParser.java:
- parsers with missing dependencies wehre not ignored correctly
*) passing a logger instance to the parsers modules which can be used
for logging purposes by the parsers (not done yet)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@276 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
361ba50056
*) changing reference to logger
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@261 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
7b4778b4b9
*) changing reference to logger
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@260 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
f57d280afd
serverLog instead of System.out.println
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@179 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
a8b2dccc7a
New Translation System.
...
Use Wordlists like these:
de.lng
Network\ Overview=Netzwerk-Übersicht
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@176 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2aa5fe8f50
*) Import statements reorganized
...
Now it's easier to determine which class really uses which other class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@82 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
248077d3f0
initial load with yacy 0.36
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago