<li>Higher crawling speed possible caused by better ram cache flush methods</li>
<li>The crawl balancer now has a security function which prevents that remote web servers are accessed more than two times in one second. In case a crawling from a single domain is made, this means the crawling speed is restricted to not more than 120 pages per minute</li>
<li>The crawl balancer chooses better urls. Newly added urls are now prevented from beeing hidden by masses of links generated by the crawler. The effect is that in most cases the security function described above is not needed.</li>
<li>Added a crawling speed button on the crawling monitor page.</li>
<li>Crawl targets get informed about the yacy bot; a link to http://yacy.net/yacy/bot.html is attached to each crawl request; the page explains YaCy and that YaCy respects robots.txt</li>
</ul>
<li>Better Monitoring</li>
<ul>
<li>New search result page SearchStatistics_p.html shows local and remote search requests; remote requests are anonymized</li>
<li>Added network-wide QPM (queries per minute) computation to show how much the network is used for web search. The statistics are not reported from searching peers, but from searched peers; therefore the accumulation preserves privacy of the searcher</li>
<li>New page LogStatistics_p.html which shows an evaluation of entries from the log.</li>
<li>New page BlacklistCleaner_p.html to clean up wrong blacklist entries. The page allows categorization of blacklist error case, correction of error and the optional deletion of the blacklist entry.</li>
<li>Added RSS feed for YaCyNews</li>
</ul>
<li>Enhanced User Interface</li>
<ul>
<li>Added a robots.txt configuration menu to enable/disable external crawlers to access the yacy user interface</li>
<li>New wiki-parser</li>
<li>Blog entries may now have user-comments</li>
<li>The network list page now provides links to the users blog pages</li>
<li>The menu points had been rearranged</li>
</ul>
<li>Less Memory Usage and Better Memory Management</li>
<ul>
<li>All caches (node cache, object cache) now have enhanced self-organization and dont need fixed size assigments</li>
<li>Memory protection by disallowing collection arrays beyond kca-7. Collections larger than those are written to 'common' files.</li>
<li>The network picture uses less memory</li>
</ul>
<li>Bugfixes: a very large number of bugfixes were made.</li>
orbiter
18 years ago