<p>This is essentially the release change-log. We have a <ahref="http://www.yacy-websuche.de/wiki/index.php/Dev:Roadmap">release roadmap</a> and releases published here will (hopefully) match the milestones from the roadmap's vision.
<li>Re-designed search page: this shows now a simplified page without the left navigation column. The column is only shown if the user is logged-in.</li>
<li>Added Picture-option to search results: this shows only pictures that are on the found page in thumbnail size.</li>
<li>Added 'prefer' option to search: this will move search results with matching headlines to the top of the search results.</li>
</ul>
<li>Enhanced Crawler Options</li>
<ul>
<li>Introduced a re-crawl option for crawl starts of already indexed web pages.</li>
<li>Introduced a auto-dom-filter for special steering of the crawler in case that the crawl start is a link-list and crawls should stay within only the domains of the link-list.</li>
<li>Introduced a link-per-domain limitation for the crawler.</li>
</ul>
<li>Other User Interface Enhancements</li>
<ul>
<li>Re-designed (re-ordered) left navigation bar.</li>
<li>Added blog to Communication/Publication section of navigation bar.</li>
<li>New index cleaner job and interface to delete blacklisted indexes from the database.</li>
<li>The Profile-Viewer may now also show the local profile. A link to a profile-presentation of the local peer is included on the new simplified search page.</li>
</ul>
<li>Performance Enhancements</li>
<ul>
<li>Introduced a second RAM cache for indexes that are received from DHT transmission and removed second cache flush limit from normal index cache.</li>
<li>Added a period of rest to the cache flush process: this prevents that indexes are flushed to disc if they could be candidates for DHT transmission.</li>
<li>Enhanced html parser to support image tags.</li>
<li>Added checks to protect peers from wrong seeds.</li>
<li>Re-design of kelondroTree-iterator to prevent concurrent modification exceptions.</li>
<li>Blacklists are now also applied to urls during DHT receives.</li>
<li>Fixed bugs with index cache flushs.</li>
<li>Fixed behavior of remote crawl trigger that caused long pauses in case of local index queue overflow</li>
<li>Better robots.txt support.</li>
<li>Many minor bugfixes concerning file type handling, url normal form computation, crawl profile storage, network peer counting, parser errors, wiki code, htcache storage etc.</li>
<li>ajax-driven search result enrichment with snippet post-fetch. Snippets that are not available on page load time will be fetched using ajax requests.</li>
<li>New 1-2-3(-4) configuration menu makes it easier to configure YaCy for first-time-users</li>
<li>New yacy.badwords filters bad topwords</li>
<li>Show public Bookmarks in Bookmarks.html, private ones, if the user is logged in.</li>
<li>Added a yacybar.xpi to the release</li>
</ul>
<li>Bugfixes</li>
<ul>
<li>fixed conjunctive search; was broken because of wrong data structures</li>
<li>special chars (like german umlauts) are now allowed in bookmark tagNames</li>
<li>/xml/bookmarks/* now uses one file for private/public entries. private only with password</li>
<li>disabled write cache to avoid database corruption in case of crash</li>
<li>bugfixes to HTTP/0.9 header handling</li>
<li>fixed re-search bug</li>
</ul>
<li>Enhancements</li>
<ul>
<li>index write access (dht transmission, indexing, dht deletion) is now completely synchronized, which increases speed and reduces IO</li>
<li>there is now a real streaming support for lage files</li>
<li>support of chunked transfer-encoding for http/1.1 clients</li>
<li>support of gzip content-encoding suitable clients</li>
<li>automatic TOC generation for pages in wiki</li>
<li>changed user-agent string for yacy crawl access to 'yacybot'</li>
<li>Introduction of Block Rank; Generation, transmission and collection of block rank statistics; computation of block rank tables for new search ranking YBR (YaCy Block Rank)</li>
<li>New network picture on network page</li>
<li>New Connection tracking page</li>
<li>New ICAP support and SQUID-like redirectors</li>
<li>Many small changes</li>
</ul>
<li>Improvements</li>
<ul>
<li>New search behavior with better search results and less search time.</li>
<li>Index transfer with less computation overhead.</li>
<li>Better robots.txt parser.</li>
<li>New write cache for less IO resulting in better database speed.</li>
<li>Asynchronous queueing of crawl jobs and better crawl+indexing performance</li>
<li>More document parser. The following document formats are supported:</li>
<li>Prevention of unwanted DDoS effects caused by YaCy crawls by doing a target-server load balancing; further prevention is done by robots.txt</li>
<li>New setting of proxy cache size and storage path in submenu 'Proxy Indexing'</li>
<li>Modified cleanup of HTCache</li>
<li>Sorted config list in Config_p.html and sorted file list in Cache Admin menu</li>
<li>Code clean-up: added finals and constants</li>
<li>New Network menu design; showing peer status, Index Receive and Crawl Receive properties as images</li>
<li>Added ICAP support; now an experimental ICAP Server is embedded and Yacy allows other proxies to use the indexing service via icap response modification requests</li>
<li>Single-Peer permanent index transfer (full flush to other peer)</li>
<li>Added a templateCache to httpFileHandler</li>
<li>Added blacklist support for https requests and crawler</li>
<li>Adding functionality to delete entries from Indexing and Crawler Queue</li>
<li>Adding functionality to clear the whole indexing queue</li>
<li>Proxy now supports the X-Forwarded-For Header</li>
<li>Indexing queue now displays total size of enqueued content in kb</li>
<li>Remembering Crawler-isPaused setting by storing status into config file</li>
<li>Splitting of status page into a private and a public accessible part</li>
<li>Adding Queue overview to status page</li>
<li>New symbols for Peer Status, connection status, Index-Receive-Granted and Crawl-Receive-Granted in Network menu, replacing old separate columns</li>
<li>Support for robots.txt</li>
<ul>
<li>Implementation of a robots.txt parser</li>
<li>Control of remote crawls with robots.txt</li>
</ul>
<li>Performance enhancements</li>
<ul>
<li>Better Database Caching</li>
<ul>
<li>Better usage of memory in kelondro Record-Nodes and less IO access</li>
<li>New cache-control menu within the performance menu</li>
<li>Better cache-size default values</li>
</ul>
<li>Accelerated Blacklists import; makes big lists possible</li>
<li>Content-Encoding GZIP support for http post requests on index transfer/distribution</li>
</ul>
<li>Bugfixes</li>
<ul>
<li>Normalization problems: prevent URLs with ':80'</li>
<li>Many bug fixes for NULL pointer occurrences</li>
<li>Display of an proxy error page instead of a white page if the server has closed the connection before yacy was able to receive the http response line</li>
<li>Crawler Redirection bug fixed</li>
<li>Indexer now gets the mimeType now from the parsed document instead of the responseHeader (this is especially necessary if mimeType has to be detected by the MimeType parser)</li>
<li>URLs pointing to a server having a private ip addess will not be indexed anymore</li>
<li>Unsupported MimeTypes and fileExtensions will not be queued by the cachemanager in the indexer queue anymore (to reduce unneccesary IO)</li>
</ul>
<li>.. many more small changes and bugfixes. for details please see the <ahref="http://svn.berlios.de/wsvn/yacy/?op=log&rev=0&sc=0&isdir=1">SVN history</a></li>
<li>Index distribution to DHT now fully active</li>
<ul>
<li>All peers, principal, senior and junior peers distribute their index to other principal or senior peers. Juniors send their index to only one principal/senior peer, while principal/senior peers distribute to 3 redundant other peers.</li>
<li>The index distribution and the index receive flag on Index-Control must be set to enable global search on the own peer</li>
<li>For a global search, only relevant peers according to DHT rules are selected</li>
<li>A new menu 'News' shows the news processing queues for incoming and outgoing news. However, news shall not be monitored here but they influence the presentation of other information throughout the system:</li>
<li>The Index-Create menu now shows a list of previously started crawls of other peers. They are distinguished between Crawls in progress and finished crawls.</li>
<li>When a Crawl in the Index-Create menu is started with the 'RemoteIndexing' - flag set on, then automatically a YaCyNews is generated to inform other peers about that crawl start. A message can be attached to explain why this crawl was startet.</li>
<li>When a personal profile is changed, a News Message is generated.</li>
<li>When a Wiki entry is changed, a Message is generated.</li>
<li>Within the Network menu, Alerts for Profile Updates, Wiki Updates and Cralws in Progress may appear.</li>
<li>The YaCy wiki has enhanced and was moved out from the 'Lab' to the main menu. The Wiki-System now supports embedded images and can show a preview.</li>
<li>New logging policy: logs are now exclusively written to the rotating logs in <app-home>/log/</li>
<li>Search was enhanced using the intermission feature (all other processes are paused while search is in progress)</li>
<li>Added snippets to search results. Snippets are fetched by searching peer from original web sites and are also transported during result transmission from remote search results.</li>
<li>Proxy shows now an error page in case of errors.</li>
<li>Preparation for localization: started (not finished) German translation</li>
<li>Status page shows now memory amount, transfer volume and indexing speed as PPM (pages per minute). A global PPM (sum over all peers) is also computed.</li>
<li>Re-Structuring of Index-Creation Menue: added more submenues and queue monitors</li>
<li>Added feature to start crawling on bookmark files</li>
<li>Added port forwarding for remote peer connections (the peer may now be connected to an configurable address)</li>
<li>Added bbCode for Profiles</li>
<li>Memory Management in Performance Menu: a memory-limit can be set as condition for queue execution.</li>
<li>Added option to do performance-limited remote crawls (use this instead to switch off remote indexing if you are scared about too much performance loss on your machine)</li>
<li>Enhanced logging, configuration with yacy.logging</li>
<li>Enhanced caching in database (less memory occupation)</li>
<li>Replaced RAM-queue after indexing by a file-based queue (makes long queues possible)</li>
<li>Changed assortment cache-flush procedure: words may now appear in any assortment, not only one assortment. This prevents assortment-flushes, increases the capacity and prevents creation of files in DATA/PLASMADB/WORDS, which further speeds up indexing.</li>
<li>Speed-up of start-up and shut-down by replacement of stack by array. The dumped index takes also less memory on disk now. Because dumping is faster, the cache may be bigger which also increases indexing speed.</li>
</ul>
<li>Bugfixes:</li>
<ul>
<li>Better shut-down behavior, time-out on sockets, less exceptions</li>
<li>Fixed gzip decoding and content-length in http-client</li>
<li>Better httpd header validation</li>
<li>Fixed possible memory leaks</li>
<li>Fixed 100% CPU bug (caused by repeated GC when memory was low)</li>
<li>There are now 3 different crawl threads: local crawling, global crawl trigger and remote-triggered crawl jobs.</li>
<li>The thread pools can now be configured through the Performance-Menu and a customized number of crawling threads is possible.</li>
<li>Crawling can be paused and resumed.</li>
<li>Changed method of index caching; this speeds up crawling and provides a more economic data structure.</li>
</ul>
</li>
<li>Enhanced Proxy: added transparent proxy support. It is now possible to route http traffic through yacy without setting a proxy configuration in browsers. Example: set your iptables configuration with <br><tt>iptables -t nat -A PREROUTING -p tcp -s 192.168.0.0/16 --dport 80 -j DNAT --to 192.168.0.1:8080</tt></li>
<li>Extended seed-upload methods for principal peers: more configuration options, better extensibility. Added support for scp.</li>
<li>More external parsers. YaCy now supports tar, zip, gzip, bzip, rss, rtf, pdf, doc. To use these parsers, an additional libx-library must be installed which comes separately to the YaCy core distribution.</li>
<li>Enhanced Shutdown procedure: many unnesessary threads had been removed, a shutdown hook had been added. Missing file closings hat been added. The new index caching method flushes the cache faster.</li>
<li>Added support for localization: it is now possible to extend YaCy with localization data; added languages can be accessed with the new Language-Menu</li>
<li>YaCy's source code is now hosted in a Subversion/svn version control system on developer.berlios.de: <ahref="http://developer.berlios.de/projects/yacy/">yacy@berlios.de</a></li>
<li>new Thread-Pools and performance enhancements from Martin Thelian: much faster http-server and more responsive web interface</li>
<li>fixed bug in database caching that prevented from caching at all; now database much faster. This also speeded up proxy mode (must read http-header from database)</li>
<li>modified thread control for non-blocking dequeueing</li>
<li>Enhanced thread control and added performance menu: this can be used to steer scheduling tasks and for profiling.</li>
<li>Enhanced search result ranking.</li>
</ul>
<br><p>v0.35_build20050306
<ul>
<li>new Features</li>
<ul>
<li>new user-profile management and remote access of profiles through the network-page</li>
<li>new cookie-monitor. Will be used to manage cookie-filter</li>
<li>new template engine and re-design of many administration pages as preparation for upcoming localization</li>
<li>now permanent storage of passive peers</li>
<li>enabled switch-of of proxy-cache</li>
<li>new proxy-indexing monitor and moved proxy-indexing configuration to that new page</li>
<li>more functions to DHT-management:</li>
<ul>
<li>remote indexing tagets now selected by DHT rule</li>
<li>remote search now selects hierarchically with DHT-rule</li>
</ul>
<li>enhanced access control to YaCy administration</li>
<ul>
<li>passwords are now encoded to MD5-Hashes before stored to httpProxy.conf</li>
<li>brute-force password-hack prevention by additional delay's</li>
<li>added new 'steering' servlets for automated processes that need authorization</li>
</ul>
<li>re-design</li>
<ul>
<li>re-designed main menu: new sub-menu for proxy functions</li>
<li>re-design of Network Monitor page</li>
<li>re-design of seed database management and implementation of seed-action interface</li>
</ul>
</ul>
<li>fixed bugs:</li>
<ul>
<li>fixed a bug with cache-control</li>
<li>fixed a bug with peer-list uploading</li>
<li>fixed a bug that provoked indexing of YaCy's own web pages</li>
<li>fixed a bug that prevented loading of some web pages: (JavaScript bug) doublequote/singlequote mixture removed</li>
<li>better binary-check on files before indexing</li>
<li>fixed misbehavior of Network-Page: re-design of enumeration method and auto-heal function in kelondroTree</li>
</ul>
</ul>
<br><p>v0.34_build20050208
<ul>
<li>Remote transmission of index (RWI) information to other peers with correct DHT position</li>
<ul>
<li>implemented two new yacy-protocol - commands: yacy/transferRWI and yacy/transferURL for RWI partition transfer</li>
<li>selection of DHT positions and selection of correct RWI partitions for transmission</li>
<li>performing full flush of index if peer is running in junior mode: now these juniors can contribute to the global index.</li>
<li>default full receive of index transmission in senior peers; these peers will currently not transfer indexes. This is a test configuration and senior2senior RWI transmission will be enabled in future releases.</li>
<li>Configuration flags (grant/do not grant) in 'Index Control' menu.</li>
</ul>
<li>Enhanced remote search</li>
<ul>
<li>selelction of less result values: less traffic, faster response.</li>
<li>pre-sorting of results in remote peers before transmission: better results</li>
</ul>
<li>more properties in seeds</li>
<ul>
<li>Flags for "accept remote crawls" and "accept remote indexes"</li>
<li>Flags for "grant index distribution" and "grant index receive"</li>
<li>Control values for received/send RWI/URL</li>
<li>All flag values are shown on Network page</li>
</ul>
<li>Bug-fixes:</li>
<ul>
<li>no re-set of remote crawl delay after re-connect</li>
<li>proxy fail (shows white pages) fixed: better timeout value</li>
<li>local indexing = off did not work, fixed.</li>
<li>auto-heal of seed.db - fail</li>
<li>many minor bug fixed</li>
</ul>
<li>new <ahref="http://www.yacy-forum.de">german forum at http://www.yacy-forum.de</a>, provided by Roland Ramthun</li>
</ul>
<br><p>v0.33_build20050107
<ul>
<li>Support for Stop-Words; default stopwords are included; stopwords are excluded for indexing and in search query results</li>
<li>Skin support</li>
<li>New start/stop-script for unix/linux daemon init process</li>
<li>File-Share entries can now have description entries</li>
<li>Enhanced File-Sharing Menu</li>
<ul>
<li>Every entry can have a comment attached</li>
<li>Comments or picture preview visible in file list</li>
<li>File name and comment field can be indexed and globaly searched</li>
<li>Files found with search interface are dynamically linked to the actual IP of the peer hosting the file</li>
</ul>
</ul>
<br><p>v0.32_build20041221
<ul>
<li>New Crawling-Profiles for Crawl-Threads
<ul>
<li>every crawl start now defines it's own crawl job; new crawls do not interfere with previously started and still running jobs; all started jobs may run concurrently</li>
<li>New Design, new documentation, new mascot 'Kaskelix' (appears on search page), new home page location <ahref="http://www.yacy.net/yacy">http://www.yacy.net/yacy</a></li>
<li>Promotion-String on search page</li>
<li>New shutdown-trigger (no more file polling, new stop scripts)</li>
<li>Principal-peer gaining after file generation</li>
<li>New 'Log'-menu: view the application log on the web interface</li>
<li>Bug-fixes
<ul>
<li>Termination process should succeed now.</li>
<li>Cross-Site-Scripting bug removed</li>
<li>Removed deadlock occurred during concurrent crawl job starts</li>
</ul>
</li>
</ul>
<br><p>v0.31_build20041209
<ul>
<li>Integrated url filter for crawl jobs (Index Creation - page) and search requests (Search Page).</li>
<li>Removed a bug that caused sudden termination when a not-valid url was crawled.</li>
<li>Massively enhanced indexing speed by implementation of an additional word index cache.</li>
<li>Added button to delete/empty the crawl url stack.</li>
<li>Every Senior and Principal Peer may now start Remote Crawls: The initiating peer starts with the crawl and may assign URLs to qualified other peers. Those peers load the assigned resource, index them and return the index statistics back to the initiator. Executing peer may only be a Senior or Principal peer.</li>
<li>Extended URL management: URLs are now organized in three different sets: Noticed-URLs (not loaded but possibly queued for crawling), Error-URLs (not loaded but may be re-loaded to avoid index loss in case of temporary target server downtime or network problems) and Loaded-URLs. The Loaded-URLs are again divided into six categories:</li>
<li>partly remote/local index: result of search queries</li>
<li>partly remote/local index: result of index transfer (to be implemented soon)</li>
<li>local index: result of proxy fetch/prefetch</li>
<li>local index: result of local crawling</li>
<li>local index: result of remote crawling requests</li>
</ol>
<li>New monitoring pages: Local Index Monitor for results of LURL's (see above), cases 1-5 and the Global Index Monitor for case 6. Because the results of global crawls are not personal to the peer owner, the monitor page is not protected.</li>
<li>Options to allow or disallow remote crawling; either as initiating or executing peer.</li>
<li>Idle/Due-Time - management for each peer: to organize remote-crawl load-balancing, a delay time is used to schedule remote crawls. The seed management was extended to store and maintain these delay times.</li>
<li>process enhancements in httpc and httpd classes</li>
<li>gzip-load mode in httpc fixed</li>
<li>removed DNS bottleneck (the java DNS blocks while accessed simultanously)</li>
<li>integrated DNS-prefetch</li>
</ul>
</li>
<li>Implemented Shut-Down Procedure
<ul>
<li>Integrated notifier procedure in all threads.</li>
<li>The application now creates a file 'yacyProxy.control' after start-up.</li>
<li>To stop the yacyProxy, remove the control file.</li>
<li>Integrated a 'Shutdown' - button on the 'Status'-page which also triggers shut-down</li>
<li>After shut-down is initiated the application first processes all scheduled crawling- and indexing tasks which may last some minutes in the worst case.</li>
</ul>
</li>
<li>Removed bugs
<ul>
<li>URL normalization</li>
<li>many minor bugs</li>
</ul>
</li>
</ul>
<br><p>v0.29_build20041022
<ul>
<li>New option to start explicit crawling jobs: a start url and a crawling depth
(differently from the prefetch depth) can be set.</li>
<li>Integrated monitoring interface for prefetch/crawling activities.
The user can now observe the crawling and indexing activity in detail.
There is also a report page that lists all newly indexed pages with the option
to delete these indexes again. The interface also reports the initiator
of the crawling/indexing tasks which can be currently either the prefetch mechanism
or explicit crawling requests. In future releases the initiator may also refer to
remote crawling requests.
</li>
<li>New caching procedure for database requests on file-system level.
<li>Extended blacklist url matching: parts of a domain may now be matched with wildcards '*'. (the URL's path may be matched with regular expressions)</li>
<li>The application will be re-named. Many parts now refer to the new application name 'yacy', but not all.</li>
</ul>
<br><p>v0.28_build20041001
<ul>
<li>Search results are now searched again for characteristic word patterns.
The patterns are statistically evaluated and are used to generate
"search associations",
shown as hints for further combined search.
</li>
<li>Parallelized peer propagation process. This results in very rapid bootstraping.
</li>
<li>Integrated new 'score' library for rapid element sorting - used for search
patterns and rapid bootstraping. May help in future releases to speed up indexing.
</li>
<li>Minor bug-fixes.</li>
</ul>
<br><p>v0.27_build20040924
<ul>
<li>Bug fix in remote search result preparation.</li>
<li>Speed enhancements on search client when doing remote search.</li>
<li>Small changes in file sharing interface.</li>
</ul>
<br><p>v0.26_build20040916
<ul>
<li>Introduced new 'virtual' TLD (top-level-domain) '.yacy' that the proxy resolves into the peers IP and port numbers:</li>
<ul>
<li>Every yacy-peer can now be contacted using the peer's name as domain name:
Proxies users can obtain any other proxy-hosted pages using the url 'http://<peer-name>.yacy'.</li>
<li>Implemented sub-level domains for yacy TLD's: they are matched to subdirectories of the peer's individual web root HTDOCS. (see below)</li>
</ul>
<li>Support for individual web pages:</li>
<ul>
<li>Every proxy host can serve it's individual web page. We implemented two paths for each server: one default path pointing to <application-root>/htroot for administrative pages and an alternative path for individual use at <application-root>/DATA/HTDOCS.</li>
<li>The individual web pages may be accessed either using the new '.yacy' TLD's through another proxy, or optionaly by using the peer's IP:port address. The recommended default address of a proxy is 'http://www.<peer-name>.yacy', which is mapped to <application-root>/DATA/HTDOCS/www/. </li>
<li>Integrated an upload/download interface for individial web pages: additional accounts for uploaders and downloaders ensure appropriate authorization. The file-sharing web space can be browsed with an directory servlet. A default sub-domain is assigned to 'http://share.[peer-name].yacy', which is mapped into <application-root>/DATA/HTDOCS/share/.</li>
<li>Web clients not using the proxy may contact the new individual default subdomains using the URLs http://<peer-IP>:<peer-port>/www/ and http://<peer-IP>:<peer-port>/share/.
</ul>
<li>Several Bug-fixes:</li>
<ul>
<li>Date bug appearing when accessing the proxy httpd with the proxy.</li>
<li>Additional Time-out catch-up at httpc when a file is submitted without length tag. Also extended general retrieve - time.out.</li>
<li>Terminal line restriction of 1000 bytes was too tight (cookies may have 4kb length).</li>
<li>Introduced global general time measurement for peer synchronization and balanced hello - round-robin.</li>
</ul>
<li>Enhanced proxy-proxy - mode: 'no-proxy' settings as list of patterns to exceptional dis-allow usage of remote proxies.</li>
<li>Better display of cache content in the Cache Administration.</li>
</ul>
<br><p>v0.24_build20040816
<ul>
<li>New 'Cache' Menu item: The proxy cache can now be inspected. It shows a directory list with http response headers and content to each file in the proxy cache.</li>
<li>Faster Bootstraping: The connection policy was changed: as long as the proxy status is 'virgin', the most recent known connection is used for bootstraping; then later the least recent for peer distribution.</li>
<li>Better Formatting in Network Menu.</li>
</ul>
<br><p>v0.23_build20040808
<ul>
<li>Blacklists now provide management of several lists and more import options.</li>
<li>code cleanup + many minor bugs</li>
<ul>
<li>Messages now work (corrected POST implementation, this also cleaned the way to index distribution); improved message sending, displaying etc.</li>
<li>httpd-speedup (no more temporary files, template prefetch without double-load)</li>
<li>much better Bootstraping and more intelligent yacy-peer updating</li>
<li>auto-migration of new settings from httpProxy.init</li>
<li>much better logging; extensive log configuration options for all parts of the application now in httpProxy.init</li>
<li>better search requesting (more results)</li>
<li>yacy protocol may now also use other proxies in proxy-proxy-mode</li>
</ul>
<li>more documentation</li>
<ul>
<li>permanent demo-page at yacy.net/home.html with wiki</li>
<li>new FAQ at http://www.yacy.net/yacy/FAQ.html</li>
<li>first step to move YACY to new home http://sourceforge.net/projects/yacy/</li>
</ul>
</ul>
<br><p>v0.22_build20040711
<ul>
<li>More security bug fixes (dementia accountia, '..' usage in server path, server blacklist too tight for local clients)</li>
<li>Another advance in better peer distribution and recognition (distinguishes between 'real' disconnected peers and 'hearsay' disconnected peers. Keeps track of online time. No preferences of principal peers in link distribution)</li>
<li>An option to switch the peer to online mode without using the proxy. This makes life much easier for newbie's.</li>
<li>A new message function. Within the Network page, one can hit the 'm' and may then send a message to the other peer. The owner of that
peer can read the message in his/her private message inbox. This function is only in alpha statdium; it works only in rare cases and
we don't know why. Only for testing.</li>
<li>Cleaned up the mess of different database and configuration files. All run-time data is now accumulated in the new folder 'DATA'. If you previously generated an index and want to migrate, you simply need to put your old PLASMADB folder into the new DATA folder.</li>
<li>Clean up of the source mess and partition of them into separate packages</li>
<li>Some design enhancements of the online interface</li>
</ul>
<br><p>v0.21_build20040627
<p>After an announcement on freshmeat.net we got many hits in the newly build p2p-network. We learned from the p2p-propagation behavior and
implemented a lot of new routines to stabilize the YACY network.
<ul>
<li>Better peer analysis, statistics, propagation/distribution (more properties and bug fixes).</li>
<li>No more JavaScript in online Interface. New template logic for httpd and new online interface look-and-feel, using the new features.</li>
<li>New FAQ in documentation.</li>
<li>Protection against hacker and virus attacks: new self-configuring client-IP blocking in serverCore.java</li>
<li>More information and warnings about security settings to the operator to protect the own peer</li>
<li>Network statistics and monitor shows status of remote peers and the distributed index</li>
</ul>
<br><p>v0.20_build20040614
<p>The first step into the p2p-world: introduction the YACY (yet another cyberspace) p2p network propagation and information wares distribution system. YACY enables in this release a rudimentary index exchange so that you can use YACY to bootstrap a world-wide distributed search engine.</p>
<ul>
<li>Added status page on web interface and automatic opening of web browser on status page. Can be switched off on the satus page.</li>
<li>Implemented still missing element removal and AVL balancing for element insterts in the kelondro database. This ensures logarithmic efforts on database access, which influences the proxy and the search service. Now only AVL balancing after removal must be implented, but it's missing is not critical.</li>
<li>Added blacklist enhancements and web interface for blacklist editing from Alexander Schier.</li>
<li>More and better documentation.</li>
<li>Many minor bug fixes, i.e. non-cacheabilty of web interface, exception catch-up on startup when proxy is used before coloured lists are loaded.</li>
<li>First p2p elements implemented: every peer on startup looks for other peers and announces it's own startup. The function does not yet actively implement an index exchange, but can repond to remote index queries.</li>
</ul>
<br><p>v0.16_build20040503
<p>This release is a major step to make the proxy enterprise-ready: we introduced several security mechanism and access
restrictions for the proxy and the server. Every security setting can be configured through a web page. Thanks to the new
HTTPS proxy, the proxy can now be considered as 'complete'.
<ul>
<li>implemented a HTTPS proxy, sharing the same proxy port with http;
this does not help for more/better indexing since the SSL data is simply passed through.
But we can now state to be a 'full' http and https proxy, usable in enterprise environments and internet cafe's.</li>
<li>two security layers for web server and proxy access: implemented Client-IP - filtering, which adds a virtual Firewall to
the application. Every client that does not match the client-IP-filter is blocked. The second layer is a PROXY password
protection. All attributes can be configured through a new web page at http://localhost:8080 (standard configuration).</li>
<li>to protect the configuration pages of the web server, we introduced a password protection for special pages on the web server.
Every page that ends with '_p.html' has a protection; the corresponding account can also be set through the local web server.
Users shall be encouraged to set this administration account first.</li>
</ul>
<br><p>v0.15_build20040318
<ul>
<li>Extensive code re-engineering</li>
<ul>
<li>Inserted and further generalized the proxy's genericServer into the AnomicFTPD project. After further enhancements within that project,
it was re-inserted in the HTTPProxy. The Switchboard interface now belongs to the genericServer, which is now called the serverCore.</li>
<li>Removed the old html parser and replaced it by the new htmlFilter library, which now parses the html files during reading from
the remote server. Real-time parsing during streaming html pages is done extremely fast and does not slow down file passing through
the proxy. The new htmlFile provides a filter interface, which is now used to filter out content that is defined by keywords.
Currently the bluelist 'httpProxy.blue' is used to define these words.</li>
<li>Re-engineered the crawler interface and implemented a crawler. Since the crawler does not work in all cases, it is still
disabled in this release. You can switch it on by setting the prefetchDepth in the configuration file httpProxy.init</li>
</ul>
<li>Implemented a 304 response. This speeds up all responses in the case of a cache hit combined with a conditional request.
Since this combination is fairly common, it noticeable speeds up the proxy.</li>
<li>New documentation design</li>
<li>New Search Page design</li>
</ul>
<br><p>v0.14-build20040213
<ul>
<li>More Structure to the whole system to lay the basis for the Crawler</li>
<ul>
<li>The new structure will distinguish between the <i>httpd</i> with it's servlets, the file-servlet and proxy-servlet;
the <i>crawler</i> which also holds responsibility for the http cache that is used by the http proxy and the <i>indexing</i>
engine 'PLASMA', which is again accessed by the http file server. But even with the crawler concept on board here, we still don't have prefetch now.</li>
<li>Moved plasmaTextProvider to httpCrawlerScraper, httpdProxyCache to httpCrawlerCache and httpdSwitchboard to httpCrawlerSwitchboard</li>
<li>New configuration value proxyCacheSize: limits the memory amount of the cache; if the cache exceeds this value the oldest entries are deleted</li>
</ul>
<li>Bug fixes:</li>
<ul>
<li>Found and eliminated nasty bug that prevented using yahoo mail. (they send several cookies at once)</li>
<li>No more indexing of URLs with 'cgi' in name or ending with '.js', '.ico', or '.css' (checking content-type for 'text' is not enough; some servers do not transfer right value)</li>
<li>Fixed search for words containing numbers and german Umlaute</li>
<li>adopted acrypt.java to no using javax.crypt, this was not supported by debian blackdown java 1.3.1. Furthermore, removed -server - flag from httpProxy.sh, that also made blackdown to crash. (you probably want to insert that flag again in your installation)</li>
</ul>
<li>The proxy can now be configured to access another proxy</li>
</ul>
<br><p>v0.13-build20040210
<ul>
<li>Bug fixes:</li>
<ul>
<li>removed forced unzipping for special cases: either if the file to be transported is 'naturally' in gzip format (.gz, .tgz and .zip) or if zipping would not make sense because it would not yield any compression, as for images. Now the 'Accept-Encoding', created by the browser and send to the server has omitted gzip attributes in this cases. This should lead to less overhead (no gzip en/de-coding) and thus to more speed.</li>
<li>now transport of httpc failure response body (especially 404; seemed to be unneccesary, but is not)</li>
<li>search result bug (mixed up appearence) removed</li>
</ul>
<li>Performance and structure enhancements:</li>
<ul>
<li>Extended database capabilities to hold content of dynamic size; new files kelondroRA.java, kelondroAbstractRA.java, kelondroFileRA.java, kelondroDyn.java</li>
<li>Used new database features to store the response header information for all files in the cache into one database file. This saves 50% of the number of files in the cache (no more need for the .header - files)</li>
<li>Implemented a scheduling that moved the time of cache creation into an proxy-idle - time. This reduces the file operation on a single user system by 50% during web page retrievement.</li>
</ul>
</ul>
<br><p>v0.12-build20040204
<ul>
<li>now a <ahref="roadmap.txt">release roadmap</a> exists</li>
<li>enhanced proxy and caching:</li>
<ul>
<li>integrated blacklist 'httpProxy.black' idea and data from Alexander Schier: forced 404 response for blacklisted hosts. This can be used to 'switch off' specific domains, especially AGIS servers. Can also be used for child protection/parental control. Does not filter content!</li>
<li>cache write bug if same file and directory name is used (can be done in URL, but not in cache file system) removed. </li>
<li>detailed 404 debugging response in case of failure</li>
<li>new config value maxSessions for limit the number of concurrent connections to the proxy</li>
<li>Host property bug in httpc for HTTP/1.1 servers removed: now better access to more servers</li>
</ul>
<li>enhanced indexing and searching:</li>
<ul>
<li>implemented rudimentary ranking and ordering of search results either by quality or by date</li>
<li>implemented bluelist 'httpProxy.blue': filtering of all blue-listed words in search expression, result-url and result-description</li>
<li>bugfix for combined search, fixed date attached to search results</li>
</ul>
<li>first contact with <ahref="http://gnugle.sourceforge.net">Gnugle</a> project and knowledge exchange</li>
</ul>
<br><p>v0.11-build20040124
<ul>
<li>non-empty field servlet bug in index.java</li>
<li>greatly enhanced indexing</li>
<ul>
<li>better structure: new classes plasmaIndexEntry, plasmaSearch, plasmaIndex, plasmaIndexCache, plasmaURL</li>
<li>index entry caching and transparent flushing implemented</li>
</ul>
<li>catch-up of sleeping connections, enhanced idle check in genericServer.java</li>
</ul>
<br><p>v0.1-build20040119
<ul>
<li>first time published on www.anomic.de!</li>
<li>client user agent forwarding according to 'yellow'-list</li>
