Commit Graph

695 Commits (adf01c676e587d4235fce1b9d8896c490ee8ec71)

Author SHA1 Message Date
orbiter ce57de6cb3 - fixed re-setting of DHT Send/Receive settings
17 years ago
f1ori 7afa084207 * add nativ java trayicon, using reflections
17 years ago
orbiter 6e7d113eac fix for wrong index initialization after network switch
17 years ago
orbiter 7b35d54c6c fixed some problems with network switching (was not completely 'clean')
17 years ago
orbiter 2b13705839 fixed a mistake in indexing queue processing: documents had been parsed before it was checked if they should be indexed or not. parsing was not necessary for this check, so the check was moved in the queue in front of the document parsing
17 years ago
orbiter 1bbf362cef update to the crawl balancer: better organization and better crawl delay prediction
17 years ago
orbiter 670244849d fix for http://forum.yacy-websuche.de/viewtopic.php?p=9835#p9835
17 years ago
orbiter 1fb1665e71 increased dht interval to avoid peer selection failure
17 years ago
lotus b68d06a6e8 performance settings based on network's remote crawl speed
17 years ago
orbiter 3c6e8d2015 set default ppm when network is switched
17 years ago
orbiter 3288c19c1a reduce remote crawl PPM for fresh peers in freeworld to 6 PPM
17 years ago
orbiter 05dbba4bab added logging conditions to all fine and finest log line calls
17 years ago
danielr 9ff4fc11da partial fix (images,audio,video) for proxy and content-type problem http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
17 years ago
lotus d9d9c522a1 addendum to last commit
17 years ago
orbiter 536e77e8b7 modifications towards a single database operation to read/write http header and cached file at once:
17 years ago
danielr 753a1ae430 - changed default browser from netscape to firefox
17 years ago
orbiter 7989335ed6 Preparations to replace the HTCache with a new storage data structure:
17 years ago
orbiter bdae051d9a - extended new performance graph (better timing)
17 years ago
danielr a087090bbb fixed starting crawl results in "No parser available to parse mimetype 'application/octet-stream'"
17 years ago
danielr 621b473b18 * removed some warnings of findbugs (http://findbugs.sf.net)
17 years ago
orbiter ebb40d324b enhanced memory chart: shows now also the size of the word cache as third vector.
17 years ago
danielr 17b7845eb5 * refactoring
17 years ago
danielr 3bb870bfcd added final where possible
17 years ago
orbiter 50ef5c406f - refactoring of robots parser (removed opaque Objects[] result vector)
17 years ago
lotus 62afea0c9f some improvements for yacyTray
17 years ago
lotus fa695c2d9f tray is now only shown on Windows and doesn't block on linux
17 years ago
lotus d77ed28e2f temporary disabled tray because of flaws on only-shell-linux
17 years ago
lotus f8a1e3175e new yacyTray
17 years ago
orbiter 1e6d12f146 Major update to BLOB data structures:
17 years ago
orbiter 1400cdc91e - refactoring of resourceObserver (moved it to crawler)
17 years ago
orbiter a6719dfd2b - refactoring of robots parser
17 years ago
orbiter e81be7d4f2 added many missing user-agent declarations for yacy http client connections.
17 years ago
orbiter 474659a71f - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order
17 years ago
orbiter 69aac0d74c modified the diskUsage class regarding the following two aspects:
17 years ago
danielr 0c1dc703e4 - set staticIP at startUp
17 years ago
orbiter c998dc6556 - added security functions to flush url and search caches in case that memory is full
17 years ago
danielr 7feae906aa - organize imports
17 years ago
det f597185026 Initial import of the resource observer framework
17 years ago
orbiter e0e7f86f82 some bugfixes for the peer-ping process
17 years ago
orbiter 40d7f485f3 - fixed several NPE bugs
17 years ago
orbiter 2f381b8d7a - fixed at least two causes for a NPE after a use case switch.
17 years ago
orbiter 2a604b7402 added superfast search result computation which can be obtained for local search when snippet fetching is disabled. An example search for the rss interface would be:
17 years ago
orbiter 9bef20b537 - added cleanup for unused server loggings: they are removed after the client had not been seen since one hour
17 years ago
orbiter 1a1841392c small fix
17 years ago
orbiter faed00d75d added use cases to basic configuration
17 years ago
orbiter 4229cd275c fixed several details about network switching, default password, random password and localhost authentification
17 years ago
orbiter c1d721dd2d fix for attacks on localhost-authorized peers from web pages with links to localhost addresses:
17 years ago
orbiter 3bd1db776a implemented switch for admin authorization from localhost:
17 years ago
orbiter cfe6790498 - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet),
17 years ago
orbiter 78087da287 - changed seed file storage to clear text
17 years ago
orbiter 5fde679acb - fixed problem in performance configuration
17 years ago
orbiter 239cc4428d - better domain graph, faster when more links exist, looks better
17 years ago
orbiter dd75b3cabc - patch for bad profiles
17 years ago
orbiter b32736762c enhanced rssTerminal
17 years ago
orbiter fbb712c669 refactoring:
17 years ago
orbiter 1689030ee8 refactoring: moved all crawler classes into their own package
17 years ago
orbiter d2ba1fd2ab major step forward to network switching (target is easy switch to intranet or other networks .. and back)
17 years ago
danielr d4bce6affd refactoring (initialized static fields, removed empty if/else, serialized some fields in serializable classes)
17 years ago
orbiter 483e9a2066 - shifted tld recognition methods from yacyURL to serverDomains
17 years ago
orbiter 32b5b057b9 - modified, simplified old kelondroHTCache object; I believe it should be replaced by something completely new
17 years ago
orbiter e024e3b9cf added new default profiles to distinguish snippet fetch for local and global search
17 years ago
orbiter 1995faef8d - refactoring of Colage back-end: move to plasma package
17 years ago
orbiter 5e3ce46339 - better logging when rejecting a url because it is not in declared domain
17 years ago
orbiter 8313d58ae7 - integrated the collage into the Web Visualization menu
17 years ago
orbiter 82bf9ac1c8 - added Collage servlet from datengrab and modified it:
17 years ago
danielr 959f448e5f - disabled redirects in proxy (so client sees real path)
17 years ago
orbiter 5d1fbb25e7 fix for bad deploy:
17 years ago
orbiter 2c1c3bb6eb - some refactoring (sorry Daniel, hab in deinem Code rumgewütet)
17 years ago
orbiter 14404d31a8 - enhanced performance graph (more info)
17 years ago
orbiter 117ae78001 speed enhancement for reading of eco-table indexes
17 years ago
danielr 5c3c1fdf41 replaced httpc with Apache Jakarta Commons HttpClient (includes some refactoring ;)
17 years ago
orbiter 764a40e37d speed enhancements for crawler and url retrieval (affects also search speed)
17 years ago
orbiter 368593e449 enhanced the concurrency handling of indexing process (better queue size control, better data concept, better shutdown behavior)
17 years ago
orbiter 0241d070bc added concurrency to indexing process:
17 years ago
orbiter bca87f1e38 - refactoring of serverThreads: renaming to distinguish busy-threads and blocking-threads
17 years ago
orbiter 968c775025 - preparation of parsing/indexing queue for concurrent execution
17 years ago
orbiter 9b0e20fb06 next refactoring step in document indexing to prepare concurrency environment for document parsing
17 years ago
orbiter 7f9f639d20 - refactoring and abstraction of index reference (urls) handling: blacklisting is part of reference filtering
17 years ago
orbiter d6050b9ffb - separated the LURL data storage and Crawl result stack for process supervision.
17 years ago
orbiter 8d6a13bc07 refactoring of parsing-condensing-indexing process:
17 years ago
orbiter 541b817502 refactoring of switchboard queueing
17 years ago
orbiter 7150b463ff changed handling of default values and database paths:
17 years ago
orbiter 9c989fe5f7 fixed deadlock
17 years ago
orbiter b4ed937f1e - modified zone navigation (does still not work correctly)
17 years ago
orbiter bfed9c2da6 - some refactoring in search process
17 years ago
orbiter 253a453413 removed possible synchronization deadlock
17 years ago
orbiter f4c73d8c68 - fixed highslide usage
17 years ago
orbiter 2327451653 - changed order of database initialisation (index first)
17 years ago
orbiter 61a81820e3 - refactoring of search tracker
17 years ago
orbiter bd63999801 - faster search: using different data structures that avoid multiplr calculations
17 years ago
orbiter acf771d5e1 - fixed bug with too much RAM in crawler queue
17 years ago
orbiter a8a5df4a51 - more dublin core naming of page metadata
17 years ago
orbiter 15397298dc - refactoring of indexControlRWIs: moved statics to own class; better Dublin Core naming
17 years ago
orbiter efd0b8371a - added parsing of Dublin Core - compliant metadata (see RFC 5013 and ISO 15836) to html parser
17 years ago
orbiter f4e9ff6ce9 more generics
17 years ago
orbiter 45339c3db5 more generics
17 years ago
orbiter dc26d6262b - removed write buffer from kelondroCache (was never used because buggy; will now be replaced by new EcoBuffer)
17 years ago
orbiter a5054c038d - added large number of generics
17 years ago
orbiter 03e7782269 more generics
17 years ago
orbiter 4dc438f7e7 moved to Java 1.5:
17 years ago
fuchsi 21b8d1b918 small cosmetic change for static fields in serverCore (special protocol ASCII entities) to improve readability
17 years ago
orbiter f243e338cf implemented online caution also for local and remote search
17 years ago
orbiter b46bcaa5d8 changed method of profiling
17 years ago
orbiter aefb3f7765 added memory graph picture to PerformanceMemory_p.html
17 years ago
orbiter 9b0ae4b989 added referrer to remote crawl url list
17 years ago
orbiter 89b9b2b02a redesigned remote crawl process:
17 years ago
orbiter af10f729df fixed image search and favicon loading
17 years ago
orbiter c527969185 - enhanced monitoring of ranking parameters
17 years ago
orbiter 6eaa5a0e64 enhanced local search speed. The ranking process is now 6 times faster that before.
17 years ago
fuchsi 425e4ead66 Allow absolute paths in configuration settings.
17 years ago
borg-0300 a5d28785b1 less OOM (works for me)
18 years ago
orbiter ccbfb15b6b enhancement to crawl stacker enqueue order
18 years ago
orbiter 55c87b3b12 changed behavior of crawl stacker
18 years ago
orbiter a31b9097a4 preparations for mass remote crawls:
18 years ago
fuchsi 508de558f7 sbStackCrawlThread is null during first cleanProfiles() run at startup.
18 years ago
fuchsi 70614385ef Attempt to fix the "lost profile handle" bug.
18 years ago
orbiter 33fb2f756d added emergency fail case in remote crawls
18 years ago
fuchsi 03c5b4ad68 more fixes to the yacysearch.rss, it's now 100% valid according to http://feedvalidator.org
18 years ago
fuchsi 7404f2c35c Fix some of the issues with the RSS search interface, see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=392
18 years ago
orbiter 98abe0804d another enhancement to crawl starts with link files
18 years ago
orbiter 01e0669264 re-designed some parts of DHT position calculation (effect is the same as before)
18 years ago
orbiter 842308ea97 - redesigned crawl start menu, integrated monitoring pages
18 years ago
orbiter 3c74014004 automatic deletion of dead client connections
18 years ago
orbiter 4275727d69 fix for peer ping problem (implemented a 3-time re-ping); cause for 'Connection reset' still unknown
18 years ago
fuchsi 5b0c1449e1 various fixes and cleanups for blacklist handling:
18 years ago
orbiter 6c819a6fd9 added cache to favicon display
18 years ago
orbiter daf0f74361 joined anomic.net.URL, plasmaURL and url hash computation:
18 years ago
orbiter f81ef40cc4 no dht activity for small networks; this is not needed if the network is small
18 years ago
orbiter a34d9b8609 * added a search history cache that maintains search results for 10 minutes
18 years ago
orbiter bb426565f0 added new yacy protocol for mass url-pull for better remote crawling distribution
18 years ago
orbiter b5346141b3 made the plasmaHTCache static (there is only one internet, so we need only one cache)
18 years ago
orbiter 947fc46904 refactoring of search process:
18 years ago
orbiter 5605887571 refactoring of search processes
18 years ago
orbiter 62347b50f4 added security layer for ViewImage:
18 years ago
orbiter 69d640b041 added missing synchronization in crawl balancer
18 years ago
orbiter 9628db6cdc enhanced memory allocation during database access:
18 years ago
orbiter e76fe1c078 - replaced unicode characters in copyright holder name ('Brausse')
18 years ago
orbiter 7ff4357184 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=206&hilit=&p=1130#p1130
18 years ago
orbiter 9ca46a8c69 indexing of local (intranet) urls enabled
18 years ago
orbiter 40b0547611 - documentaton changes (removed old forum links)
18 years ago
orbiter b6d9cca67e - fixed problem with yacyVersion and own version generation
18 years ago
orbiter a4e8ad95ab enhancements to news and switchboard queue processing
18 years ago
orbiter a45216b479 fix to prevent bad-formed news messages
18 years ago
orbiter bec4dbc753 added options and execution methods for automated updates
18 years ago
orbiter 208b5297f1 enhanced handling of news records:
18 years ago
orbiter 3421c64d26 implemented update function:
18 years ago
orbiter f1ed91a8e4 added option to allow/disallow DHT transmission during indexing
18 years ago
orbiter 9bbd39b67c - removed unfinished auto-updater from roland and martin
18 years ago
orbiter 1782ef57e5 - added SSI parser and include directive for <!--# include virtual="<file>" -->
18 years ago
orbiter 0e57a8062b added network definition for different YaCy networks
18 years ago