Commit Graph

4670 Commits (f0651e5f2f797e5f45b9b68fa2891d89a7d68394)

Author SHA1 Message Date
orbiter 90531f78ff refactoring of the cora package to get subpackages for http and ftp (smb to come)
14 years ago
orbiter d0fb6bc2bc cleaned up superfluous classes after sixcoolers migration to HttpComponents-Client-4.x
14 years ago
sixcooler 661867923a ... migrating to HttpComponents-Client-4.x ...
14 years ago
orbiter 7aa860c505 - more logging
14 years ago
orbiter 4d5446d641 code cleanup
14 years ago
orbiter 66ac3a7d9d corrected database row iteration
14 years ago
orbiter dfd416e3fb removed a mysterious image buffer
14 years ago
orbiter e10cd115a9 - added a new RSS reader interface. This is not finished but you can now load and look at RSS feeds. It will be used to index RSS feeds in a way that is appropriate for such kind of data.
14 years ago
orbiter 933dc1a600 removed old rss parser (will be replaced with parser from cora package)
14 years ago
orbiter 70dd26ec95 added the new crawl scheduling function to the crawl start menu:
14 years ago
orbiter 5a994c9796 added a scheduler based on API actions
14 years ago
orbiter 189a986ebd - modified api-call interface to record api calls with references to api-call database (carries pk)
14 years ago
orbiter 054c22e2c6 added TLDs from http://www.opennicproject.org
14 years ago
orbiter 86d7f8a989 - the web visualization can now be generated in custom color
14 years ago
orbiter 64d4204f44 fix for NPE in network image computation
14 years ago
orbiter 7fdb17bb96 redirect uncaught exceptions to logging + small other changes
14 years ago
f1ori 92df768c39 * fix http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2929&hilit=
14 years ago
orbiter eb97bed1df patch for http://forum.yacy-websuche.de/viewtopic.php?p=20576#p20576
14 years ago
orbiter 87b1684211 additional double-check in balancer
14 years ago
orbiter 0d81731e88 fixed crawler bug caused by NPE in logging
14 years ago
orbiter a82a93f2fc - better url double check in crawler
14 years ago
sixcooler a6ed6e8cb9 ... migrating to HttpComponents-Client-4.x ...
14 years ago
orbiter 171f2bd84e - removed unused network oanet
14 years ago
sixcooler 1802c54317 LGPL-Header
14 years ago
orbiter a835a22b32 fixed isLocal() property (better recognition of intranet hosts)
14 years ago
orbiter b480b7a4d0 fix for bug in last commit
14 years ago
orbiter b12bfe1f91 better usage of OSM tile cache and YaCy cache by usage of better tile server computation based on a coordinate hash
14 years ago
orbiter 388aa021c2 - concurrent loading of OSM tiles
14 years ago
orbiter 670c746dc5 dual-licensed HttpConnectionInfo for LGPL
14 years ago
orbiter 301a59e07f moved browser access method from kelondro/util/OS to gui/framework/Browser
14 years ago
orbiter ec72387165 added a very early test version of a YaCy gui component.
14 years ago
sixcooler d88b9606d1 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2923
14 years ago
orbiter 6388a58fc7 better memory management and slightly less (in total and temporary) RAM allocation:
14 years ago
orbiter 5924a0d851 - enhanced concurrency in database index access for multicore
14 years ago
orbiter 55a2536bcf enhancement in drawing speed and reduction of object allocation during drawing
14 years ago
orbiter 610855e362 do not use network graph cache if called from authorized account
14 years ago
orbiter 9ab06bc333 enhancement in sorting efficiency (database root operation): less object allocation
14 years ago
sixcooler 39d96abbb5 fix yacyRelease download
14 years ago
sixcooler 349e4dee9d ... migrating to HttpComponents-Client-4.x ...
14 years ago
sixcooler c29f24a519 ... migrating to HttpComponents-Client-4.x ...
14 years ago
orbiter e7ea3b3cc5 added a buffer for network images to reduced load on yacy.net network image server
15 years ago
orbiter d5c65b17a6 added another network activity visualization: show strong query activity as radiation around peer
15 years ago
orbiter 989948e1a9 fixed generic image parser
15 years ago
orbiter e1015ead2c static access to constants
15 years ago
orbiter 27d8a8b53e removed wrong com.sun.codec class access in generic image parser
15 years ago
orbiter bbf887d879 added generics to UPnP classes
15 years ago
sixcooler 15e8c13526 ... migrating to HttpComponents-Client-4.x ...
15 years ago
mikeworks 6b13101d18 Collage.html: Fixed problem where the German translation broke the action that contained Collage in a form
15 years ago
mikeworks aa663cda4d ConfigUpdate_p.html and ConfigUpdate_p.java: Added check for downloaded releases and disabled buttons in case no new releases available
15 years ago
mikeworks b12db14b9f Added Generics to new net.yacy.upnp.* classes to eliminate compiler warnings
15 years ago
sixcooler b7102eff92 ... migrating to HttpComponents-Client-4.x ...
15 years ago
mikeworks 572e429eff - fixes UPnP not working discussion on forum: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2881
15 years ago
mikeworks 2a20282505 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6987 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus 965aa97993 including sbbi upnplib as source again
15 years ago
lotus 74f6fd229e some comments + debug code
15 years ago
orbiter 60caade056 removed debug output
15 years ago
sixcooler 52718e6dcb ... migrating to HttpComponents-Client-4.x ...
15 years ago
orbiter 89b0f5bce8 fix for exception in http://forum.yacy-websuche.de/viewtopic.php?p=20418#p20418
15 years ago
sixcooler 5fa8038f10 ... migrating to HttpComponents-Client-4.x ...
15 years ago
orbiter dec1419bc3 ;-)
15 years ago
orbiter 22dbbcfa56 better (and corrected) recognition of intranet and internet-addresses. This corrects the isLocal property that is used by network definitions to restrict index ranges to local and global addresses. Address locations (intranet or internet) had been partly identified by the top level domain of the host address. Since intranet addresses can also be addressed using a host name that is in a country domain it is necessary to do a dns resolving for each check. The check is supported by a local dns cache so the intranet/internet check should not affect network traffic too much. To ensure that the cache works properly the cache class was upgraded to better concurrency data structures.
15 years ago
orbiter 8674a65488 removed override directive which caused a compile error in eclipse helios
15 years ago
low012 dc5f0e357c *) fixed SVN properties
15 years ago
low012 01d6b952f0 *) minor changes for easier to read code, no functional changes
15 years ago
low012 0e6fed1fb6 *) less HTML errors (according to https://addons.mozilla.org/de/firefox/addon/249/)
15 years ago
sixcooler 0e56d29335 ... migrating to HttpComponents-Client-4.x ...
15 years ago
sixcooler 2ad5829b26 correct Timeoutparamter at HttpComponents-Client-4.x
15 years ago
sixcooler e1316d12d0 ... migrating to HttpComponents-Client-4.x ...
15 years ago
sixcooler c5c67f0504 start migrating to HttpComponents-Client-4.x
15 years ago
orbiter 7188c54ddb patch to get dht access to developer peers
15 years ago
orbiter 25024d6ab2 fix for problen when accessing the metadata index. The index was not available for all peers with no RAM table copy.
15 years ago
orbiter b6fb239e74 redesign of parser interface:
15 years ago
low012 d4851441b0 *) Added Android packages to parser in order to be able to create a decentralized search for direct downloads of Android apps.
15 years ago
orbiter 150cf42a1b migrated all my LGPL 3 -licensed files to the LGPL 2.1 because LGPL 3 is not compatible to the GPL 2
15 years ago
orbiter 11b7853940 added a configuration page for search heuristics. currently you can switch on there:
15 years ago
orbiter 5d00888c95 - added animated visualization for DHT-in and DHT-out in network graphic
15 years ago
orbiter bf25407fdd added peer hash to internal RSSFeed. The hash will be used to display news activities in the network graphic.
15 years ago
orbiter 1557e0f2d0 - some refactoring for internal RSSFeed (protocol of all actions as seen on status page)
15 years ago
orbiter 5a4684f21f allow words with length >= 2 (you can't search for 'wm' with 3-letter words...)
15 years ago
orbiter 37b8827a7a - removed the UPnP library sources from sbbi and added the jar library again. The library was included to get support for fedora releases, but after this time the fact that the sbbi cannot be part of fedora should be re-discussed. If this will still not be possible, then we may integrate the sbbi UPnP package using reflection.
15 years ago
orbiter dcd01698b4 added a 'transition feature' that shall lower the barrier to move from g**gle to yacy (yes!):
15 years ago
orbiter 103c848af8 enhancements in image drawing speed
15 years ago
orbiter 3a9dc52ac2 added a fascinating new way to search _and_ start a web crawl at the same time:
15 years ago
orbiter 2b4f8f6c06 animated network graphic!
15 years ago
orbiter 777195e8d1 more abstraction for access of LoaderDispatcher and cache
15 years ago
orbiter 7bcfa033c9 more abstraction of the htcache when using the LoaderDispatcher:
15 years ago
orbiter 7e2d6fac12 patch for bad values during local search join
15 years ago
orbiter 2ddb952a5c added the (fixed and anhanced) secondary search process. The process was disabled since some time.
15 years ago
orbiter 58035ef784 fix in snippet loading
15 years ago
orbiter 986d4f34d9 added a consistency check for new queues
15 years ago
orbiter 73f03e05ee fixed a bug in snippet fetch strategy: cache only does not help if resource can only be found in web
15 years ago
orbiter fbf021bb50 redesign of index abstract processing - currently disabled until enough peers have fix in SVN 6928
15 years ago
orbiter 87087f12fe - scanned remote search process and enhanced some data structure and synchronizations here and there
15 years ago
orbiter 7ddb70e7c6 new license for ai.greedy component: LGPL (nobody else than me modified that code)
15 years ago
orbiter b62fb38344 fix for case where no release provider responds during auto-update (caused NPE)
15 years ago
orbiter de4f30bb2e UTF-8 fix
15 years ago
orbiter 3a1cebb598 bugfixes
15 years ago
orbiter 989819a28c - reduced peer-ping time-out from 30 to 10 seconds
15 years ago
orbiter 51332b787d reverted SVN 6869 as discussed with dulcedo in car after LinuxTag:
15 years ago
orbiter b03caaa57a better handling of OOM situations
15 years ago
orbiter 56ff9d5fd4 - extended news size from 512 to 1024 characters
15 years ago
orbiter 353a924760 - changed default memory to 500m
15 years ago
orbiter c71d829bb5 more time-out properties for http connection manager
15 years ago
orbiter 60e71876ad - more abstraction (HashMap -> Map)
15 years ago
orbiter a83772c71b fixes and enhancements for balancer:
15 years ago
orbiter 9cde05418f fixed url crawl list display
15 years ago
orbiter 2eea806005 less errors in image parser
15 years ago
orbiter 30b337fa9f fixes to balancer when crawling filesystem (problem was: host == null)
15 years ago
orbiter 844853243a fixed balancer time guessing
15 years ago
orbiter 3f93a0cc8f redesign of remote proxy settings
15 years ago
orbiter 11639aef35 - added new protocol loader for 'file'-type URLs
15 years ago
orbiter 6950d8a33d fixes to SMB crawler
15 years ago
orbiter bfdb9f4e06 extended statistics on Network servlet page
15 years ago
orbiter 98c1d65415 - show up to 10 locations (maps) after search (instead of a max of 5)
15 years ago
orbiter 9842fab6e4 - fixes to query parameter
15 years ago
orbiter 6ec9ced4cd - fix for multi-word search for locations
15 years ago
orbiter 1defd580bc - added option to localization search to distinguish between a search for a location according to the search word only or for the relation between a web search results and locations found in the metadata fields
15 years ago
low012 ad823a4716 *) minor changes (only cosmetics, no functional changes)
15 years ago
low012 dcac90d2f9 *) removed unnecessary import
15 years ago
orbiter bd0a9df895 fix for bad location double check
15 years ago
orbiter e43e61e502 added another geolocalization data source: GeoNames
15 years ago
orbiter 118d589eff replaced the very very old data structure 'Records' with a simple table to fix the problem from
15 years ago
orbiter 2a8f70f0ca - fix for caching of OSM tiles. if you want that this fix applies to your peer, please delete the crawl profiles
15 years ago
orbiter 2126c03a62 - removed download-limit that can be given for the crawler for non-crawler download tasks. This was necessary because the same procedure was used for other downloads like for the download of dictionary files where a limit is not useful. The limit still stays for the indexer
15 years ago
orbiter 3661cb692c added dictionary loader servlet that can be used to get the geolocalization file:
15 years ago
orbiter 90fa8fd4d4 - support gpx file extension
15 years ago
orbiter 439b44be9e removed exit from computation in ReferenceContainerArray.get merge method
15 years ago
orbiter 7b880d73d0 adjustments to granted query size
15 years ago
orbiter 789c6b26ce added a location search service: using the following servlet/example:
15 years ago
orbiter f23cbd2dab more bugfixes to date parser
15 years ago
orbiter cf43bdc87e This is a large bugfix and enhancement commit to support a better location detection for data
15 years ago
orbiter 6eba2cb96b fix in bmp parser
15 years ago
orbiter c45117f81f fixed dates in metadata
15 years ago
orbiter 0a5fd15703 :-(
15 years ago
orbiter ac16f582aa fix for http://forum.yacy-websuche.de/viewtopic.php?p=20017#p20017
15 years ago
orbiter a7d038bb7a The oai ListFriends source list becomes configurable: just write them into defaults/oaiListFriendsSource.xml
15 years ago
orbiter 7ab207d93a better presentation of search result metadata and fixes to htcache loading
15 years ago
orbiter 5fbf866cae - fixed resumption token generation for oai-pmh import
15 years ago
orbiter fc5efcc05a enhanced and fixed OAI-PMH import
15 years ago
sixcooler c2098f9399 close unused connections if there to many for DHT
15 years ago
orbiter 455a763d7c performance hacks
15 years ago
orbiter 40a8d132d9 tried to fix 100% CPU when calling Balancer.top()
15 years ago
orbiter b6cce08019 fixed a bug in rwi storage data size allocation
15 years ago
orbiter 90c3e5d6f6 - cleanup, removed unused imports
15 years ago
orbiter 3aad50d38e :-(
15 years ago
orbiter 9edd38fbc5 connectionCount limit too low?
15 years ago
orbiter 7a05db0fcb fixed to prevent that too many open connections exist
15 years ago
orbiter a9b9bf667b fix for http://forum.yacy-websuche.de/viewtopic.php?p=19910#p19910
15 years ago
orbiter b18a7606a0 some performance hacks and fixed after reading dump in
15 years ago
orbiter 2bc3cba6f1 - fix for 'do not write to cache' rule.
15 years ago
orbiter 4cd5418963 removed finalize methods because of a hint in
15 years ago
orbiter bfa35d6d20 possible fix for ZURL.list counter
15 years ago
orbiter cff8ed134f added index check to prevent blocking in synchronization
15 years ago
orbiter 65f383e70b some adjustments to the httpc after testing with a very slow httpd
15 years ago
orbiter 5ab5ac80fe fix for NPE in TextParser
15 years ago
orbiter b95ae2518b fix for assert
15 years ago
orbiter 3247f0e901 fix for deadlocks caused by self-blocking access to TreeMap in concurrent environments. The TreeMap was replaced by a ConcurrentHashMap and additional care that the strings are compared all in lowercase
15 years ago
orbiter 027b971bde fix for concurrent quicksort: catch jobs from ThreadPoolExecutor that had been rejected because of full processing queues.
15 years ago
orbiter 8c40f1cb8e self-healing for broken table files (may cause other problems, but better than nothing)
15 years ago
sixcooler 13f5b8e7ba fix for storing/getting bookmark-folders
15 years ago
orbiter 7b69d79727 enhanced remove() operation: in many cases it is not necessary to return the removed object to the called.
15 years ago
orbiter 93ea0a4789 enhanced remove operation in search consequences (which are triggered when the snippet fetch proves that the word has disappeared from the page that was stored in the index)
15 years ago
orbiter 7a59012632 fix for NPE
15 years ago
orbiter 1a6c2f77b4 fix for NPE in statistic servlet
15 years ago
orbiter 64f29f990e a collection of performance hacks and code cleanup:
15 years ago
orbiter 8b8107b2a3 reduced IO-load and synchronization/blocking
15 years ago
orbiter ed07046870 flush only when > 3000 RWIs present + code cleanup
15 years ago
orbiter 3a50b5aa04 enhanced object hash computation
15 years ago
orbiter 1a8a134e0c continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775 and continued in SVN 6790
15 years ago
orbiter dde394a977 - shifted some computation out of synchronization to allow more concurrency
15 years ago
orbiter f204076d25 removed usage of temporary files: causes too much IO
15 years ago
orbiter 48b9371735 changed balancer re-load counter. causes less blocking here doing intranet indexing.
15 years ago
orbiter 650be3599f added a time-out to the RWI cache to flush the cache if it has not been written for ten minutes. This additional dump criteria is necessary because some data sources repeat their vocabulary and may cause that the number of words in a RWI does not increase while the number of references in the RWI set increases. Now the RWI Buffer is flushed every 10 minutes or later if at that time already a dump is ongoing.
15 years ago
orbiter ff6cf24b80 replaced RowSetArray in ObjectIndexCache with RowSet to reduce complexity in MergeIterator. This complexity caused too much computing overhead when the RowSetArray had become very large.
15 years ago
orbiter 0d04ab1422 new access tracking data type strategy; previous data types may have caused deadlocks of httpd when performing statistic cleanups
15 years ago
low012 fc43f3028e *) hopefully fixing NPE issue introduced in r6797
15 years ago
orbiter 55d8e686ea performance hacks
15 years ago
orbiter 2f181d0027 introduced concurrency in HTCACHE storage compression
15 years ago
orbiter 2e26744f4e more concurrency when normalizing RWI entries + cleanup
15 years ago
orbiter 555b333041 fix for wrong count of server processes. may fix non-access problems in some cases
15 years ago
orbiter aa083fc45c try to get a fix for OOM problem in case that there is no real problem with missing memory.
15 years ago
orbiter 70e6222978 more concurrency during search requests
15 years ago
orbiter 4917f96729 fixes for some changes in SVN 6797 that caused NPEs when the bookmarks initialized
15 years ago
low012 dff660441a *) changes for better code readability
15 years ago
low012 15d9ea8375 *) changes for better code readability
15 years ago
low012 2bc459252e *) changes for better code readability
15 years ago
low012 dc93cec3a8 *) Java 1.5 compatibility (see http://forum.yacy-websuche.de/viewtopic.php?f=8&t=2764)
15 years ago
orbiter 67ec58d8e7 search performance enhancement
15 years ago
hermens 4ec0092677 more null == proxy fixes
15 years ago
hermens 2f90f0ad56 Remove asserts blocking proxy use cases
15 years ago
hermens ef467a0303 Another workaround for the second part of http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770
15 years ago
sixcooler eb2a4bb555 workaround(?) for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770&start=0&st=0&sk=t&sd=a&hilit=DefaultCharsetStringPart
15 years ago
orbiter 25aef069a6 continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775
15 years ago
low012 b97ad0f380 *) some minor changes for better code readability
15 years ago
orbiter ba51d140e1 added more info in assert in balancer
15 years ago
orbiter a85c5bb8a7 added support for multiple (fail-over) network definition locations when http-locations are given. multiple locations can be given with a comma-separated list of urls pointing to the network definition file
15 years ago
orbiter 9b3840cb66 performance hacks for the template engine + cleanup
15 years ago
orbiter 5c10f8bc5f enhanced latest hack
15 years ago
orbiter b3238bec83 performance hack for httpd
15 years ago
orbiter 1e8e79b9ef redesign of reference hash (URL-hash) parameter hand-over:
15 years ago
orbiter 72d8e9897b removed unnecessary cache flush call in backend of BufferedRecords
15 years ago
orbiter 749ffbd642 - added another catch case for the index dump and index merge process that should cause non-blocking behavior in case that index dump and/or index merge caused any unexpected exception.
15 years ago
orbiter 9ddb8e4a43 set an option for the java-internal image parser that prevents that the image is cached using the file-system in a temporary file. This should speed up image parsing during image indexing dramatically and should also cause better performance when showing the yacy banner and OSM tiles.
15 years ago
orbiter 312ca5d917 removed flush at end of every rwi entry since this reduces the write performance.
15 years ago
orbiter 0018163c07 moved table row/column matching method from front-end to back-end
15 years ago
orbiter e12f1fd821 - added setting of access rights for executable scripts after auto-installation
15 years ago
orbiter 31e29a8831 - removed synchronization during index dump and index cleaning
15 years ago
orbiter 95f31da8da increase dump cache queue length from 1 to 2
15 years ago
orbiter 6c093d6aed - enhanced domain navigator computation
15 years ago
orbiter bb63c5d075 using a Pattern object with precompiled regular expressions to apply must-match constraints to search results: should speed up pre-sorting of search results and should cause richer search result sets
15 years ago
orbiter e0da0a84b0 performance fix in http parser
15 years ago
orbiter 90dd197ae7 - no latency for local crawls
15 years ago
orbiter bfb518cd47 some refactoring to get the LoaderDispatcher a little bit more independent from the switchboard
15 years ago
orbiter 36bd843ece for for RFC5322 comformance as suggested by Quix0r in http://forum.yacy-websuche.de/viewtopic.php?p=19585#p19585
15 years ago
orbiter c855fc48c6 only load robots.txt for http and http protocol
15 years ago
orbiter 748abfcffa added patches to prevent yacy-protocol DoS settings
15 years ago
orbiter e820ed061a avoiding excessive DNS lookups to determine localhost
15 years ago
orbiter 11983bc936 redesigned some parts of the parser entry point:
15 years ago
orbiter de88200e11 - added Byte Order Mark recognition to serverObjects
15 years ago
orbiter 89b4fff1c2 adopted ant script for new exif library
15 years ago
orbiter 24e5faee75 added exif parsing for jpg images
15 years ago
orbiter 82f76e1296 removed log line
15 years ago
orbiter 0f8004f9da enhanced html parser to recognize a href tags inside header tags
15 years ago
orbiter 3300930fc5 - (almost) fixed FTP crawler
15 years ago
orbiter 1198b9989d bugfixes, more sorttable
15 years ago
orbiter 9623d9e6d2 added a smb loader component for the YaCy crawler
15 years ago
orbiter ae2f3f000f better handling of table copy abandon .. prevent memory leak
15 years ago
orbiter 0769517129 added a robots.txt monitor in the crawler monitor submenu
15 years ago
orbiter 48995e71c4 added soft-auth to general authentication scheme
15 years ago
orbiter 72f00dee59 removed never-used server access account function
15 years ago
orbiter 57e1eae95e longer time-out for url fetching .. may help to show all that links that the statistic say for a search result
15 years ago
orbiter 9e639603e3 after frequent occurrences of 100% CPU usages and permanent blockings I try to disable a function in a method that may cause the problem when calling an external library (apache http client 3.x). The thread dump that shows the problem is attached here.
15 years ago
orbiter 4144927d94 show less errors
15 years ago
orbiter b88f5fbb4b slightly changed crawling policy
15 years ago
orbiter de01fe0e6d fix for bug in url parser
15 years ago
orbiter 7684a575c4 fix for deletion of error database each time when YaCy starts up
15 years ago
orbiter f561e340c6 show more results of single domains when not authorized fully (up to 100)
15 years ago
orbiter c4bdb1e7f2 added one more option in ViewFile to show an iframe like for the orginal web page content but using the cache than the direct link to the content in the web. Upgraded the very old and previously not any more used CacheResource_p servlet to a new and working version.
15 years ago
orbiter c09a995930 better logging of double occurrences of urls in the crawler
15 years ago
orbiter 1bbe14d23f SVN 6716 unfortunately contained parts of the unfinished SMB integration. To fix compile errors the remaining parts of the SMB implementation stub is added with this commit.
15 years ago
orbiter 884b262130 - added a new Wiki Namespace Navigator
15 years ago
orbiter 617dfbbd06 allo 'authorization by encoded password' also if requesting client is not from localhost but from the same host as yacy is running on.
15 years ago
orbiter 270fb38674 - fixed some bugs in Table viewer
15 years ago
orbiter 599c3766c4 added authentication to automated API call
15 years ago
orbiter 727dd9b193 - fixed a bug in robots.txt parser
15 years ago
orbiter 54af9e6b49 - added parsing of robots meta-tag in html headers to detect a noindexing request
15 years ago
orbiter 46c4f8b68a better look-ahead into the crawl queue: show more on crawl monitor
15 years ago
lotus 7b546415dc added svn6695 for windows
15 years ago
orbiter f175f9a2d3 changed way how number of search requests are counted:
15 years ago
orbiter 84222e3b4f fix for auto-updater: delete old libraries before copy of new one
15 years ago