You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter a82a93f2fc - better url double check in crawler ... - more logging for error urls git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7032 6c8d7289-2bf4-0310-a012-ef5d649a1542		15 years ago
..
retrieval	- better url double check in crawler	15 years ago
AbstractImporter.java	- cleanup, removed unused imports	15 years ago
Balancer.java	- better url double check in crawler	15 years ago
CrawlProfile.java	more abstraction of the htcache when using the LoaderDispatcher:	15 years ago
CrawlQueues.java	- better url double check in crawler	15 years ago
CrawlStacker.java	- better url double check in crawler	15 years ago
CrawlSwitchboard.java	fixed a bug in snippet fetch strategy: cache only does not help if resource can only be found in web	15 years ago
Importer.java	refactoring:	17 years ago
ImporterException.java	added final where possible	17 years ago
ImporterManager.java	*) some minor changes for better code readability	15 years ago
Latency.java	better (and corrected) recognition of intranet and internet-addresses. This corrects the isLocal property that is used by network definitions to restrict index ranges to local and global addresses. Address locations (intranet or internet) had been partly identified by the top level domain of the host address. Since intranet addresses can also be addressed using a host name that is in a country domain it is necessary to do a dns resolving for each check. The check is supported by a local dns cache so the intranet/internet check should not affect network traffic too much. To ensure that the cache works properly the cache class was upgraded to better concurrency data structures.	15 years ago
NoticedURL.java	- better url double check in crawler	15 years ago
ResourceObserver.java	allow global search if res. observer disabled index transmission	15 years ago
ResultImages.java	redesign of parser interface:	15 years ago
ResultURLs.java	- more abstraction (HashMap -> Map)	15 years ago
RobotsEntry.java	redesign of remote proxy settings	15 years ago
RobotsTxt.java	... migrating to HttpComponents-Client-4.x ...	15 years ago
SitemapImporter.java	applied code changes that are recommended by PMD	15 years ago
ZURL.java	- better url double check in crawler	15 years ago
robotsParser.java	- fixed a bug in robots.txt parser	15 years ago