You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter e81be7d4f2 added many missing user-agent declarations for yacy http client connections. ... the most important fix was the addition of the yacybot user-agent for robots.txt loading, because web masters look for that access to see if the crawler behaves correctly. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4968 6c8d7289-2bf4-0310-a012-ef5d649a1542		17 years ago
..
AbstractImporter.java	…
Balancer.java	added options to configure the 'corporate identity'-icons, the home page link and the greeting line from	17 years ago
CrawlEntry.java	…
CrawlProfile.java	- WatchCrawler shows status without JavaScript	17 years ago
CrawlQueues.java	- modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order	17 years ago
CrawlStacker.java	some code-cleanup and possible speed enhancements in different core methods	17 years ago
ErrorURL.java	…
FTPLoader.java	…
HTTPLoader.java	added many missing user-agent declarations for yacy http client connections.	17 years ago
Importer.java	…
ImporterException.java	…
ImporterManager.java	…
IndexingStack.java	added options to configure the 'corporate identity'-icons, the home page link and the greeting line from	17 years ago
LoaderMessage.java	…
NoticeURLImporter.java	…
NoticedURL.java	- modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order	17 years ago
ProtocolLoader.java	…
ResultImages.java	…
ResultURLs.java	some code-cleanup and possible speed enhancements in different core methods	17 years ago
RobotsTxt.java	refactoring:	17 years ago
SitemapImporter.java	…
ZURL.java	some code-cleanup and possible speed enhancements in different core methods	17 years ago