You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
yacy_search_server/source/de/anomic/crawler
orbiter a6719dfd2b
- refactoring of robots parser
17 years ago
..
AbstractImporter.java refactoring: 17 years ago
Balancer.java - refactoring of robots parser 17 years ago
CrawlEntry.java fix for npe in crawler 17 years ago
CrawlProfile.java - WatchCrawler shows status without JavaScript 17 years ago
CrawlQueues.java - refactoring of robots parser 17 years ago
CrawlStacker.java - refactoring of robots parser 17 years ago
ErrorURL.java refactoring: moved all crawler classes into their own package 17 years ago
FTPLoader.java - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet), 17 years ago
HTTPLoader.java added many missing user-agent declarations for yacy http client connections. 17 years ago
Importer.java refactoring: 17 years ago
ImporterException.java refactoring: 17 years ago
ImporterManager.java - organize imports 17 years ago
IndexingStack.java added options to configure the 'corporate identity'-icons, the home page link and the greeting line from 17 years ago
LoaderMessage.java refactoring: moved all crawler classes into their own package 17 years ago
NoticeURLImporter.java - added option to switch between yacy networks, especially between the two default networks (freeworld and intranet), 17 years ago
NoticedURL.java - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 17 years ago
ProtocolLoader.java more protection against remote shutdown attacks: prevent loading using the crawler 17 years ago
ResultImages.java refactoring: moved all crawler classes into their own package 17 years ago
ResultURLs.java some code-cleanup and possible speed enhancements in different core methods 17 years ago
RobotsTxt.java - refactoring of robots parser 17 years ago
SitemapImporter.java refactoring: 17 years ago
ZURL.java - refactoring of robots parser 17 years ago
robotsParser.java - refactoring of robots parser 17 years ago