You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
yacy_search_server/source/de/anomic/crawler
orbiter 2b32248079
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1516&p=10545#p10545
16 years ago
..
AbstractImporter.java
Balancer.java
CrawlEntry.java removed active profiling in kelondroRowSet 16 years ago
CrawlProfile.java - added fast fingerprint computation for files (any). Will be used in new index dump method 16 years ago
CrawlQueues.java
CrawlStacker.java fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1516&p=10545#p10545 16 years ago
FTPLoader.java
HTTPLoader.java
Importer.java
ImporterException.java
ImporterManager.java
IndexingStack.java
LoaderMessage.java
NoticeURLImporter.java
NoticedURL.java
ProtocolLoader.java removed the permanent cache flush and replaced it with a periodic cache flush 16 years ago
ResourceObserver.java
ResultImages.java
ResultURLs.java
RobotsTxt.java
SitemapImporter.java
ZURL.java
robotsParser.java * Robots.txt: don't interpret Crawl-Delays for other robots 16 years ago