You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen e6a87e0426 enhanced crawler ... a main problem when crawling is long waiting time cuased by crawl-delay values from robots.txt entries. that attribute is not supported by google and interpreted by yandex and bing in different ways. In large crawls there is always one host which blocks the whole crawl with extreme large values. YaCy now still obeys crawl-delay but limits them to 10 seconds. Additionally the blocking logic when loading new robots.txt was analyzed and a deadlock was removed. Furthermore the construction of new queue lists was redesigned and it was ensured that always a large list of different hosts for host-balancing is provided for the loader.		4 years ago
..
data	enhanced crawler	4 years ago
retrieval	Merge pull request #404 from lnceballosz/master	4 years ago
robots	enhanced crawler	4 years ago
Balancer.java	enhanced crawler	4 years ago
CrawlStacker.java	enhanced crawler	4 years ago
CrawlStarterFromScraper.java	adding SPDX license and copyright headers	4 years ago
CrawlSwitchboard.java	adding SPDX license and copyright headers	4 years ago
FileCrawlStarterTask.java	adding SPDX license and copyright headers	4 years ago
HarvestProcess.java	adding SPDX license and copyright headers	4 years ago
HostBalancer.java	enhanced crawler	4 years ago
HostQueue.java	enhanced crawler	4 years ago
IllegalCrawlProfileException.java	adding SPDX license and copyright headers	4 years ago
LegacyBalancer.java	enhanced crawler	4 years ago
RecrawlBusyThread.java	enhanced crawler	4 years ago