You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen e6a87e0426 enhanced crawler ... a main problem when crawling is long waiting time cuased by crawl-delay values from robots.txt entries. that attribute is not supported by google and interpreted by yandex and bing in different ways. In large crawls there is always one host which blocks the whole crawl with extreme large values. YaCy now still obeys crawl-delay but limits them to 10 seconds. Additionally the blocking logic when loading new robots.txt was analyzed and a deadlock was removed. Furthermore the construction of new queue lists was redesigned and it was ensured that always a large list of different hosts for host-balancing is provided for the loader.		4 years ago
..
Cache.java	…
CrawlProfile.java	…
CrawlQueues.java	…
Latency.java	enhanced crawler	4 years ago
NoticedURL.java	removed call to class.finalize() because of deprecation in java 9	4 years ago
ResultImages.java	…
ResultURLs.java	…
Snapshots.java	…
Transactions.java	adding SPDX license and copyright headers	4 years ago