You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Michael Peter Christen e6a87e0426 enhanced crawler ... a main problem when crawling is long waiting time cuased by crawl-delay values from robots.txt entries. that attribute is not supported by google and interpreted by yandex and bing in different ways. In large crawls there is always one host which blocks the whole crawl with extreme large values. YaCy now still obeys crawl-delay but limits them to 10 seconds. Additionally the blocking logic when loading new robots.txt was analyzed and a deadlock was removed. Furthermore the construction of new queue lists was redesigned and it was ensured that always a large list of different hosts for host-balancing is provided for the loader.		4 years ago
..
bayes	Fixed some JavaDocs broken links.	8 years ago
date	fix for json importer	4 years ago
document	Added "Show Metadata" links to the ViewFile.html links mode	6 years ago
federate	enhanced crawler	4 years ago
geo	Ensure proper closing of file input streams.	8 years ago
language	Added basic support for autotagging microdata annotated item types.	7 years ago
lod	Provide user interface messages on vocabulary creation read/write errors	7 years ago
order	fixed some newInstance() warnings	4 years ago
plugin	…
protocol	added hazelcast and some modifications to align legacy YaCy with	4 years ago
sorting	replaced new Number(Number) with Number.instanceOf	4 years ago
storage	removes some warnings: unused imports, params	5 years ago
util	fix for jdk 14 bug	4 years ago