You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter b2fe4b7b1a added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer ... (directly or indirectly) and it grants a crawl-delay of 0. Then all forced pause mechanisms in YaCy are switched off and the domain is crawled at full speed. crawl delay values can be assigned to either - all yacy peers using the user-agent yacybot - a specific peer with peer name <peer-name>.yacy or - a specific peer with peer hash <peer-hash>.yacyh git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7639 6c8d7289-2bf4-0310-a012-ef5d649a1542		14 years ago
..
anomic	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	14 years ago