You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter b2fe4b7b1a added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer ... (directly or indirectly) and it grants a crawl-delay of 0. Then all forced pause mechanisms in YaCy are switched off and the domain is crawled at full speed. crawl delay values can be assigned to either - all yacy peers using the user-agent yacybot - a specific peer with peer name <peer-name>.yacy or - a specific peer with peer hash <peer-hash>.yacyh git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7639 6c8d7289-2bf4-0310-a012-ef5d649a1542		14 years ago
..
getpageinfo_p.java	added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer	14 years ago
getpageinfo_p.xml	replaced auto-dom filter with easy-to-understand Site Link-List crawler option	15 years ago
ynetSearch.java	preparations to move the HTCache into cora:	15 years ago
ynetSearch.json	- fixed YaCy-UI sciencenet search	16 years ago
ynetSearch.xml	moved all servlets from htroot/xml to htroot/api	16 years ago