boxtec
/
yacy_search_server
mirror of https://github.com/yacy/yacy_search_server
4
1
Fork 0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'ac5fa9fe48'
${ noResults }
yacy_search_server/source/net/yacy/crawler/robots
History
Michael Peter Christen af465cdca5
fix for wrong robots.txt loading for https protocol 		12 years ago
..
RobotsTxt.java fix for wrong robots.txt loading for https protocol 12 years ago
RobotsTxtEntry.java better balancing and duetime-cumputation also for no-delay intranet 12 years ago
RobotsTxtParser.java replaced the custom robots.txt loader by the standard http loader 12 years ago