boxtec
/
yacy_search_server
mirror of https://github.com/yacy/yacy_search_server
4
1
Fork 0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '2d9e577ad0'
${ noResults }
yacy_search_server/source/net/yacy/crawler/robots
History
Michael Peter Christen 2d9e577ad0
replaced the custom robots.txt loader by the standard http loader 		13 years ago
..
RobotsTxt.java replaced the custom robots.txt loader by the standard http loader 13 years ago
RobotsTxtEntry.java refactoring 13 years ago
RobotsTxtParser.java replaced the custom robots.txt loader by the standard http loader 13 years ago