You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter 727dd9b193 - fixed a bug in robots.txt parser ... - moved storage of robots.txt entries to WorkTables, so it is now possible to browse the robots entries with the table browser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6710 6c8d7289-2bf4-0310-a012-ef5d649a1542		15 years ago
..
crawler	- fixed a bug in robots.txt parser	15 years ago
data	- fixed a bug in robots.txt parser	15 years ago
http	fix for display problem in Firefox on MacOS X	15 years ago
net	added an api recording function: it shall be possible to record	15 years ago
search	- fixed a bug in robots.txt parser	15 years ago
server	- added Tables abstraction in data.Tables.java	15 years ago
tools	applied code changes that are recommended by PMD	15 years ago
yacy	- added parsing of robots meta-tag in html headers to detect a noindexing request	15 years ago