6 Commits (b743e6d79f9314ed17116478afc3346da10c0e9e)

Author	SHA1	Message	Date
cominch	8a91f4fa42	local robots.txt: disallow external crawlers to follow the URL proxy	13 years ago
cominch	6456a1656a	changed local robots.txt to prevent external crawlers to submit random ... search queries	13 years ago
orbiter	0ab6a462ee	- added a missing entry in YaCy interface robots.txt for bookmarks ... - changed default robots.txt deny list to include some more interface pages because the loading of such pages are a peer load issue for YaCy when crawlers come by and information on these pages are not useful for public search. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7112 6c8d7289-2bf4-0310-a012-ef5d649a1542	15 years ago
karlchenofhell	669f840eab	- added ViewProfile / Impressum (default on) to local peer's robots.txt ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542	18 years ago
karlchenofhell	93e1ad2bca	- fix for last commit ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3424 6c8d7289-2bf4-0310-a012-ef5d649a1542	18 years ago
karlchenofhell	88245e44d8	- improved version of robots.txt (delete your old htroot/robots.txt before updating): ... - robots.txt is a servlet now - no need to rewrite the whole file each time a section is added or removed - user-defined disallows, added manually, won't be overwritten anymore - new config-setting: httpd.robots.txt, holding names of the disallowed sections git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542	18 years ago