yacy_search_server

Commit Graph

Author	SHA1	Message	Date
karlchenofhell	88245e44d8	- improved version of robots.txt (delete your old htroot/robots.txt before updating): - robots.txt is a servlet now - no need to rewrite the whole file each time a section is added or removed - user-defined disallows, added manually, won't be overwritten anymore - new config-setting: httpd.robots.txt, holding names of the disallowed sections git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542	18 years ago

Author

SHA1

Message

Date

karlchenofhell

88245e44d8

- improved version of robots.txt (delete your old htroot/robots.txt before updating):

- robots.txt is a servlet now
  - no need to rewrite the whole file each time a section is added or removed
  - user-defined disallows, added manually, won't be overwritten anymore
- new config-setting: httpd.robots.txt, holding names of the disallowed sections

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542

1 Commits (159bd0cab56a10deb9f06bab14b4d0ed8b90101e)