orbiter
|
c93f10417a
|
add a bookmark automatically each time a new crawl is started
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8063 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
13 years ago |
|
orbiter
|
017a01714d
|
- enhanced logging in robots.txt parser for remote debugging
- robots.txt is now more robust against database operations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
13 years ago |
|
cominch
|
cef8ebc41d
|
getpageinfo: Checks if there is a OAI repository behind the URL.
This check is only performed if oai parameter is set when calling e.g. getpageinfo_p.xml?actions=oai
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8039 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
14 years ago |
|
orbiter
|
eb1c7c041d
|
write info about robots.txt evaluation into getpageinfo_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
14 years ago |
|
orbiter
|
f8b8c82421
|
- refactoring of getpageinfo_p.xml (moved out of util)
- added more logging in getpageinfo_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8037 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
14 years ago |
