theli
|
023be89586
|
*) Bugfix for "Robots.txt wird immer wieder geladen"
See: http://www.yacy-forum.de/viewtopic.php?p=10241#10233
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@794 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
orbiter
|
3fcc95a82c
|
integrated crawl-profiles db in memory-performance monitor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@788 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
theli
|
fe6a6abc0b
|
*) Adding robots.txt db to Performance Settings for Memory menue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@785 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
orbiter
|
3274ae725e
|
increased cache size of robots database; however, this should be integrated into new memory control
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@784 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
orbiter
|
2148c0cf49
|
replaced kelondro storage core; much less objects in kelondro cache now; less IO from DB
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@724 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
theli
|
0aafb83edc
|
*) Bugfix for robots.txt isDisallowed Check.
Setting path to "/" if it is null or empty.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@677 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
theli
|
f8ad65eae1
|
*) First trial implementation of robots.txt support
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@674 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
