theli
|
023be89586
|
*) Bugfix for "Robots.txt wird immer wieder geladen"
See: http://www.yacy-forum.de/viewtopic.php?p=10241#10233
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@794 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
orbiter
|
dc474aa22f
|
various bug-fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
rramthun
|
9dfbd93c7b
|
Updated german language file
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@748 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
theli
|
2cd695f376
|
*) Bugfix path-entries of robots.txt were not decoded correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@676 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
theli
|
f8ad65eae1
|
*) First trial implementation of robots.txt support
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@674 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
9300689dde
|
bugfix *gr*
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@662 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
ebc39a7b9a
|
minor fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@659 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
f90f699ab1
|
missing package line.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@655 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
06a451768f
|
a simple robotsParser.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@652 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
