|
|
@ -10,7 +10,7 @@
|
|
|
|
<h2>Exclude Web-Spiders</h2>
|
|
|
|
<h2>Exclude Web-Spiders</h2>
|
|
|
|
<p>
|
|
|
|
<p>
|
|
|
|
Here you can set up a robots.txt for all webcrawlers that try to access the webinterface of your peer.
|
|
|
|
Here you can set up a robots.txt for all webcrawlers that try to access the webinterface of your peer.
|
|
|
|
<span class="tt">robots.txt</span> is a volunteer agreement most search-engines (including YaCy) follow.
|
|
|
|
<span class="tt">robots.txt</span> is a voluntary agreement most search-engines (including YaCy) follow.
|
|
|
|
It disallows crawlers to access webpages or even entire domains.
|
|
|
|
It disallows crawlers to access webpages or even entire domains.
|
|
|
|
</p>
|
|
|
|
</p>
|
|
|
|
#(error)#
|
|
|
|
#(error)#
|
|
|
|