yacy_search_server/doc/bot.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<title>YaCy: Bot Info</title>
<meta http-equiv="content-type" content="text/html;charset=iso-8859-1">
<!-- <meta name="Content-Language" content="German, Deutsch, de, at, ch"> -->
<meta name="Content-Language" content="English, Englisch">
<meta name="keywords" content="YaCy HTTP Proxy search engine spider indexer java network open free download Mac Windwos Software development">
<meta name="description" content="YaCy P2P Web Search: Bot Info">
<meta name="copyright" content="Michael Christen">
<script src="navigation.js" type="text/javascript"></script>
<link rel="stylesheet" media="all" href="style.css">
<!-- Realisation: Michael Christen; Contact: mc<at>anomic.de-->
</head>
<body bgcolor="#fefefe" marginheight="0" marginwidth="0" leftmargin="0" topmargin="0">
<SCRIPT LANGUAGE="JavaScript1.1"><!--
globalheader();
//--></SCRIPT>
<NOSCRIPT>
<table border="0" cellspacing="0" cellpadding="0" width="100%">
<tr><td></td></tr>
<tr><td height="1" bgcolor="#000000"></td></tr>
<tr><td>
<!-- start headline -->
<table bgcolor="#4070A0" border="0" cellspacing="0" cellpadding="0" width="100%">
<tr><td width="180" height="80" rowspan="3"><a href="http://www.yacy.net"><img border="0" src="grafics/yacy.gif" align="top"></a></td>
<td></td><td width="120"></td></tr>
</table>
<!-- end headline -->
</td></tr>
<tr><td height="2"></td></tr>
<tr><td>
<table border="0" cellspacing="0" cellpadding="0" width="100%">
<tr>
<td width="100" valign="top">
<!-- start lmenue -->
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr><td height="2"></td></tr>
<tr><td height="20" class="white" bgcolor="#BDCDD4" valign="middle">&nbsp;<a href="index.html" class="dark">Main Index</a></td></tr>
</table>
<!-- end lmenue -->
</td>
<td width="10" valign="top"></td>
<td valign="top">
<table border="0" cellspacing="0" cellpadding="0" width="100%">
<tr><td height="2"></td></tr>
<tr><td><br>
</NOSCRIPT>
<!-- ----- HERE STARTS CONTENT PART ----- -->
<h2>YaCy-Bot</h2><br>
You are most propably here, because you have seen accesses from an user with the useragent "yacybot" in your webserver-logs.
<h3>What is yacybot?</h3>
"yacybot" is the useragent of the software <a href="http://yacy.net/yacy/">YaCy</a>, which is an approach to build a Peer-to-Peer based websearch-engine.
"yacybot" is the useragent of the YaCy webcrawlers which are maintained by the YaCy users and work the same way as e.g. the googlebot.
<h3>How to control yacybot?</h3>
YaCy follows the rules defined in a <b>robots.txt</b> file.<br>
If you do not wish that YaCy crawls your website, please set up such a file. It is the standard way to control all web-spiders like Google, Yahoo! and also YaCy.<br>
In case you don't know how to make such a file please read <a href="http://www.robotstxt.org/">http://www.robotstxt.org/</a> for further instructions.<br>
<br>
After some hours all yacybots will obey your instructions.
<h3>This didn't help me.</h3>
If there are any questions left please visit our <a href="http://forum.yacy.net">forum</a> and ask for help.
<!-- ----- HERE ENDS CONTENT PART ----- -->
<SCRIPT LANGUAGE="JavaScript1.1"><!--
globalfooter();
//--></SCRIPT>
<NOSCRIPT>
<br><br></td></tr></table>
</td>
<td width="10" valign="top">
</td>
</tr></table>
</td></tr></table>
</NOSCRIPT>
</body>
</html>
added yacybot info-page from Roland git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3462 6c8d7289-2bf4-0310-a012-ef5d649a1542 18 years ago			`<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">`
			`<html>`
			`<head>`
			`<title>YaCy: Bot Info</title>`
			`<meta http-equiv="content-type" content="text/html;charset=iso-8859-1">`
			`<!-- <meta name="Content-Language" content="German, Deutsch, de, at, ch"> -->`
			`<meta name="Content-Language" content="English, Englisch">`
			`<meta name="keywords" content="YaCy HTTP Proxy search engine spider indexer java network open free download Mac Windwos Software development">`
			`<meta name="description" content="YaCy P2P Web Search: Bot Info">`
			`<meta name="copyright" content="Michael Christen">`
			`<script src="navigation.js" type="text/javascript"></script>`
			`<link rel="stylesheet" media="all" href="style.css">`
			`<!-- Realisation: Michael Christen; Contact: mc<at>anomic.de-->`
			`</head>`
			`<body bgcolor="#fefefe" marginheight="0" marginwidth="0" leftmargin="0" topmargin="0">`
			`<SCRIPT LANGUAGE="JavaScript1.1"><!--`
			`globalheader();`
			`//--></SCRIPT>`
			`<NOSCRIPT>`
			`<table border="0" cellspacing="0" cellpadding="0" width="100%">`
			`<tr><td></td></tr>`
			`<tr><td height="1" bgcolor="#000000"></td></tr>`
			`<tr><td>`
			`<!-- start headline -->`
			`<table bgcolor="#4070A0" border="0" cellspacing="0" cellpadding="0" width="100%">`
			`<tr><td width="180" height="80" rowspan="3"><a href="http://www.yacy.net"><img border="0" src="grafics/yacy.gif" align="top"></a></td>`
			`<td></td><td width="120"></td></tr>`
			`</table>`
			`<!-- end headline -->`
			`</td></tr>`
			`<tr><td height="2"></td></tr>`
			`<tr><td>`
			`<table border="0" cellspacing="0" cellpadding="0" width="100%">`
			`<tr>`
			`<td width="100" valign="top">`
			`<!-- start lmenue -->`
			`<table width="100%" border="0" cellspacing="0" cellpadding="0">`
			`<tr><td height="2"></td></tr>`
			`<tr><td height="20" class="white" bgcolor="#BDCDD4" valign="middle"> <a href="index.html" class="dark">Main Index</a></td></tr>`
			`</table>`
			`<!-- end lmenue -->`
			`</td>`
			`<td width="10" valign="top"></td>`
			`<td valign="top">`
			`<table border="0" cellspacing="0" cellpadding="0" width="100%">`
			`<tr><td height="2"></td></tr>`
			`<tr><td><br>`
			`</NOSCRIPT>`
			`<!-- ----- HERE STARTS CONTENT PART ----- -->`
			`<h2>YaCy-Bot</h2><br>`
			`You are most propably here, because you have seen accesses from an user with the useragent "yacybot" in your webserver-logs.`
			`<h3>What is yacybot?</h3>`
			`"yacybot" is the useragent of the software <a href="http://yacy.net/yacy/">YaCy</a>, which is an approach to build a Peer-to-Peer based websearch-engine.`
			`"yacybot" is the useragent of the YaCy webcrawlers which are maintained by the YaCy users and work the same way as e.g. the googlebot.`
			`<h3>How to control yacybot?</h3>`
			`YaCy follows the rules defined in a <b>robots.txt</b> file.<br>`
			`If you do not wish that YaCy crawls your website, please set up such a file. It is the standard way to control all web-spiders like Google, Yahoo! and also YaCy.<br>`
			`In case you don't know how to make such a file please read <a href="http://www.robotstxt.org/">http://www.robotstxt.org/</a> for further instructions.<br>`
			`<br>`
			`After some hours all yacybots will obey your instructions.`
			`<h3>This didn't help me.</h3>`
- documentaton changes (removed old forum links) - different handling of link quotation - different handling of link normalization - enhanced html/unicode en/de-coding git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3993 6c8d7289-2bf4-0310-a012-ef5d649a1542 18 years ago			`If there are any questions left please visit our <a href="http://forum.yacy.net">forum</a> and ask for help.`
added yacybot info-page from Roland git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3462 6c8d7289-2bf4-0310-a012-ef5d649a1542 18 years ago			`<!-- ----- HERE ENDS CONTENT PART ----- -->`
			`<SCRIPT LANGUAGE="JavaScript1.1"><!--`
			`globalfooter();`
			`//--></SCRIPT>`
			`<NOSCRIPT>`
			`<br><br></td></tr></table>`
			`</td>`
			`<td width="10" valign="top">`
			`</td>`
			`</tr></table>`
			`</td></tr></table>`
			`</NOSCRIPT>`
			`</body>`
			`</html>`