boxtec
/
yacy_search_server
mirror of https://github.com/yacy/yacy_search_server
4
1
Fork 0
Code Issues Projects Releases Wiki Activity

fixed doku link

Browse Source
pull/419/head
Michael Peter Christen 4 years ago
parent c4659f0fb0
commit 3959d43a5c
6 changed files with 191 additions and 187 deletions
Show Stats Download Patch File Download Diff File

4
htroot/CrawlStartExpert.html
Unescape View File

@ -217,7 +217,7 @@
#%env/templates/submenuIndexCreate.template%#
<div id="api">
<a href="http://www.yacy-websearch.net/wiki/index.php/Dev:APICrawler" id="apilink" target="_blank"><img src="env/grafics/api.png" width="60" height="40" alt="API"/></a>
<a href="https://wiki.yacy.net/index.php/Dev:APICrawler" id="apilink" target="_blank"><img src="env/grafics/api.png" width="60" height="40" alt="API"/></a>
<span>Click on this API button to see a documentation of the POST request parameter for crawl starts.</span>
</div>
@ -228,7 +228,7 @@
You can define URLs as start points for Web page crawling and start crawling here.
"Crawling" means that YaCy will download the given website, extract all links in it and then download the content behind these links.
This is repeated as long as specified under "Crawling Depth".
A crawl can also be started using wget and the <a href="http://www.yacy-websearch.net/wiki/index.php/Dev:APICrawler" target="_blank">post arguments</a> for this web page.
A crawl can also be started using wget and the <a href="https://wiki.yacy.net/index.php/Dev:APICrawler" target="_blank">post arguments</a> for this web page.
</p>
<form id="Crawler" action="Crawler_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">

2
htroot/IndexFederated_p.html
Unescape View File

@ -34,7 +34,7 @@
If you switch off this index, a remote Solr must be activated.</dd>
<dt>Use remote Solr server(s)&nbsp;<input type="checkbox" name="solr.indexing.solrremote" id="solr_indexing_solrremote" #(solr.indexing.solrremote.checked)#:: checked="checked"#(/solr.indexing.solrremote.checked)# onclick="if(!document.getElementById('config').solr_indexing_solrremote.checked) {document.getElementById('config').core_service_fulltext.checked = true;}"/></dt>
<dd>It's easy to <a href="http://www.yacy-websearch.net/wiki/index.php/Dev:Solr" target="_blank">attach an external Solr to YaCy</a>.
<dd>It's easy to <a href="https://wiki.yacy.net/index.php/Dev:Solr" target="_blank">attach an external Solr to YaCy</a>.
This external Solr can be used instead the internal Solr. It can also be used additionally to the internal Solr, then both Solr indexes are mirrored.
</dd>

2
locales/master.lng.xlf
Unescape View File

@ -4935,7 +4935,7 @@
<source>Index Size</source>
</trans-unit>
<trans-unit id="e587b435" xml:space="preserve" approved="no" translate="yes">
<source>It's easy to &lt;a href="http://www.yacy-websearch.net/wiki/index.php/Dev:Solr" target="_blank"&gt;attach an external Solr to YaCy&lt;/a&gt;.</source>
<source>It's easy to &lt;a href="https://wiki.yacy.net/index.php/Dev:Solr" target="_blank"&gt;attach an external Solr to YaCy&lt;/a&gt;.</source>
</trans-unit>
<trans-unit id="35e7810" xml:space="preserve" approved="no" translate="yes">
<source>This external Solr can be used instead the internal Solr. It can also be used additionally to the internal Solr, then both Solr indexes are mirrored.</source>

2
locales/ru.lng
Unescape View File

@ -578,7 +578,7 @@ Use remote Solr server(s)==Использовать удалённую базу
Solr Hosts==Хосты Solr
Solr Host Administration Interface==Интерфейс управления Solr
Index Size==Документов в индексе
It's easy to <a href="http://www.yacy-websearch.net/wiki/index.php/Dev:Solr" target="_blank">attach an external Solr to YaCy</a>.==Присоединить внешнюю базу Solr <a href="http://www.yacy-websearch.net/wiki/index.php/Dev:Solr" target="_blank">просто</a>.
It's easy to <a href="https://wiki.yacy.net/index.php/Dev:Solr" target="_blank">attach an external Solr to YaCy</a>.==Присоединить внешнюю базу Solr <a href="https://wiki.yacy.net/index.php/Dev:Solr" target="_blank">просто</a>.
This external Solr can be used instead the internal Solr. It can also be used additionally to the internal Solr, then both Solr indexes are mirrored.==Внешняя база данных Solr будет использоваться вместо встроенной. Вы также можете использовать дополнительно встроенную базу, но тогда индексы будут сохраняться в обе базы.
Solr URL(s)==Ссылки на базу Solr
You can set one or more Solr targets here which are accessed as a shard. For several targets, list them using a ',' (comma) as separator.==Вы можете установить одну или более баз Solr, которые будут доступны распределённо. Адреса нескольких баз указывайте через запятую.

14
startYACY.sh
Unescape View File

@ -184,16 +184,20 @@ fi
#turn on MMap for Solr if OS is a 64bit OS
if [ -n "`uname -m | grep 64`" ]; then JAVA_ARGS="$JAVA_ARGS -Dsolr.directoryFactory=solr.MMapDirectoryFactory"; fi
if [ ! -f $CONFIGFILE -a -f "$YACY_PARENT_DATA_PATH/DATA/SETTINGS/httpProxy.conf" ]
then
# old config if new does not exist
CONFIGFILE="$YACY_PARENT_DATA_PATH/DATA/SETTINGS/httpProxy.conf"
fi
if [ -f $CONFIGFILE ]
then
# startup memory
if [ -z "$YACY_JAVASTART_XMX" ]
then
# When YACY_JAVASTART_XMX is not set or empty:
# Read from $CONFIGFILE
j="`grep javastart_Xmx $CONFIGFILE | sed 's/^[^=]*=//'`";
if [ -n "$j" ]; then JAVA_ARGS="-$j $JAVA_ARGS"; fi;
else
# use the YACY_JAVASTART_XMX variable
JAVA_ARGS="-$YACY_JAVASTART_XMX $JAVA_ARGS"
fi
# Priority
j="`grep javastart_priority $CONFIGFILE | sed 's/^[^=]*=//'`";

42
test/java/net/yacy/document/parser/GenericXMLParserTest.java
Unescape View File

@ -23,9 +23,9 @@
package net.yacy.document.parser;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.assertFalse;
import java.io.ByteArrayInputStream;
import java.io.File;
@ -58,7 +58,7 @@ public class GenericXMLParserTest {
@Before
public void setUp() {
this.parser = new GenericXMLParser();
parser = new GenericXMLParser();
}
/**
@ -77,7 +77,7 @@ public class GenericXMLParserTest {
FileInputStream inStream = new FileInputStream(new File(folder, fileName));
DigestURL location = new DigestURL("http://localhost/" + fileName);
try {
Document[] documents = this.parser.parse(location, "text/xml", null, new VocabularyScraper(), 0,
Document[] documents = parser.parse(location, "text/xml", null, new VocabularyScraper(), 0,
inStream);
assertNotNull("Parser result must not be null for file " + fileName, documents);
assertNotNull("Parsed text must not be empty for file " + fileName, documents[0].getTextString());
@ -138,7 +138,7 @@ public class GenericXMLParserTest {
*/
byte[] encodedXML = ("<?xml version=\"1.0\" encoding=\"utf-8\"?>" + UMLAUT_TEXT_TAG)
.getBytes(StandardCharsets.UTF_8);
testCharsetDetection(this.parser, encodedXML, "application/xml; charset=utf-8", StandardCharsets.UTF_8.name(),
testCharsetDetection(parser, encodedXML, "application/xml; charset=utf-8", StandardCharsets.UTF_8.name(),
"Maßkrügen");
/*
@ -146,7 +146,7 @@ public class GenericXMLParserTest {
* declaration
*/
encodedXML = ("<?xml version=\"1.0\"?>" + UMLAUT_TEXT_TAG).getBytes(StandardCharsets.UTF_8);
testCharsetDetection(this.parser, encodedXML, "application/xml; charset=utf-8", StandardCharsets.UTF_8.name(),
testCharsetDetection(parser, encodedXML, "application/xml; charset=utf-8", StandardCharsets.UTF_8.name(),
"Maßkrügen");
}
@ -169,7 +169,7 @@ public class GenericXMLParserTest {
*/
byte[] encodedXML = ("<?xml version=\"1.0\" encoding=\"utf-16\"?>" + UMLAUT_TEXT_TAG)
.getBytes(StandardCharsets.UTF_16);
testCharsetDetection(this.parser, encodedXML, "application/xml; charset=utf-16", StandardCharsets.UTF_16.name(),
testCharsetDetection(parser, encodedXML, "application/xml; charset=utf-16", StandardCharsets.UTF_16.name(),
"Maßkrügen");
/*
@ -177,7 +177,7 @@ public class GenericXMLParserTest {
* XML declaration having only BOM (Byte Order Mark)
*/
encodedXML = ("<?xml version=\"1.0\"?>" + UMLAUT_TEXT_TAG).getBytes(StandardCharsets.UTF_16);
testCharsetDetection(this.parser, encodedXML, "application/xml; charset=utf-16",
testCharsetDetection(parser, encodedXML, "application/xml; charset=utf-16",
StandardCharsets.UTF_16BE.name(), "Maßkrügen");
/*
@ -186,14 +186,14 @@ public class GenericXMLParserTest {
*/
encodedXML = ("<?xml version=\"1.0\" encoding=\"utf-16\"?>" + UMLAUT_TEXT_TAG)
.getBytes(StandardCharsets.UTF_16);
testCharsetDetection(this.parser, encodedXML, "application/xml", StandardCharsets.UTF_16.name(), "Maßkrügen");
testCharsetDetection(parser, encodedXML, "application/xml", StandardCharsets.UTF_16.name(), "Maßkrügen");
/*
* Charset is omitted in both Content-Type HTTP header and XML
* declaration with BOM (Byte Order Mark)
*/
encodedXML = ("<?xml version=\"1.0\"?>" + UMLAUT_TEXT_TAG).getBytes(StandardCharsets.UTF_16);
testCharsetDetection(this.parser, encodedXML, "application/xml", StandardCharsets.UTF_16BE.name(), "Maßkrügen");
testCharsetDetection(parser, encodedXML, "application/xml", StandardCharsets.UTF_16BE.name(), "Maßkrügen");
}
/**
@ -213,7 +213,7 @@ public class GenericXMLParserTest {
*/
byte[] encodedXML = ("<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>" + UMLAUT_TEXT_TAG)
.getBytes(StandardCharsets.ISO_8859_1);
testCharsetDetection(this.parser, encodedXML, "application/xml", StandardCharsets.ISO_8859_1.name(),
testCharsetDetection(parser, encodedXML, "application/xml", StandardCharsets.ISO_8859_1.name(),
"Maßkrügen");
}
@ -242,7 +242,7 @@ public class GenericXMLParserTest {
encodedXML = ("<?xml version=\"1.0\"?>"
+ "<text>In M&#x000FC;nchen steht ein Hofbr&#x000E4;uhaus, dort gibt es Bier in Ma&#x000DF;kr&#x000FC;gen</text>")
.getBytes(StandardCharsets.US_ASCII);
testCharsetDetection(this.parser, encodedXML, "application/xml", StandardCharsets.UTF_8.name(), "Maßkrügen");
testCharsetDetection(parser, encodedXML, "application/xml", StandardCharsets.UTF_8.name(), "Maßkrügen");
}
/**
@ -262,7 +262,7 @@ public class GenericXMLParserTest {
*/
byte[] encodedXML = ("<?xml version='1.0' encoding='utf-16be'?>" + UMLAUT_TEXT_TAG)
.getBytes(StandardCharsets.UTF_16BE);
testCharsetDetection(this.parser, encodedXML, "application/xml; charset=utf-16be",
testCharsetDetection(parser, encodedXML, "application/xml; charset=utf-16be",
StandardCharsets.UTF_16BE.name(), "Maßkrügen");
}
@ -288,7 +288,7 @@ public class GenericXMLParserTest {
String charsetFromHttpHeader = HeaderFramework.getCharacterEncoding(contentTypeHeader);
DigestURL location = new DigestURL("http://localhost/testfile.xml");
try {
Document[] documents = this.parser.parse(location, contentTypeHeader, charsetFromHttpHeader,
Document[] documents = parser.parse(location, contentTypeHeader, charsetFromHttpHeader,
new VocabularyScraper(), 0, inStream);
assertEquals(1, documents.length);
Collection<AnchorURL> detectedAnchors = documents[0].getAnchors();
@ -324,7 +324,7 @@ public class GenericXMLParserTest {
String charsetFromHttpHeader = HeaderFramework.getCharacterEncoding(contentTypeHeader);
DigestURL location = new DigestURL("http://localhost/testfile.xml");
try {
Document[] documents = this.parser.parse(location, contentTypeHeader, charsetFromHttpHeader,
Document[] documents = parser.parse(location, contentTypeHeader, charsetFromHttpHeader,
new VocabularyScraper(), 0, inStream);
assertEquals(1, documents.length);
Collection<AnchorURL> detectedAnchors = documents[0].getAnchors();
@ -351,7 +351,7 @@ public class GenericXMLParserTest {
String charsetFromHttpHeader = HeaderFramework.getCharacterEncoding(contentTypeHeader);
DigestURL location = new DigestURL("http://localhost/testfile.xml");
try {
Document[] documents = this.parser.parse(location, contentTypeHeader, charsetFromHttpHeader,
Document[] documents = parser.parse(location, contentTypeHeader, charsetFromHttpHeader,
new VocabularyScraper(), 0, inStream);
assertEquals(1, documents.length);
assertEquals("Node content1 Node content2", documents[0].getTextString());
@ -376,7 +376,7 @@ public class GenericXMLParserTest {
+ "Home page : http://yacy.net - International Forum : "
+ "https://searchlab.eu "
+ "and this is a mention to a relative URL : /document.html</p>"
+ "<p>Here are YaCy<a href=\"http://mantis.tokeek.de\">bug tracker</a> and <a href=\"http://www.yacy-websearch.net/wiki/\">Wiki</a>."
+ "<p>Here are YaCy<a href=\"http://mantis.tokeek.de\">bug tracker</a> and <a href=\"https://wiki.yacy.net/index.php/\">Wiki</a>."
+ "And this is a relative link to another <a href=\"/document2.html\">sub document</a></p>"
+ "</body>" + "</html>";
@ -386,7 +386,7 @@ public class GenericXMLParserTest {
String charsetFromHttpHeader = HeaderFramework.getCharacterEncoding(contentTypeHeader);
DigestURL location = new DigestURL("http://localhost/testfile.xml");
try {
Document[] documents = this.parser.parseWithLimits(location, contentTypeHeader, charsetFromHttpHeader, new VocabularyScraper(), 0, inStream, Integer.MAX_VALUE, Long.MAX_VALUE);
Document[] documents = parser.parseWithLimits(location, contentTypeHeader, charsetFromHttpHeader, new VocabularyScraper(), 0, inStream, Integer.MAX_VALUE, Long.MAX_VALUE);
assertEquals(1, documents.length);
assertFalse(documents[0].isPartiallyParsed());
@ -399,7 +399,7 @@ public class GenericXMLParserTest {
assertTrue(detectedAnchors.contains(new AnchorURL("http://yacy.net")));
assertTrue(detectedAnchors.contains(new AnchorURL("https://searchlab.eu")));
assertTrue(detectedAnchors.contains(new AnchorURL("http://mantis.tokeek.de")));
assertTrue(detectedAnchors.contains(new AnchorURL("http://www.yacy-websearch.net/wiki/")));
assertTrue(detectedAnchors.contains(new AnchorURL("https://wiki.yacy.net/index.php/")));
} finally {
inStream.close();
}
@ -407,7 +407,7 @@ public class GenericXMLParserTest {
/* Links limit exceeded */
inStream = new ByteArrayInputStream(xhtml.getBytes(StandardCharsets.UTF_8.name()));
try {
Document[] documents = this.parser.parseWithLimits(location, contentTypeHeader, charsetFromHttpHeader,
Document[] documents = parser.parseWithLimits(location, contentTypeHeader, charsetFromHttpHeader,
new VocabularyScraper(), 0, inStream, 2, Long.MAX_VALUE);
assertEquals(1, documents.length);
assertTrue(documents[0].isPartiallyParsed());
@ -443,12 +443,12 @@ public class GenericXMLParserTest {
}
int firstBytes = xhtmlBuilder.toString().getBytes(StandardCharsets.UTF_8.name()).length;
xhtmlBuilder.append("<p>Here are YaCy<a href=\"http://mantis.tokeek.de\">bug tracker</a> and <a href=\"http://www.yacy-websearch.net/wiki/\">Wiki</a>.")
xhtmlBuilder.append("<p>Here are YaCy<a href=\"http://mantis.tokeek.de\">bug tracker</a> and <a href=\"https://wiki.yacy.net/index.php/\">Wiki</a>.")
.append("And this is a relative link to another <a href=\"/document2.html\">sub document</a></p>")
.append("</body></html>");
inStream = new ByteArrayInputStream(xhtmlBuilder.toString().getBytes(StandardCharsets.UTF_8.name()));
try {
Document[] documents = this.parser.parseWithLimits(location, contentTypeHeader, charsetFromHttpHeader, new VocabularyScraper(), 0, inStream, Integer.MAX_VALUE, firstBytes);
Document[] documents = parser.parseWithLimits(location, contentTypeHeader, charsetFromHttpHeader, new VocabularyScraper(), 0, inStream, Integer.MAX_VALUE, firstBytes);
assertEquals(1, documents.length);
assertTrue(documents[0].isPartiallyParsed());

Write Preview
Loading…
Cancel
Save