You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
137 lines
7.5 KiB
137 lines
7.5 KiB
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
|
|
<!-- This page is only XHTML 1.0 Transitional because target is being used in a links -->
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
#(reload)#::<meta http-equiv="REFRESH" content="5; url=/IndexExport_p.html">#(/reload)#
|
|
<head>
|
|
<title>YaCy '#[clientname]#': URL Database Administration</title>
|
|
#%env/templates/metas.template%#
|
|
</head>
|
|
<body id="IndexControl">
|
|
#%env/templates/header.template%#
|
|
#%env/templates/submenuIndexImport.template%#
|
|
|
|
<h2>Index Export</h2>
|
|
<p>The local index currently contains #[ucount]# documents, only #[ucount200]# exportable with status code 200 - the remaining are error documents.</p>
|
|
|
|
#(lurlexport)#::
|
|
<form action="IndexExport_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
|
|
<fieldset><legend>Loaded URL Export</legend>
|
|
<dl>
|
|
<dt class="TableCellDark">Export Path</dt>
|
|
<dd><input type="text" name="exportfilepath" value="#[exportfilepath]#" size="120" maxlength="250" />
|
|
</dd>
|
|
<dt class="TableCellDark">URL Filter</dt>
|
|
<dd><input type="text" name="exportfilter" value=".*.*" size="20" maxlength="250" /> .*.* (default) is a catch-all; format: java regex
|
|
</dd>
|
|
<dt class="TableCellDark">query</dt>
|
|
<dd><input type="text" name="exportquery" value="*:*" size="20" maxlength="250" /> *:* (default) is a catch-all; format: <field-name>:<solr-pattern>
|
|
</dd>
|
|
<dt class="TableCellDark">maximum age (seconds)</dt>
|
|
<dd><input type="text" name="exportmaxseconds" value="-1" size="20" maxlength="250" /> -1 = unlimited -> no document is too old
|
|
</dd>
|
|
<dt class="TableCellDark">maximum number of records per chunk</dt>
|
|
<dd><input type="text" name="maxchunksize" value="-1" size="20" maxlength="250" /> if exceeded: several chunks are stored; -1 = unlimited (makes only one chunk)
|
|
</dd>
|
|
<dt class="TableCellDark">Export Size</dt>
|
|
<dd>
|
|
full size, all fields:<input type="radio" name="minified" value="no" checked="checked">
|
|
minified; only fields sku, date, title, description, text_t<input type="radio" name="minified" value="yes" >
|
|
</dd>
|
|
<dt class="TableCellDark">Export Format</dt>
|
|
<dd>
|
|
<dl>
|
|
<dt>Full Data Records:</dt>
|
|
<dd><input type="radio" name="format" value="full-elasticsearch" checked="checked" />
|
|
JSON (Rich and full-text Elasticsearch data, one document per line in one flat JSON file,
|
|
can be bulk-imported to elasticsearch. Here is an example for opensearch, using docker:<br />
|
|
Start docker container of opensearch:<br />
|
|
<code>docker run --name opensearch -p 9200:9200 -d -e OPENSEARCH_JAVA_OPTS="-Xms2G -Xmx2G" -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true -v $(pwd)/opensearch_data:/usr/share/opensearch/data opensearchproject/opensearch:latest</code><br />
|
|
Unblock index creation:<br />
|
|
<code>curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
|
|
{
|
|
"persistent": {
|
|
"cluster.blocks.create_index": null
|
|
}
|
|
}'</code><br />
|
|
Create the search index:<br />
|
|
<code>curl -X PUT "http://localhost:9200/collection1/yacy"</code><br />
|
|
Bulk-upload the index file:<br />
|
|
<code>curl -XPOST "http://localhost:9200/collection1/yacy/_bulk?filter_path=took,errors" -H "Content-Type: application/x-ndjson" --data-binary @yacy_dump_XXX.flatjson</code><br />
|
|
Make a search, get 10 results, search in fields text_t, title, description with boosts:<br />
|
|
<code>curl -X POST "http://localhost:9200/collection1/yacy/_search" -H 'Content-Type: application/json' -d'
|
|
{"size": 10, "query": {"multi_match": {
|
|
"query": "one two three",
|
|
"fields": ["text_t", "title^10", "description^3"], "fuzziness": "AUTO"
|
|
}}}'</code><br />
|
|
<input type="radio" name="format" value="full-solr" />
|
|
XML (Rich and full-text Solr data, one document per line in one large xml file,
|
|
can be processed with shell tools, can be imported with DATA/SURROGATE/in/)
|
|
<br />
|
|
<input type="radio" name="format" value="full-rss" />
|
|
XML (RSS)
|
|
</dd>
|
|
<dt>Full URL List:</dt>
|
|
<dd><input type="radio" name="format" value="url-text" /> Plain Text List (URLs only)<br />
|
|
<input type="radio" name="format" value="url-html" /> HTML (URLs with title)</dd>
|
|
<dt>Only Domain:</dt>
|
|
<dd><input type="radio" name="format" value="dom-text" /> Plain Text List (domains only)<br />
|
|
<input type="radio" name="format" value="dom-html" /> HTML (domains as URLs, no title)</dd>
|
|
<dt>Only Text:</dt>
|
|
<dd><input type="radio" name="format" value="text-text" /> Fulltext of Search Index Text</dd>
|
|
</dl>
|
|
</dd>
|
|
<dt> </dt>
|
|
<dd><input type="submit" name="lurlexport" value="Export" class="btn btn-primary" style="width:240px;"/>
|
|
</dd>
|
|
</dl>
|
|
</fieldset>
|
|
</form>::
|
|
<div class="alert alert-info" style="text-decoration:blink">Export to file #[exportfile]# is running .. #[urlcount]# Documents so far</div>::
|
|
#(/lurlexport)#
|
|
|
|
#(lurlexportfinished)#::
|
|
<div class="alert alert-success">Finished export of #[urlcount]# Documents to file <a href="file://#[exportfile]#" target="_">#[exportfile]#</a><br/>
|
|
<em>Import this file by moving it to DATA/SURROGATES/in</em></div>::
|
|
#(/lurlexportfinished)#
|
|
|
|
#(lurlexporterror)#::
|
|
<div class="alert alert-warning">Export to file #[exportfile]# failed: #[exportfailmsg]#</div>::
|
|
#(/lurlexporterror)#
|
|
|
|
#(dumprestore)#::
|
|
<form action="IndexExport_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
|
|
<fieldset><legend>Dump and Restore of Solr Index</legend>
|
|
#(dumpRestoreEnabled)#<div class="alert alert-info">This feature is available only when a local embedded Solr is active.</div>::#(/dumpRestoreEnabled)#
|
|
<dl>
|
|
<dt> </dt>
|
|
<dd><input type="submit" name="indexdump" value="Create Dump" class="btn btn-primary" style="width:240px;" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
|
|
</dd>
|
|
</dl>
|
|
<dl>
|
|
<dt class="TableCellDark">Dump File</dt>
|
|
<dd><input type="text" name="dumpfile" value="#[dumpfile]#" size="80" maxlength="250" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
|
|
</dd>
|
|
<dt> </dt>
|
|
<dd><input type="submit" name="indexrestore" value="Restore Dump" class="btn btn-primary" style="width:240px;" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
|
|
</dd>
|
|
</dl>
|
|
</fieldset>
|
|
</form>::
|
|
#(/dumprestore)#
|
|
|
|
#(indexdump)#::
|
|
<div class="alert alert-success" role="alert">Stored a solr dump to file #[dumpfile]#</div>::
|
|
<div class="alert alert-danger" role="alert">Could not create the Solr dump : no embedded Solr is available.</div>::
|
|
<div class="alert alert-danger" role="alert">An error occurred while trying to create the Solr dump.</div>
|
|
#(/indexdump)#
|
|
|
|
#(indexRestore)#::
|
|
<div class="alert alert-success" role="alert">Successfully restored Solr index from dump file!</div>::
|
|
<div class="alert alert-danger" role="alert">Could not restore the Solr dump : no embedded Solr is available.</div>::
|
|
<div class="alert alert-danger" role="alert">An error occurred while trying to restore the Solr dump.</div>
|
|
#(/indexRestore)#
|
|
|
|
#%env/templates/footer.template%#
|
|
</body>
|
|
</html>
|