YaCy '#[clientname]#': MediaWiki Dump Import

MediaWiki Dump Import

#(prevReport)#::Check the last import report.#(/prevReport)#

#(status)#

No import thread is running, you can start a new thread here

Error : dump URL is malformed.

Error : file not found "#[sourceFile]#"

Error : can not read file "#[sourceFile]#"

Error : you selected a directory ("#[sourceFile]#")

Error : dump file ("#[sourceFile]#") was not modified since last import (#[lastImportDate]#).

Dumps can be stored in the local file system or on a remote server in XML format and may be compressed in gz or bz2.

Import only when modified since last import

When checked, the dump file is imported only if its last modified date is unknown or is after the last import execution date on this same file (see recorded API calls with the "dump" type).

When the import is started, the following happens:

The dump is extracted on the fly and wiki entries are translated into Dublin Core data format. The output looks like this:

    <?xml version="1.0" encoding="UTF-8"?>
<surrogates xmlns:dc="http://purl.org/dc/elements/1.1/">
  <record>
    <dc:Title><![CDATA[Alan Smithee]]></dc:Title>
    <dc:Identifier>http://de.wikipedia.org/wiki/Alan%20Smithee</dc:Identifier>
    <dc:Description><![CDATA[Der als Filmregisseur oft genannte Alan Smithee ist ein Anagramm]]></dc:Description>
    <dc:Language>de</dc:Language>
    <dc:Date>2009-05-07T06:03:48Z</dc:Date>
  </record>
  <record>
    ...
  </record> 
</surrogates>

Each 10000 wiki records are combined in one output file which is written to /DATA/SURROGATES/in into a temporary file.

When each of the generated output file is finished, it is renamed to a .xml file

Each time a xml surrogate file appears in /DATA/SURROGATES/in, the YaCy indexer fetches the file and indexes the record entries.

When a surrogate file is finished with indexing, it is moved to /DATA/SURROGATES/out

You can recycle processed surrogate files by moving them from /DATA/SURROGATES/out to /DATA/SURROGATES/in

#(status)#::

Error encountered : #[message]#