When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote
file now is directly streamed and processed, allowing import of several
GB dumps even with a low memory remote peer, and without need to
manually download the dump file first.
- ensure use of HTTP POST method : HTTP GET should only be used for
information retrieval and not to perform server side effect operations
(see HTTP standard https://tools.ietf.org/html/rfc7231#section-4.2.1)
- a transaction token is now required for these administrative form
submissions to ensure the request can not be included in an external
site and performed silently/by mistake by the user browser
Because curl and wget do not let use a hashed password as parameter,
YaCy shell scripts which require authentication are now interactive by
default when HTTP Digest is the only available authentication method.
Batch mode can still be available trough the use of an environment
variable : YACY_ADMIN_PASSWORD.
Other improvements :
- added backward compatibility for Basic Authentication
- fixed curl/wget presence detection
- do not return with exit code 0 when an API call failed, and print an
error message when the case occurs
- documented available authentication options for API calls
hash even if localhost access is disabled. This is urgently needed for
the apicall.sh script since that is used for high-availability set-up
(checkalive and indexdump for index mirroring)
table. This may be necessary if the api is called by remote command and
the recordings are not used. Then they can be deleted frequently by
calling this clear command using a cron job
the index dump. This is the reverse of indexdump.sh which takes the
output of indexdump.sh as input to restore an index.
Now it should be possible to transfer a complete YaCy Solr index from
one peer yacy1 to another peer yacy2 with the following command:
yacy2/bin/indexrestore.sh ´yacy1/bin/indexdump.sh´
dump commands
- adjusted the apicall.sh script to get the downloaded text as output to
stdout which is necessary to parse the content out of it
- added indexdump.sh script which creates a solr dump and prints out the
storage path for the index dump
- added synchronization to the Fulltext class to prevent that data is
stored to a non-existing solr index while this index is disabled during
the storage of the dump
- added cleaning of blacklist cache if cache is modified in interface
- extended cache saving to all cache types
- moved cache location to DATA/LISTS
- fixed static file path which was relative to the application path but
should be relative to data path - which is different in debian and mac
implementations
- extended metadata information in index with geolocalisation
- added display of location in yacydoc and ViewFile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7629 6c8d7289-2bf4-0310-a012-ef5d649a1542
if you have a list of oai servers you can import them all using the linux command:
bin/importOAIList.sh <name-of-oai-list-file>
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7141 6c8d7289-2bf4-0310-a012-ef5d649a1542
- move the header framework classes to cora
- move the ARC caching classes to cora
- refactoring of code to call these classes from cora
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7068 6c8d7289-2bf4-0310-a012-ef5d649a1542
- refactoring of stopYACY.sh by introduction of /bin/apicall which is able to call any api file with attached authorization
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6691 6c8d7289-2bf4-0310-a012-ef5d649a1542
luccioman