not-flushed Solr cache is now handled in this way:
- it is smaller by default
- an Solr-internal process is started to flush the cache periodically
(this does NOT clean the cache, just removes old objects)
- a Solr-external process (the standard YaCy cleanup-process) now has
direct access to the solr internal cache and flushes them completely.
The time frame for such a flush is defined by the cleanup-process
frequency, by default 10 minutes.
- see numerous idx entries with content_type image without url_file_ext_s (for various reason) which should be included in result
- try it yourself with following sample query
/solr/select?q=content_type:image/* AND -url_file_ext_s:[* TO *]&defType=edismax&fl=sku,url_file_ext_s,content_type
adresses also possible url without or deviating extension.
- use current YaCy version number
- make use of libbuild\GitRevMavenTask (maven-plugin-gitrevisionnumber)
- make yacyBuildProperties.java available for source filtering by Maven-plugin (copy to libbuild\java-templates)
- update assembly definition to include lib\yacycore.jar without version number (needed this way by startupscript)
- it injects properties which can be used in pom via ${DSTAMP} ${releaseNr} if added as plugin via
<plugin>
<groupId>net.yacy</groupId>
<artifactId>maven-plugin-gitrevisionnumber</artifactId>
<version>1.0</version>
<executions><execution>
<phase>initialize</phase>
<goals><goal>create</goal></goals>
</execution></executions>
</plugin>
the embedded Solr (the default). This was obtained by cirumventing solrj
search encapsulation and the implementation of direct index access
methods to Solr.
The effect will not only be seen during search, but this has also a
strong effect on suggestions (much more) and less CPU power usage during
index distribution (which needs many search requests)
- based on Jetty ProxyServlet
- at this time use existing HTTPD ProxyHandler for url rewrite
- add jetty-client jar (dependency in Jetty ProxyServlet)
reuse ProxyHandler.convertHeaderFromJetty in YaCyDefaultServlet
- transformed log lines to String before they are stored because the
storage space is about 1:250 (45kb for one line before transformation,
180 bytes afterwards)
- this saves up to 10MB RAM so we can increase the number of lines to
1000 again.
which had a problem because of badly used concurrency.
This fix also caused a redesign of the whole host deletion process.
This should fix bug http://bugs.yacy.net/view.php?id=250
the right content domain (i.e. identifying that it is an image, text
etc.) because it used the file extension and not an existing mime type
assignment.
- fixed the new setting that images shall be loaded for a better image
search.
- both fixes together makes it now possible to crawl
commons.wikimedia.org which makes use of 'funny' document names (i.e.
ending with .jpg while the document is html)
Michael Peter Christen