a37d067692
refactoring
11 years ago
95780eed32
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
67beef657f
strong redesign of html parser: object recursion is now made using a
...
stack on html tag objects, not using a recursive parse-again method
which may cause bad performance and huge memory allocation. The new
method also produced better parsed image objects with exact anchor text
references.
11 years ago
6bd8c6f195
fix for wrong status codes of error pages
11 years ago
9e503b3376
also delete the robots.txt file from the cache when a new crawl is
...
started
11 years ago
67501c9dda
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
1c21b3256d
fix for robots.txt handling: delete old entry before starting a new
...
crawl.
11 years ago
c250fac9f4
linkstructure refactoring to get more options for clickdepth analysis
11 years ago
8068e68474
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
bd886054cb
new structure and enhancements for link graph computation:
...
- added order option to solr queries to be able to retrieve document
lists in specific order, here: link length
- added HyperlinkEdge class which manages the link structure
- integrated the HyperlinkEdge class into clickdepth computation
- extended the linkstructure.json servlet to show also the clickdepth
and other statistic information
11 years ago
f326a67561
fix: typo in default charset in metadata2solr
...
update pom and NB build to Solr 4.7.1 libs
11 years ago
df138084c0
do solr optimization independently from memory and load constraints:
...
- not doing an optimization will likely cause a too many files exception
- without optimization performance will be even worse which would
prevent optimization in the future as well (prevent a deadlock
situation)
11 years ago
ebd44a7080
replaced solr 4.6.1 with solr 4.7.1 and added index migration to
...
lucene_47
11 years ago
0f3fbae438
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
1a6e0354db
update commons-compress.jar to 1.8
11 years ago
68417a05c5
different algorithm to test checkalive as it depends less on the
...
existence of wget (or curl) on the OS.
11 years ago
6b0e62ec59
Emergency bugfix for killYACY.sh as the file yacy00.log does not exist
...
in case that a too many open files error exist. In such a case, the file
yacy00.log does not exist but only the file yacy00.log.lck.
In the long term a different solution should be addressed.
11 years ago
ee92d748b5
test using compound file format, see UseCompoundFile in
...
https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig
This appears to be necessary as many times a
java.io.FileNotFoundException: (Too many open files) appears.
See also: https://issues.apache.org/jira/browse/SOLR-4 and desperate
users at
http://stackoverflow.com/questions/3828343/too-many-open-file-exception-while-indexin-using-solr
We cannot force users to do a "ulimit -n 1000000", so this action seems
to be required.
11 years ago
d2055f3d4b
next development version 1.71
...
It's nowhere explained or declared, but since some time we follow the
schema that uneven version numbers are used for development versions and
even numbers for release versions. That concept may change sometime but
this is used at this time to distinguish development from main.
11 years ago
d1b5180dd9
upd version in pom
11 years ago
d051d2d85f
release 1.7
11 years ago
0a95fd27f3
update of seed list
11 years ago
6e84770fd9
Merge branch 'master' of gitorious.org:yacy/icewindxs-rc1
...
Conflicts:
locales/ru.lng
11 years ago
f509cd4aab
Update russian translation
11 years ago
f296a529d5
update to german locale
11 years ago
734778c0c8
fixed a time-out problem in the default servlet which is also a logging
...
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).
11 years ago
466d90ad42
fixed a problem with resource observer; probably coming from uncatched
...
exceptions within the apache library which appear only in concurrency
environments.
11 years ago
c8d4a63604
eliminating the word 'Facet' from the interface because it is ugly. If
...
people do not know what search navigation is, then they also do not know
what a 'facet' is.
11 years ago
e8ddd415a8
enhanced the new link structure graph
11 years ago
926d28dd3f
fixed a bug which prevented crawl starts after a network switch
11 years ago
8443255e18
better link structure limit calibration
11 years ago
7f5733638b
fix for linkstructure computation: now also detecting dead links
11 years ago
3ce8eff21b
another fix for inbound/outbound detection
11 years ago
d4b5c457e4
NPE fix
11 years ago
36a66b0704
fix for parsing of numeric value in case that boolean values are given
11 years ago
41730c8048
better logging in template engine: shows filename of servlets where
...
errors in templates occur
11 years ago
3c1274057d
fixed thread dump in case of wrong seeds
11 years ago
18f9c40302
moved Edge class out of linkstructure servlet as this does not work on
...
non-eclipse driven environments (all non-dev cases)
11 years ago
de95e5e524
reduced search activity corona strength in network image
11 years ago
da413af664
move baseurl after parsing orig source in urlproxyservlet
...
to calculate absolute href links for rewrite from unmodified source.
11 years ago
af6ad20728
fix: remove obsolete ref to yacy.home
...
(use Switchboard instead)
11 years ago
a6bb9be97e
- added d3.js for visualizations using embedded svg
...
- added a servlet api/linkstructure.json which generates a link graph
information in json
- added a javascript link graph renderer hypertree.js using d3 and the
new servlet linkstructure.json
- embedded the new link graph in the crawler monitor and the host
browser
11 years ago
74ab094587
fix for solr query size; too many documents had been retrieved in case
...
that less than _pagesize_ had been requested.
11 years ago
c64c10ef00
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
48fbfa60c1
bugfix to inbound/outbound identification
11 years ago
227c42bc96
eleminate obsolete URIMetaDataRow class
...
by joining it with/into URIMetaDataNode.
11 years ago
cca851a417
introduced new solr field crawldepth_i which records the crawl depth of
...
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.
11 years ago
d321b0314e
added missing servlet html
11 years ago
b1ba764d81
fix for first start options and added german translation for popup texts
11 years ago
043d274af5
fixed crawl start path for cloned crawls
11 years ago
Michael Peter Christen