67edfd991c
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
d9173ba7ed
added more solr fields to integrate values from URIMetadataRow. All
...
writings to the Metadata-DB are now also done to solr. This includes
metadata transfer during search and rwi transfer.
The new/added solr fields are:
## time when resource was loaded
load_date_dt
## date until resource shall be considered as fresh
fresh_date_dt
## id of the host, a 6-byte hash that is part of the document id
host_id_s
## ids of referrer to this document
referrer_id_ss
## the md5 of the raw source
md5_s
## the name of the publisher of the document
publisher_t
## the language used in the document; starts with primary language
language_ss
## an external ranking value
ranking_i
## the size of the raw source
size_i
## number of links to audio resources
audiolinkscount_i
## number of links to video resources
videolinkscount_i
## number of links to application resources
applinkscount_i
13 years ago
24d9db1613
snippet retrieval loading processes may use a smaller minimum load time
...
value than crawling processes. This speeds up the search result
preparation dramatically.
13 years ago
1687737771
Abstraction of HandleMap and HandleSet
13 years ago
482afed07c
reduced logging overhead (a bit)
13 years ago
bbfa497a3c
replaced more size() > 0 by !isEmpty()
13 years ago
0cbda0b2b8
- replaced all length() == 0 and size() == 0 with isEmpty()
...
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
13 years ago
801972fe6f
fix for url camel case parser and sentence reader
13 years ago
fbc1a2030d
fix for sitemap importer: can now also import very large sitemaps within
...
small memory configurations
13 years ago
92731e5287
fix for sevenzip parser
13 years ago
8efc1c1078
- fixed a memory leak (or bad usage) during parsing/snippet fetch
...
- more logging for errors
13 years ago
b1e7c11fba
fix for pattern matcher in html parser
13 years ago
b0c408788b
made class methods static where possible
13 years ago
7c1ba99755
removed more unused method parameters
13 years ago
0301aba1e9
removed unused method parameters
13 years ago
d3964253ae
- added @SuppressWarnings to unused servlet method parameters
...
- removed unnecessary casts
- removed unnecessary throw statements
13 years ago
ea10766bfd
cleaned unnecessary nested code
13 years ago
fc0f9543fe
More SentenceReader cleanup
13 years ago
586bb0eb6a
Simplified SentenceReader (no more Reader inside..)
13 years ago
7f851d62a7
replaced HashARC with SizeLimited Objects which are less costly
13 years ago
78fc3cf8f8
refactoring and new usage of SentenceReader: this class appeared as one
...
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
13 years ago
bb8dcb4911
automatically adopt size of word cache to available memory
13 years ago
ad09b786bf
clean up parser data
13 years ago
276a66a793
Adding a limit of 1000 links that a parser shall store during indexing.
...
A limit was necessary because some web pages have such huge numbers of
links that it can easily cause a OOM just by the number of links.
The quesion if the number of 1000 links is sufficient or too weak must
be answered with the result of testing this feature.
13 years ago
de903a53a0
parser refactoring & hacks
13 years ago
1825f165b8
better integration of blacklist according to use case
13 years ago
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
13 years ago
0c345d1559
giving threads name so its easier to see whats happening during
...
debugging and within a thread dump
13 years ago
508a81b86c
added solr field 'refresh_s' which stores the refresh url contained in
...
the meta-refresh html header field.
13 years ago
f3167def64
do not fill the keywords with title content if keywords do not exist.
13 years ago
77f795756c
fixing redirects and status codes: storing of status code in
...
ResponseHeader to make it available for late evaluations, like storage
in solr.
13 years ago
dbdd697f4d
moved RDFaParser.xsl configuration file to defaults
13 years ago
786be7d175
better integration of RDFaParser
13 years ago
de3ef8ad73
removed unimportant warnings
13 years ago
24bbe359ca
integrate also geonames library files for less cities. these are more
...
useful for tagging since less normal words are false-identified as
location
13 years ago
223a5440ab
preventing that an empty pnd is inserted into the vocabularies
13 years ago
963f92ed9a
- merged files
...
- changed behaviour of delete button in vocabulary edit
- fixed size numbe in vocabulary listing
13 years ago
dd88d0ace2
more logging
13 years ago
94d54e2d91
added recognition of multi-word terms in vocabulary matching
...
this makes the PND usable: it is now possible to recognize persons and
navigate with a 'Persons' facet.
13 years ago
64c0268b2b
show triplestore metadata in yacydoc and viewfile
13 years ago
c2f0d16d2c
fixed vocabulary initialization
13 years ago
df3531f8d5
added the generation of virtual vocabularies using the pnd
13 years ago
a0f1decd82
- added loading of the dbpedia pnd triplestore in the dictionary loader
...
- renamed the dictionary loader to knowledge loader
- some refactoring in the library provider method names
13 years ago
16d8f33795
added objectlink generation to vocabulary generation and editor
13 years ago
d45718251e
refactoring (Localization -> Location)
13 years ago
b8b3c87ba7
- renamed localization to location (that was confusing)
...
- renamed 'Locale' navigator to 'Location'
- produce Location navigation only if geolocation libraries are loaded
13 years ago
e89747bb67
- added automated generation of vocabularies from url stubs
...
- added clear of all terms for vocabularies
- added deletion of vocabularies
13 years ago
79464189a4
The 'Locale' vocabulary, which is generated by geo data, has now the
...
objectspace "http://dbpedia.org/resource/ "
13 years ago
61bb52d55c
- using http://purl.org/dc/terms/references to refer from an
...
auto-annotated document to a 'pseudo-linked' document which has an url
created with an object-prefix as defined in the vocabulary file
13 years ago
50c576599b
allow multiple parser options instead of printing an error
13 years ago
orbiter