b764de424a
code cleanup
13 years ago
1168d09de8
more refactoring - integrated the code of SnippetProcess into
...
SearchEvent
13 years ago
6629e37685
tried to clean up the search process mess
13 years ago
c5f67a5d6d
fixed a problem with local search from solr results: now all results
...
from solr are shown (again)
13 years ago
276dd6452b
removed warnings
13 years ago
ce0e5b1e17
- more refactoring / private methods
...
- fix for usage of custom solr field names
13 years ago
ccc3760a47
Refactoring and redesign of data architecture to make URIMetadataRow
...
superfluous. The target is to make a solr document as the core of YaCy
documents which would cause that many conversions can be removed. On the
way to this target the Equivalence of URIMetadataRow and URIMetadataNode
had to be removed to expose the usage of the old URIMetadataRow data
structure.
This refactoring already removes unneccessary conversions and should
make memory usage during indexing lower.
13 years ago
e5b3c172ff
removed hack which translated Solr documents to virtual RWI entries
...
which had been then mixed with remote RWIs. Now these Solr documents are
feeded into the result set as they appear during local and remote
search. That makes the search much faster.
13 years ago
5d16c23a1f
specified more URIMetadata as URIMetadataNode
13 years ago
43f3345c90
- removed dependencies from URIMetadataRow and made direct access to
...
URIMetadataNode which creates the opportunity to access Solr objects
directly and use their information richness
- lazy initialization of the URIMetadataNode object - should cause less
computation and memory usage during search.
- removed dead code
13 years ago
36c13ed15b
less solr prefetch
13 years ago
5f0ab25382
removed the option to prevent removal of & parts inside of the
...
MultiProtocolURI during normalform computation because that should
always be done and also be done during initialization of the
MultiProtocolURI Object. The new normalform method takes only one
argument which should be 'true' unless you know exactly what you are
doing.
13 years ago
584663ae8c
- redesign of solr query construction
...
- fix for solr boosts and location search
- fix for number of search results in local search
13 years ago
4fed4a86d8
another fix to location search
13 years ago
0f7a54452d
fix for location search query encoding
13 years ago
f8a3ab2d82
added the usage of synonyms to the GSA search interface
13 years ago
ca313e404f
- if a "/date" modifier is used, the solr remote query applies an
...
ordering by date (ascending)
- added also some 'anti-timetravel' protection (check if date is in the
future within any metadata date field)
13 years ago
5ac61591f3
better abstraction for solr query params
13 years ago
1533bfd63b
refactoring
13 years ago
e49359cc95
removed tenant query attribute since it is not used any more and is
...
replaced by the site-operator in the GSA interface. This operator can
also be simulated in the Solr interface using the collections_sxt field.
13 years ago
872f83ebe0
refactoring
13 years ago
fb9460f0a8
using the search filter to drill down search to file types.
...
A search like "mp3 filetype:mp3" will now maybe surprise you.
13 years ago
e57bf2ca39
simplified DHT classes
13 years ago
8219a445f3
refactoring
13 years ago
00c1c777fa
refactoring
13 years ago
563d584420
removed more dependencies in cora from kelondro
13 years ago
63762d8f89
removed kelondro dependencies from cora
13 years ago
4d29f59a27
removed warnings
13 years ago
31d4d38804
- extended the solr interface by a references-by-word-count method
...
- reduced danger that a non-existing RWI database causes NPEs
- added Solr queries to did-you-mean: this makes it possible that our
did-you-mean algorithm works together with only Solr and without RWIs
13 years ago
75d5e3475d
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
dc468dad01
add content control features for custom filter lists
13 years ago
316b5fe116
- added a solr type definition verifier
...
- fixed type definition found by the verifier
- added multivalue-string fields for solr with extension 'sxt'
- added multivalue-integer fields for solr with extension 'val'
- renamed some solr attributes from txt to sxt
- changed solr query line to an explicit AND/OR structure
- added a country code second level domain list to Domains class; with
parser
- added a host string parser to get domain class name, country-code
second-level domain and subdomain out of it
- removed old coordinate attributes
13 years ago
4521d63c92
added boosts to solr search queries
13 years ago
e8acd542b5
- added faceted drill-down for host and geolocation to solr queries
...
- added a new geolocation field to index schema, the old values are
migrated if possible
13 years ago
48a82bc705
log queries anonymous from gsa+solr requests
13 years ago
ab6ec4ec52
added snippet computation to solr/rss and gsa result writer
13 years ago
653645c1cf
corrected solr query syntax
13 years ago
a049761e0c
fixed double-check
13 years ago
f42a57cd7d
gsa format update
13 years ago
ff3eaa21b0
added remote search to solr on YaCy peers!
...
- when doing a remote search, node peers are selected for solr queries
- the solr query is done concurrently to the standard YaCy rwi search
- the solr search result is feeded into the same data structure that
prepares the rwi search result
- the same remote seach that is done to several outside peers is done to
the local solr index
- the search process works now also without any 'old' RWI data using
solr
13 years ago
a06123aec6
more abstraction and less parameter overhead for remote search
13 years ago
f00733186b
code simplifications
13 years ago
db0d438709
fix for http://bugs.yacy.net/view.php?id=206
13 years ago
404b0aab09
refactoring in remote search and stub for remote node peer selection
13 years ago
99ef57f103
reduced sleep times
13 years ago
0cab06c47c
refactoring
13 years ago
40c0856489
refactoring
13 years ago
06a78eecb7
code simplification
13 years ago
9bece5ac5f
enhanced snippet fetch - removed a bug that caused documents to be
...
parsed even if a solr text was available
13 years ago
18f989dfb1
- refactoring (load -> getMetadata)
...
- added getDocument to retrieve Solr documents which shall replace
getMetadata
13 years ago
395b78a0d8
using the solr search index to concurrently search within solr and the
...
rwis during local search requests.
13 years ago
6197caf698
added clear-text search words in query params
13 years ago
e5ef840f40
- renamed DoubleSolrConnector to MirrorSolrConnector and added a
...
hit/miss/document cache to the MirrorSolrConnector.
- more abstraction to SolrDocument in Connector interface
- bugfixes in Solr field reader
13 years ago
136fcb1ad9
refactoring
13 years ago
24d9db1613
snippet retrieval loading processes may use a smaller minimum load time
...
value than crawling processes. This speeds up the search result
preparation dramatically.
13 years ago
ef488a15f7
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
1687737771
Abstraction of HandleMap and HandleSet
13 years ago
76b037a20a
check content domain fix:
...
search image/media should not show pages containing image/media
search text should show all/text but image/media
13 years ago
3bcd9d622b
cleaned up classes and methods which are either superfluous at this time
...
or will be superfluous or subject of complete redesign after the
migration to solr. Removing these things now will make the transition to
solr more simple.
13 years ago
6f1ddb2519
Moved solr index-add method to the same method where the YaCy index is
...
written. Also done some code-cleanup.
13 years ago
76202f068e
extended abstraction of local and remote solr index using one front-end
...
for index administration and querying.
13 years ago
69e743d9e3
- more abstraction for the RWI index as preparation for solr integration
...
- added options in search index to switch parts of the index on or off
13 years ago
c00a3cf74d
less usage of generic logger to avoid logger generation overhead
13 years ago
0cbda0b2b8
- replaced all length() == 0 and size() == 0 with isEmpty()
...
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
13 years ago
c7afa8bc48
using SwitchboardConstants for solr attributes
13 years ago
62202e2d71
refactoring of query attribute variable names for better consistency
...
with (next) stored query words
13 years ago
0301aba1e9
removed unused method parameters
13 years ago
241dd8410a
removed snippet pattern filter - it was not used
13 years ago
ea10766bfd
cleaned unnecessary nested code
13 years ago
613b45f604
- better data structures in secondary search
...
- fixed a big memory leak in secondary search
13 years ago
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
13 years ago
0c345d1559
giving threads name so its easier to see whats happening during
...
debugging and within a thread dump
13 years ago
b9dfca4b0a
- fixed IndexFederated Servlet / a embedded Solr can now be selected
...
- added code stub for an embedded Solr but generation of Solr store is
still commented out (it works but is not yet ready for usage)
13 years ago
9264d8b4af
removed old navigation practice using subject tags in favor of
...
triplestore-tags
13 years ago
64c0268b2b
show triplestore metadata in yacydoc and viewfile
13 years ago
8b53771db2
changed behavior of navigation processing:
...
- vocabulary annotation is not done any more into the metadata of urldb
- vocabularies are written into the jena triplestore using a rdf
vocabulary
- vocabularies for rdf tripel must be updated; refactoring done
- with the new navigation tags in the triplestore a faster
pre-urldb-lookup is possible: navigation is processed now within the RWI
during pre-ranking retrieval
- added also a Owl vocabulary stub to add the plain-text url to the
triplestore using the owl:sameas predicate
13 years ago
5fc6524ca8
- moved triple store to net.yacy.cora.lod (should be generalized there
...
later
- added abstract add, delete, get methods in the triplestore
- added generation of triples after auto-annotation
- migrated all MultiProtocolURI objects to DigestURI in the parser since
the url hash is needed as subject value in the triples in the triple
store
13 years ago
edaa09b9b1
Rewrote all String blacklist types to enum 'BlacklistType', closes bug
...
#143
Conflicts:
htroot/Supporter.java
htroot/yacy/crawlReceipt.java
htroot/yacy/transferRWI.java
htroot/yacy/transferURL.java
source/de/anomic/crawler/CrawlStacker.java
source/de/anomic/data/ListManager.java
source/net/yacy/peers/Protocol.java
source/net/yacy/repository/Blacklist.java
source/net/yacy/repository/LoaderDispatcher.java
source/net/yacy/search/Switchboard.java
source/net/yacy/search/index/MetadataRepository.java
source/net/yacy/search/index/Segment.java
source/net/yacy/search/query/RWIProcess.java
source/net/yacy/search/snippet/MediaSnippet.java
13 years ago
65c5826d93
bugfix
...
Conflicts:
source/net/yacy/document/parser/augment/AugmentParser.java
13 years ago
701b9a28a0
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
htroot/PerformanceMemory_p.java
13 years ago
ab7107b34b
fixed RWIProcess queue limits: now discovering hidden results for mass
...
result retrieval
13 years ago
96e9d77270
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
13 years ago
00f2df1120
a variety of possible memory leak fixes
13 years ago
461a0ce052
removed warnings
13 years ago
407fdf6968
more bug fixes and performance hacks for search process
13 years ago
a1fe65b115
performance hacks
13 years ago
2fe207f813
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
5e562dcdb7
adopted vocabulary usage within anotation/naviagtion feature of search
...
to new SimpleVocabulary class
13 years ago
240045cf7c
fix for bad distance computation
13 years ago
e0d8643226
- performance hacks
...
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
13 years ago
9b4c699526
ehanced location search:
...
- search request are now made using a map boundary
- search results are only computed for the map boundary
- the number of results is adopted to the results in the visible range
- added a double-buffering for the search result markers
- added a search query option for the search results:
/radius/<lat>/<lon>/<radius>
13 years ago
834dc6b263
store more data from interface access
13 years ago
7c1feefb28
introduced a default 10 second time-out in rwi normalization time
...
uring search process to prevent endless deadlocks after a very long
running search
13 years ago
7bf421b9dd
- fixed image search page navigation
...
- removed some deadlocks and ConcurrentModificationExceptions during
DidYouMean collection
13 years ago
c6558cba08
more classification bugs
13 years ago
082831b9d6
search contentdom was checked in wrong way - fixed
13 years ago
f294f2e295
bugfix to http://bugs.yacy.net/view.php?id=181
...
tried to make a bit less 'noise' to dns server
also included: less processes in snippet fetch to reduce load during
search on small computers
13 years ago
3e1bc9477f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
52d307c735
prevent that the snippet fectch process removes catchall entries
13 years ago
89142d1e8d
removed (not all) warnings
13 years ago
Michael Peter Christen