f9c0e6e950
- Implemented and integrated the URIMetadataNode object which is a
...
metadata representation from the solr index. This shall replace metadata
from the built-in database in the future.
- added the Solr-driven metadata into the search index of YaCy which
makes it now possible to run YaCy without the old metadata index. This
is a major stept forward to a full migration to Solr.
13 years ago
b2b480fff2
more abstraction of the YaCySchema -> Opensearch matching process
13 years ago
73f6d69d03
more abstraction for solr query params parsing
13 years ago
24462e9baa
set the title every time, it is possible that it has changed
13 years ago
136fcb1ad9
refactoring
13 years ago
a12f693ec9
added two response writer for embedded solr interface:
...
a rss/opensearch writer and an enhanced solr xml writer.
The enhanced solr writer has less configuration overhead than the
original writer and should by slightly faster. The rss/opensearch writer
is at this time slightly incomplete compared with the already existing
rss search result form YaCy and also snippets are missing at this time.
To test the new interface, open for example:
http://localhost:8090/solr/select?wt=rss&q=olympia
The wt-code for the new result writers are=
wt=rss for opensearch
wt=exml for the enhanced solr xml writer.
Additionally, the SRU search parameters had been added to the solr
interface which can now also be used for a normal solr/xml search.
13 years ago
67edfd991c
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
d9173ba7ed
added more solr fields to integrate values from URIMetadataRow. All
...
writings to the Metadata-DB are now also done to solr. This includes
metadata transfer during search and rwi transfer.
The new/added solr fields are:
## time when resource was loaded
load_date_dt
## date until resource shall be considered as fresh
fresh_date_dt
## id of the host, a 6-byte hash that is part of the document id
host_id_s
## ids of referrer to this document
referrer_id_ss
## the md5 of the raw source
md5_s
## the name of the publisher of the document
publisher_t
## the language used in the document; starts with primary language
language_ss
## an external ranking value
ranking_i
## the size of the raw source
size_i
## number of links to audio resources
audiolinkscount_i
## number of links to video resources
videolinkscount_i
## number of links to application resources
applinkscount_i
13 years ago
70b10e8316
added the JSON response writer to solr interface, add &wt=json to the
...
servlet GET properties to use this format
13 years ago
8d944f6517
nowrap from gaston in forum
...
http://forum.yacy-websuche.de/viewtopic.php?p=26815#p26815
13 years ago
24d9db1613
snippet retrieval loading processes may use a smaller minimum load time
...
value than crawling processes. This speeds up the search result
preparation dramatically.
13 years ago
1687737771
Abstraction of HandleMap and HandleSet
13 years ago
3bcd9d622b
cleaned up classes and methods which are either superfluous at this time
...
or will be superfluous or subject of complete redesign after the
migration to solr. Removing these things now will make the transition to
solr more simple.
13 years ago
6f1ddb2519
Moved solr index-add method to the same method where the YaCy index is
...
written. Also done some code-cleanup.
13 years ago
315d83cfa0
cleanup
13 years ago
76202f068e
extended abstraction of local and remote solr index using one front-end
...
for index administration and querying.
13 years ago
7ec7341f60
added user-authentication protection to solr search (same as implemented
...
for yacysearch)
13 years ago
e2a97ef8f6
better explain how to access the embedded solr
13 years ago
826967513b
changed options in IndexFederated_p to switch on/off parts of the index
...
individually. The settings are experimental and the values of the
settings will be overwritten when an index migration from urldb to solr
starts.
13 years ago
cba4ab862e
fix for http://bugs.yacy.net/view.php?id=202
13 years ago
36c9875b6e
removed localized number formatting from num-results_totalcount response (this is only used in xml and json where localized format is not valid)
13 years ago
69e743d9e3
- more abstraction for the RWI index as preparation for solr integration
...
- added options in search index to switch parts of the index on or off
13 years ago
6cc5d1094e
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
05a3ffd03a
patches to ensure that solr connectors are active ony if they have a
...
solr object assigned and vice versa
13 years ago
5a3c829872
embedded solr is only initiated if it is activated with
...
IndexFederated_p.html
13 years ago
3a350a2f83
partial html fix for
...
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4454
13 years ago
97b7bcf2a6
added a solr search index
...
- by default, a (empty) solr storage instance is created at
SEGMENTS/solr_36
- the index is written if in /IndexFederated_p.html the flag "embedded
solr search index" is switched on
- a standard solr query interface is available now with a new servlet at
http://127.0.0.1:8090/solr/select
To test this, do the following:
- switch to webportal mode
- switch on the feature as described
- do a crawl. this fills the solr index. The normal YaCy search will NOT
work now!
- do a solr query, like:
http://127.0.0.1:8090/solr/select?q= *:*
http://127.0.0.1:8090/solr/select?q=text_t:Help
play with different search fields as you can see in
/IndexFederated_p.html
You can use the standard solr query attributes as described in
http://wiki.apache.org/solr/SearchHandler
13 years ago
f78ce93a80
collection of speed and memory saving hacks
13 years ago
c00a3cf74d
less usage of generic logger to avoid logger generation overhead
13 years ago
e76159040b
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
bbfa497a3c
replaced more size() > 0 by !isEmpty()
13 years ago
e3aa05b9dd
added creation of subpath pattern when crawl start is 'from file'
13 years ago
0cbda0b2b8
- replaced all length() == 0 and size() == 0 with isEmpty()
...
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
13 years ago
aef9dd0350
- removed cleaning of blacklist cache on startup
...
- added cleaning of blacklist cache if cache is modified in interface
- extended cache saving to all cache types
- moved cache location to DATA/LISTS
- fixed static file path which was relative to the application path but
should be relative to data path - which is different in debian and mac
implementations
13 years ago
c7afa8bc48
using SwitchboardConstants for solr attributes
13 years ago
62202e2d71
refactoring of query attribute variable names for better consistency
...
with (next) stored query words
13 years ago
91f14ea38e
fix to solr configuration (case where the external solr was not online)
13 years ago
2c5b68d932
more abstraction of error message
13 years ago
9758c521ab
abstraction of error message
13 years ago
9b6e4e46ca
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4430
13 years ago
b0c408788b
made class methods static where possible
13 years ago
5bd3c90907
- removed unnecessary semicolons
...
- added default case for switch
13 years ago
7c1ba99755
removed more unused method parameters
13 years ago
0301aba1e9
removed unused method parameters
13 years ago
241dd8410a
removed snippet pattern filter - it was not used
13 years ago
d3964253ae
- added @SuppressWarnings to unused servlet method parameters
...
- removed unnecessary casts
- removed unnecessary throw statements
13 years ago
ea10766bfd
cleaned unnecessary nested code
13 years ago
78fc3cf8f8
refactoring and new usage of SentenceReader: this class appeared as one
...
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
13 years ago
276a66a793
Adding a limit of 1000 links that a parser shall store during indexing.
...
A limit was necessary because some web pages have such huge numbers of
links that it can easily cause a OOM just by the number of links.
The quesion if the number of 1000 links is sufficient or too weak must
be answered with the result of testing this feature.
13 years ago
1825f165b8
better integration of blacklist according to use case
13 years ago
c18fa9fa75
Merge branch 'master' of git://gitorious.org/~reger/yacy/bbyacy-rc1
13 years ago
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
13 years ago
067728bccc
add search result heuristic. adding a crawl job with depth-1 for every displayed search result (crawling every external linked page of displayed search result pages)
13 years ago
03280fb161
removed segments-concept and the Segments class:
...
the segments had been there to create a tenant-infrastructure but were
never be used since that was all much too complex. There will be a
replacement using a solr navigation using a segment field in the search
index.
13 years ago
9116013c64
- allow lazy initialization of solr value (if using 'lazy', then no
...
0-values and no empty strings are written). This may save a lot of
memory (in ram and on disc) if excessive 0-values or empty strings
appear)
- do not allow default boolean values for checkboxes because that does
not make sense: browsers may omit the checkbox attribute name if the box
is not checked. A default value 'true' would not comply with the
semantic of the browsers response.
- add a checkbox in IndexFederated_p for the lazy initialization of solr
fields.
13 years ago
e6792ed37d
Merge remote-tracking branch 'original yacy/master'
13 years ago
96aeb127e3
generalized localhost naming.
...
this is also a preparation for a better IPv6 implementation.
13 years ago
77f795756c
fixing redirects and status codes: storing of status code in
...
ResponseHeader to make it available for late evaluations, like storage
in solr.
13 years ago
8dd469b9dd
added option to configure the autocommit delay time of solr on-the-fly
13 years ago
b9dfca4b0a
- fixed IndexFederated Servlet / a embedded Solr can now be selected
...
- added code stub for an embedded Solr but generation of Solr store is
still commented out (it works but is not yet ready for usage)
13 years ago
fad3b14813
added jetty libraries, needed for future use as web server and as
...
application server for the solr search interface
13 years ago
b9d42fd9c8
using com.google.common.io.Files instead of homebrew methods
13 years ago
a5eb91fa60
refactoring
13 years ago
c1ba58ae51
Augmented browsing: Small CSS fix
13 years ago
b2b205aa38
Augmented browsing: small js fix
13 years ago
dc9ee0cdb3
Augmented browsing: CSS fix
13 years ago
74fcc6f8c5
Augmented browsing: small UI modifications
13 years ago
c63c3a4495
Show additional interaction elements in footer section on each page, if
...
activated in ConfigPortal.html.
This footer is also visible in augmented browsing proxy mode.
13 years ago
fa98657bb3
Augmented Browsing: changed the settings page
13 years ago
751eeade0d
Merge remote-tracking branch 'original yacy/master'
13 years ago
84a11ec48c
Corrected loading of default page settings on ConfigPortal.html
13 years ago
bea002dc15
correct table in new look of Crawler_p
13 years ago
8738336408
set Xms lower than Xmx
13 years ago
6b4545d6b0
Only load tag information if necessary
13 years ago
011f8a5818
Auto Tagging: Add hyperlinks to tags (provisional)
13 years ago
1d4e206b2b
bugfix in vocabulary generation
13 years ago
2c89975378
Merge remote-tracking branch 'original yacy/master'
13 years ago
71047fe63a
Augmented browsing: CSS fix
13 years ago
52f5d40043
better abstraction of document model generation
13 years ago
8b7c4d3144
produce a rdf output containing the triplestore with yacydoc; ie:
...
http://localhost:8090/api/yacydoc.rdf?urlhash=yOiCM7Fh1hyQ
13 years ago
f7160dae5c
Merge remote-tracking branch 'original yacy/master'
13 years ago
e4555cbee3
Augmented browsing: Pass on additional action parameter
13 years ago
24bbe359ca
integrate also geonames library files for less cities. these are more
...
useful for tagging since less normal words are false-identified as
location
13 years ago
5a41e739b4
better apilink description
13 years ago
e16e4bd2ba
added ontology extraction in xml as api call for vocabularies
13 years ago
8cf47a8335
Merge remote-tracking branch 'original yacy/master'
13 years ago
b85f01a14e
Augmented browsing: small UI fix
13 years ago
26cb1c65c2
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/document/importer/OAIPMHLoader.java
13 years ago
963f92ed9a
- merged files
...
- changed behaviour of delete button in vocabulary edit
- fixed size numbe in vocabulary listing
13 years ago
d8815db877
Merge remote-tracking branch 'original yacy/master'
13 years ago
e4dab19045
Augmented Browsing: added template for document info bar
13 years ago
743b0ec89f
- added size of vocabulary to vocabulary view
...
- fixed bad terms in vocabulary-from-titles autogeneration
13 years ago
22d5e33c5e
added more methods to vocabulary generation: scrape document title and
...
document author to vocabulary
13 years ago
b2d1c25ebb
removed warnings/unused entities
13 years ago
f1aa4c4390
- accept only location names wit a minimum length
...
- remove comma from synonym terms
13 years ago
cc9ad7198a
- use only names which consists of at least two parts
...
- remove word from derewo from locations
13 years ago
9264d8b4af
removed old navigation practice using subject tags in favor of
...
triplestore-tags
13 years ago
eeb4fd8b8c
refactoring (geolocalzation -> geolocation)
13 years ago
64c0268b2b
show triplestore metadata in yacydoc and viewfile
13 years ago
c2f0d16d2c
fixed vocabulary initialization
13 years ago
fbded1f466
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
df3531f8d5
added the generation of virtual vocabularies using the pnd
13 years ago
e806106b10
jquery bugfix
13 years ago
a0f1decd82
- added loading of the dbpedia pnd triplestore in the dictionary loader
...
- renamed the dictionary loader to knowledge loader
- some refactoring in the library provider method names
13 years ago
6d17686258
made triplestore persistent by default
...
added a size display in triplestore servlet
13 years ago
8d6e77ad0c
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
2ac7a5c1f2
Augmented browsing: Add overlay bar which shows the vocabulary tags
13 years ago
777d22e145
renamed "augmented proxy" to "augmented browsing"
13 years ago
bddac2839e
add missing files for tag display
13 years ago
441430f507
Merge remote-tracking branch 'original yacy/master'
13 years ago
3c255c025b
Show tags in search results (if activated in ConfigPortal_p.html)
13 years ago
1f9120d189
create new vocabularies also without an objectspace. this creates an
...
empty vocabulary
13 years ago
a5cdfb91de
- fixed Cache link (below snippet)
...
- added 'Augmented Proxy' link below snippet
- added configuration options for augmented proxy
13 years ago
492b3e09f2
added api icon to triplestore
13 years ago
16d8f33795
added objectlink generation to vocabulary generation and editor
13 years ago
f1f97b7c95
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
b3eaaf5ebc
check also delete triplestore by default
13 years ago
f2f07a11f1
hotfix for unresolved pattern
13 years ago
5fd1a15fcf
hotfix until we have updated query routine for tags
13 years ago
f49d92d8da
Cleanup of interaction class and helper routines
13 years ago
56b0115054
Triplestore: modify routines to access per user store
13 years ago
d45718251e
refactoring (Localization -> Location)
13 years ago
b8b3c87ba7
- renamed localization to location (that was confusing)
...
- renamed 'Locale' navigator to 'Location'
- produce Location navigation only if geolocation libraries are loaded
13 years ago
f64e78497a
fix for reload-feature in Crawler_p
13 years ago
e89747bb67
- added automated generation of vocabularies from url stubs
...
- added clear of all terms for vocabularies
- added deletion of vocabularies
13 years ago
79464189a4
The 'Locale' vocabulary, which is generated by geo data, has now the
...
objectspace "http://dbpedia.org/resource/ "
13 years ago
eca38c53e7
added a vocabulary editor
13 years ago
80e8aaabc8
moved new servlets into one submenu "Content Semantic"
13 years ago
2bbb6c52cf
added option to clean the triplestore when deleting the index
13 years ago
8b53771db2
changed behavior of navigation processing:
...
- vocabulary annotation is not done any more into the metadata of urldb
- vocabularies are written into the jena triplestore using a rdf
vocabulary
- vocabularies for rdf tripel must be updated; refactoring done
- with the new navigation tags in the triplestore a faster
pre-urldb-lookup is possible: navigation is processed now within the RWI
during pre-ranking retrieval
- added also a Owl vocabulary stub to add the plain-text url to the
triplestore using the owl:sameas predicate
13 years ago
5fc6524ca8
- moved triple store to net.yacy.cora.lod (should be generalized there
...
later
- added abstract add, delete, get methods in the triplestore
- added generation of triples after auto-annotation
- migrated all MultiProtocolURI objects to DigestURI in the parser since
the url hash is needed as subject value in the triples in the triple
store
13 years ago
c90f174799
preparation and generalization of augmented browsing methods
13 years ago
edaa09b9b1
Rewrote all String blacklist types to enum 'BlacklistType', closes bug
...
#143
Conflicts:
htroot/Supporter.java
htroot/yacy/crawlReceipt.java
htroot/yacy/transferRWI.java
htroot/yacy/transferURL.java
source/de/anomic/crawler/CrawlStacker.java
source/de/anomic/data/ListManager.java
source/net/yacy/peers/Protocol.java
source/net/yacy/repository/Blacklist.java
source/net/yacy/repository/LoaderDispatcher.java
source/net/yacy/search/Switchboard.java
source/net/yacy/search/index/MetadataRepository.java
source/net/yacy/search/index/Segment.java
source/net/yacy/search/query/RWIProcess.java
source/net/yacy/search/snippet/MediaSnippet.java
13 years ago
213f006bf1
One is okay ...
...
Conflicts:
htroot/Trails.html
13 years ago
af5a597e47
Scroogle is not comming back, remove dead code
...
Conflicts:
source/net/yacy/search/Switchboard.java
13 years ago
7a4dab6d1d
- removed unused variables
...
- do not replace malformed or invalid URLs in urlproxy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7835
6c8d7289-2bf4-0310-a012-ef5d649a1542
Conflicts:
source/de/anomic/http/server/HTTPDFileHandler.java
13 years ago
90c6fc4b63
load all - but not the persistent local.rdf - triples from
...
DATA/TRIPLESTORE at startup time. The local.rdf is loaded only if the
persistent switch is on (as before).
13 years ago
a9eb40c160
fix for autocomplete in index.html
13 years ago
dd020a1a8a
removed autocrawler and feedback servlet link since that was not
...
cherry-picked
13 years ago
aa0295917c
augmentation
...
Conflicts:
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
87a3fbb3c2
interaction javascript
13 years ago
ed2ea0f08e
augmented browsing modification
...
Conflicts:
htroot/interaction/OverlayInteraction.html
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
d4802dc8d5
small change
13 years ago
a120ef660b
RDF demo servlet
13 years ago
09a34cfe1b
prepare RDF dump routines
13 years ago
300b235ce8
Updated Demo Servlet
...
Conflicts:
htroot/About.html
htroot/DemoServlet.html
htroot/DemoServlet.java
htroot/interaction/interaction.js
source/net/yacy/interaction/Interaction.java
13 years ago
90512640bf
Added config switches for custom parser
...
Conflicts:
source/net/yacy/document/TextParser.java
13 years ago
a12cbcba36
Add a global value store
13 years ago
e14f2881ae
interaction: add special table interaction
...
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
4e4e7a99f8
interaction: add global variable store
...
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
Michael Peter Christen