b9d42fd9c8
using com.google.common.io.Files instead of homebrew methods
13 years ago
a5eb91fa60
refactoring
13 years ago
1be0025a9c
- added test for EmbeddedSolrConnector
...
- added needed libraries for this test
this includes most (all) files needed for an embedded solr
13 years ago
dbdd697f4d
moved RDFaParser.xsl configuration file to defaults
13 years ago
90b82ce994
using guava for host resolution (non-blocking for ips) and time-out
13 years ago
e12bb254b4
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
3f55dc7c1e
- added solr core and libraries that solr needs (lucene is missing, will
...
follow later)
- added embedded solr connector which can connect to solr
programmatically (without using a server in between)
13 years ago
c337190a00
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
c63c3a4495
Show additional interaction elements in footer section on each page, if
...
activated in ConfigPortal.html.
This footer is also visible in augmented browsing proxy mode.
13 years ago
786be7d175
better integration of RDFaParser
13 years ago
de3ef8ad73
removed unimportant warnings
13 years ago
82a682b31d
fixed problem with seed when switching network
13 years ago
8c544edee4
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
7dc59979bc
fix for npe, possibly for http://bugs.yacy.net/view.php?id=195
13 years ago
1d4e206b2b
bugfix in vocabulary generation
13 years ago
2c89975378
Merge remote-tracking branch 'original yacy/master'
13 years ago
52f5d40043
better abstraction of document model generation
13 years ago
8b7c4d3144
produce a rdf output containing the triplestore with yacydoc; ie:
...
http://localhost:8090/api/yacydoc.rdf?urlhash=yOiCM7Fh1hyQ
13 years ago
f7160dae5c
Merge remote-tracking branch 'original yacy/master'
13 years ago
e4555cbee3
Augmented browsing: Pass on additional action parameter
13 years ago
24bbe359ca
integrate also geonames library files for less cities. these are more
...
useful for tagging since less normal words are false-identified as
location
13 years ago
223a5440ab
preventing that an empty pnd is inserted into the vocabularies
13 years ago
8e97ada7c9
IPv6 bugfix
13 years ago
963f92ed9a
- merged files
...
- changed behaviour of delete button in vocabulary edit
- fixed size numbe in vocabulary listing
13 years ago
dd88d0ace2
more logging
13 years ago
94d54e2d91
added recognition of multi-word terms in vocabulary matching
...
this makes the PND usable: it is now possible to recognize persons and
navigate with a 'Persons' facet.
13 years ago
64c0268b2b
show triplestore metadata in yacydoc and viewfile
13 years ago
0fbd749207
ipv6 update
13 years ago
c2f0d16d2c
fixed vocabulary initialization
13 years ago
df3531f8d5
added the generation of virtual vocabularies using the pnd
13 years ago
a0f1decd82
- added loading of the dbpedia pnd triplestore in the dictionary loader
...
- renamed the dictionary loader to knowledge loader
- some refactoring in the library provider method names
13 years ago
2ac7a5c1f2
Augmented browsing: Add overlay bar which shows the vocabulary tags
13 years ago
3c255c025b
Show tags in search results (if activated in ConfigPortal_p.html)
13 years ago
16d8f33795
added objectlink generation to vocabulary generation and editor
13 years ago
f49d92d8da
Cleanup of interaction class and helper routines
13 years ago
56b0115054
Triplestore: modify routines to access per user store
13 years ago
a95127c9af
Triplestore: initalize per-user triplestores
13 years ago
d45718251e
refactoring (Localization -> Location)
13 years ago
b8b3c87ba7
- renamed localization to location (that was confusing)
...
- renamed 'Locale' navigator to 'Location'
- produce Location navigation only if geolocation libraries are loaded
13 years ago
e89747bb67
- added automated generation of vocabularies from url stubs
...
- added clear of all terms for vocabularies
- added deletion of vocabularies
13 years ago
79464189a4
The 'Locale' vocabulary, which is generated by geo data, has now the
...
objectspace "http://dbpedia.org/resource/ "
13 years ago
eca38c53e7
added a vocabulary editor
13 years ago
61bb52d55c
- using http://purl.org/dc/terms/references to refer from an
...
auto-annotated document to a 'pseudo-linked' document which has an url
created with an object-prefix as defined in the vocabulary file
13 years ago
2bbb6c52cf
added option to clean the triplestore when deleting the index
13 years ago
50c576599b
allow multiple parser options instead of printing an error
13 years ago
c02d742e53
proper namespaces in triplestore dump
13 years ago
8b53771db2
changed behavior of navigation processing:
...
- vocabulary annotation is not done any more into the metadata of urldb
- vocabularies are written into the jena triplestore using a rdf
vocabulary
- vocabularies for rdf tripel must be updated; refactoring done
- with the new navigation tags in the triplestore a faster
pre-urldb-lookup is possible: navigation is processed now within the RWI
during pre-ranking retrieval
- added also a Owl vocabulary stub to add the plain-text url to the
triplestore using the owl:sameas predicate
13 years ago
5fc6524ca8
- moved triple store to net.yacy.cora.lod (should be generalized there
...
later
- added abstract add, delete, get methods in the triplestore
- added generation of triples after auto-annotation
- migrated all MultiProtocolURI objects to DigestURI in the parser since
the url hash is needed as subject value in the triples in the triple
store
13 years ago
8d2e6355f8
augmented browsing: remove non-existing external snippet file
13 years ago
c90f174799
preparation and generalization of augmented browsing methods
13 years ago
bef823c247
close the reader if finished
13 years ago
4ee6fb1de9
added missing blacklist dht cache storage (maybe due to mistakes in
...
cherry picking)
13 years ago
e4d36fa5eb
Fix to make all values lower-case (this should make all existing blacklists compatible with the new enum)
13 years ago
edaa09b9b1
Rewrote all String blacklist types to enum 'BlacklistType', closes bug
...
#143
Conflicts:
htroot/Supporter.java
htroot/yacy/crawlReceipt.java
htroot/yacy/transferRWI.java
htroot/yacy/transferURL.java
source/de/anomic/crawler/CrawlStacker.java
source/de/anomic/data/ListManager.java
source/net/yacy/peers/Protocol.java
source/net/yacy/repository/Blacklist.java
source/net/yacy/repository/LoaderDispatcher.java
source/net/yacy/search/Switchboard.java
source/net/yacy/search/index/MetadataRepository.java
source/net/yacy/search/index/Segment.java
source/net/yacy/search/query/RWIProcess.java
source/net/yacy/search/snippet/MediaSnippet.java
13 years ago
af5a597e47
Scroogle is not comming back, remove dead code
...
Conflicts:
source/net/yacy/search/Switchboard.java
13 years ago
ca93835713
removed usage of deprecated methods
13 years ago
23e38bd918
do not load the "_triplestore.rdf" files which are of special use in
...
sciencenet
13 years ago
90c6fc4b63
load all - but not the persistent local.rdf - triples from
...
DATA/TRIPLESTORE at startup time. The local.rdf is loaded only if the
persistent switch is on (as before).
13 years ago
bbfc53b663
bugfix
13 years ago
65c5826d93
bugfix
...
Conflicts:
source/net/yacy/document/parser/augment/AugmentParser.java
13 years ago
aa0295917c
augmentation
...
Conflicts:
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
ed2ea0f08e
augmented browsing modification
...
Conflicts:
htroot/interaction/OverlayInteraction.html
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
6b32f7c1f6
re-enable augmented proxy
13 years ago
3b08edec2e
bugfix
...
Conflicts:
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
5f8ba7f4f2
small changes
...
Conflicts:
source/net/yacy/document/parser/augment/AugmentParser.java
source/net/yacy/interaction/Interaction.java
13 years ago
300b235ce8
Updated Demo Servlet
...
Conflicts:
htroot/About.html
htroot/DemoServlet.html
htroot/DemoServlet.java
htroot/interaction/interaction.js
source/net/yacy/interaction/Interaction.java
13 years ago
90512640bf
Added config switches for custom parser
...
Conflicts:
source/net/yacy/document/TextParser.java
13 years ago
df47f31235
interaction: add special table interaction
...
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
e14f2881ae
interaction: add special table interaction
...
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
d7326079a8
interaction: add global variable store
...
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
4e4e7a99f8
interaction: add global variable store
...
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
bde07ed7a8
Add tagging overlay element
...
Conflicts:
htroot/env/templates/jqueryheader.template
htroot/yacysearchitem.java
source/net/yacy/interaction/Interaction.java
13 years ago
b0bc0b4572
Add new demonstration module for client-side key-value store (backend:
...
triplestore): /DemoServletInteraction.html
Conflicts:
source/net/yacy/interaction/Interaction.java
13 years ago
c9dc6cda02
Demonstration: include value from interaction in search results
...
Conflicts:
htroot/interaction/OverlayInteraction.html
htroot/yacysearchitem.java
13 years ago
bcbd8eee33
Add several parsers, for RDFa and rdf files.
...
Conflicts:
source/net/yacy/document/TextParser.java
13 years ago
9ef5a80f4e
add interaction for triples and selector for augmented browsing
...
Conflicts:
htroot/interaction/interaction.js
source/net/yacy/interaction/Interaction.java
13 years ago
282c1620d6
Allow TripleStore to be persistent after reboot
13 years ago
5d20cd324a
Add Triplestore and RDF query interface
...
Conflicts:
build.xml
defaults/yacy.init
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
bc9a618e0a
augmented browsing: ignore js and css, integrate more user interaction
...
Conflicts:
htroot/interaction/Footer.html
source/net/yacy/interaction/AugmentHtmlStream.java
13 years ago
b21048892b
augmentedParser add features and integrate external html parser to
...
modify existing web pages
Conflicts:
addon/YaCy.app/Contents/Info.plist
build.xml
13 years ago
9cbfc1a1c0
augmentedProxy, which forwards every proxy request to a
...
rewrite engine to customize existing webpages. originally implemented by
Florian Richter.
Conflicts:
source/de/anomic/http/server/HTTPDProxyHandler.java
13 years ago
3b992e6b00
using utf8 String compression in Webstructure database
13 years ago
26301a538d
bugfix in Domains - dns-lookup
13 years ago
cde20911bb
saved a bit more ram using UTF8 String compression for OpenGeoDB and
...
Geonames data files.
13 years ago
225ee42879
made the GeoLocation into an interface with the current
...
integer implementation as accuracy implementation of 1.863cm
13 years ago
2280a7b276
- changed initialization order to prefer allocation of memory for table
...
files first
- bugfixes in memory amount calculation
13 years ago
0746308bc2
only the metadata tables shall be able to use the tail cache
13 years ago
7ec9bef0c3
fix for OOM
13 years ago
41c02cb10e
- less restrictions for usage of Table RAM copy
...
- new limit to use the table copy (instead of flag): 400MB available. If
less is available, then a copy is never used. If more is available, then
it can be used if there is a remaining space of at least 200MB
- flush caches more often: flush the Digest cache
13 years ago
b8f56a9803
npe bugfix
13 years ago
dd14b19c26
lazy initialization of block rank table ... only normal web search uses
...
this. When interactive search or location search is used, the block rank
is switched off
13 years ago
ba10caf89a
lazy initialization of database tables
13 years ago
701b9a28a0
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
htroot/PerformanceMemory_p.java
13 years ago
ab7107b34b
fixed RWIProcess queue limits: now discovering hidden results for mass
...
result retrieval
13 years ago
10c9c17d51
fixed handlemap spread factor and null iterator handling
13 years ago
b0095c8d3c
flush the compressor cache when a cleanup is done
13 years ago
a61f44f9e4
lazy initialization of block rank table.
...
this causes that the table is not initialized when there is no search is
done. the effect is most strong if YaCy is started headless which causes
no browser pop-up which otherwise would load the search page and
therefore trigger the initialization of the table.
13 years ago
96e9d77270
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
13 years ago
00f2df1120
a variety of possible memory leak fixes
13 years ago
3dd8376825
added automatic cleaning of cache if metadata and file database size is
...
not equal. It might happen that these data is different because one of
that caches is cleaned after a while or when it is too big. The metadata
is then not cleaned, but now wiped after a checkup process at every
application start. This should cause a bit less memory usage.
13 years ago
d0ec8018f5
fixes for bad long computation
13 years ago
6bb07afcc3
accept also files with other file prefix; used to read 'foreign' cache
...
files
13 years ago
96c8119b50
added GeoLocation / GeoPoint classes which uses less memory than
...
Location/Coordinates and has initializers with correct order of lat,lon
coordinates
13 years ago
461a0ce052
removed warnings
13 years ago
62ae9bbfda
allow more POIs, get more at once
13 years ago
407fdf6968
more bug fixes and performance hacks for search process
13 years ago
a1fe65b115
performance hacks
13 years ago
2fe207f813
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
5aee19daa4
added show from cache in search results (not yet finished)
13 years ago
5e562dcdb7
adopted vocabulary usage within anotation/naviagtion feature of search
...
to new SimpleVocabulary class
13 years ago
514700291a
moved Vocabulary to cora package (added in git
...
964406ad17
13 years ago
0284a4d88f
more fixes for double precision of coordinates
13 years ago
964406ad17
added concurrency enhancement to xml parser
13 years ago
240045cf7c
fix for bad distance computation
13 years ago
e0d8643226
- performance hacks
...
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
13 years ago
7a329465b3
using pre-compile pattern in blacklist; should enhance search speed
13 years ago
6e83b02b83
- bugfix for surrogate file reader
...
- bugfix for location search: suppress empty search
13 years ago
9b4c699526
ehanced location search:
...
- search request are now made using a map boundary
- search results are only computed for the map boundary
- the number of results is adopted to the results in the visible range
- added a double-buffering for the search result markers
- added a search query option for the search results:
/radius/<lat>/<lon>/<radius>
13 years ago
834dc6b263
store more data from interface access
13 years ago
1f48d1528b
performance hacks
13 years ago
c70aaccdc9
better location to generate a guid for rss messages
13 years ago
10da7335ea
performance hack: use a hash cache for all hashes that are computed by a
...
byte array. If this hash is used in a HashMap (which is very often the
case) then this hack eliminates a lot of re-computations of the same
hash.
13 years ago
f8a0cf6d7c
RSSMessages do not need a concurrent hash map -> removed overhead
13 years ago
07ca7e4dd1
enhanced RSS parsing by ensuring that it is parsed with a buffered input
...
stream
13 years ago
7c1feefb28
introduced a default 10 second time-out in rwi normalization time
...
uring search process to prevent endless deadlocks after a very long
running search
13 years ago
8d997d55b6
better logging
13 years ago
65d37e6a20
only ASCII needed in seed bitflags
13 years ago
0f82fb3628
using double instead float for a better release ordering
13 years ago
43c2c6e588
better logging
13 years ago
56087c1f23
bump to httpclient- httpcore-, httpmime- 4.2
13 years ago
71c3163f3d
- fixes to node identification
...
- added link to node in network list
- added marking of portal search node peers
13 years ago
4d3cc02168
replaced old bzip2 library against better documented commons-compress
...
package from http://commons.apache.org/compress/
13 years ago
ad222be7f8
added node state icon in network list
13 years ago
3c2bec681f
added a root node flag: identifies peers with short ping time
13 years ago
c846e9ca14
redesign of the crawler monitor page: show crawled pages instead of
...
queue of urls that shall be crawled
13 years ago
c15fcde1c8
add-on to latest commit
13 years ago
cf47d94888
performance hack to parse numbers inside of substrings without actually
...
generating a substring. This avoids the allocation of a String object
ech time a substring is parsed. Should affect CPU load during RWI
transmission.
13 years ago
7e0ddbd275
added a "fromCache" flag in Response object to omit one cache.has()
...
check during snippet generation. This should cause less blockings
13 years ago
81737dcb18
removed stack trace from swf parser since we cant do anything there
13 years ago
7bf421b9dd
- fixed image search page navigation
...
- removed some deadlocks and ConcurrentModificationExceptions during
DidYouMean collection
13 years ago
c6a09eab0b
synchronization needed
13 years ago
fb94b47b1a
changed queue sizes to have less memory occupied during indexing
13 years ago
76157dc2c3
bugfix for http://bugs.yacy.net/view.php?id=173
13 years ago
6696cb1313
bugfix: lookup of peernames no result for active peer in page IndexControlRWIs_p.html -> Transfer RWI to other Peer
...
SeedDB.lookupByName searche for lowercase peerNames, while MapColumnIndex.getIndex uses peername as is in the keyset.
Changed the index init to insert lowercase peer names as key
13 years ago
c6558cba08
more classification bugs
13 years ago
082831b9d6
search contentdom was checked in wrong way - fixed
13 years ago
ee553d971e
correct typo in scripts_txt comment
13 years ago
f294f2e295
bugfix to http://bugs.yacy.net/view.php?id=181
...
tried to make a bit less 'noise' to dns server
also included: less processes in snippet fetch to reduce load during
search on small computers
13 years ago
acf8d521a2
fix for http://bugs.yacy.net/view.php?id=126
13 years ago
bb88878b4d
the last commit was incomplete..
13 years ago
d320a31ae1
bugfix for http://bugs.yacy.net/view.php?id=186
13 years ago
fa735f4f04
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
3e1bc9477f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
6f8a2fef1f
small speed enhancement using a column factory
13 years ago
d10627d591
More sync in close() methods
...
Conflicts:
source/net/yacy/kelondro/logging/GuiHandler.java
source/net/yacy/kelondro/workflow/InstantBusyThread.java
13 years ago
b3ae2aa41f
With or without 'final'? At least please try it in other methods
...
Conflicts:
source/de/anomic/tools/tarTools.java
13 years ago
fbb946f913
Made a method static (Eclipse suggested it), removed unused import, pk=null check does now output a warning in logfile
13 years ago
52d307c735
prevent that the snippet fectch process removes catchall entries
13 years ago
7eece0256f
moved yacy.logging to defaults according to request in
...
http://bugs.yacy.net/view.php?id=55
13 years ago
89142d1e8d
removed (not all) warnings
13 years ago
5deebd02ea
added serialization
13 years ago
b2175ea4ef
Add possibility to set custom Solr field names for the YaCy default Solr attributes.
...
- Changing the format of YaCy's solr.key.list while maintainig backward compatibility
Federated index config screens adjusted accordingly
- modified the Solr update request to use a 3 min Solr autocommit intervall
13 years ago
15db703808
added missing serialization to remove all warnings
13 years ago
1795a7325b
made HandleSet serializable
13 years ago
e7e381d110
added configuration to switch off redirection following in crawler
13 years ago
2717c1b749
fixed bug in solr interface
13 years ago
f150bc218b
fixed bug in solr error document
13 years ago
cb54c1737b
solrj connector bugfix
13 years ago
a093ccf5eb
Now used synchronization in all close() methods to make sure all objects
...
are 'closed' in an ordered way
Conflicts:
source/de/anomic/http/server/ChunkedInputStream.java
source/de/anomic/http/server/ChunkedOutputStream.java
source/de/anomic/http/server/ContentLengthInputStream.java
source/net/yacy/cora/protocol/Domains.java
source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
source/net/yacy/document/content/dao/PhpBB3Dao.java
source/net/yacy/document/parser/html/AbstractTransformer.java
source/net/yacy/kelondro/blob/BEncodedHeap.java
source/net/yacy/kelondro/blob/HeapReader.java
source/net/yacy/kelondro/index/RAMIndexCluster.java
source/net/yacy/kelondro/io/ByteCountInputStream.java
source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
source/net/yacy/kelondro/table/SQLTable.java
13 years ago
49cab2b85f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
0d58fea210
made multiple connector default
13 years ago
7740c02c56
- enhanced the solr connector
...
- added new multiple connector (to replace singleConnector)
13 years ago
0cf3d36eae
more tolerance in case of corrupted file
13 years ago
acc6db28ff
added missing classes for solr interface
13 years ago
adeb33bb36
better abstraction for solr objects
13 years ago
8864141872
more abstraction in solr connection classes
13 years ago
c00efc2717
made the solr connection more generic
13 years ago
ea2bd43b28
patch for broken configurations
13 years ago
e5ca7f22b1
enhancement in circle drawing
13 years ago
34f4225d7e
less 'wellformed' calls without asserts
13 years ago
a691023d04
*) better formatting for network QPM
...
*) refactoring
13 years ago
77f8e9fb9b
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
ba6aaabc51
refactoring + parser bugfixes
13 years ago
2a0434efa4
Merge commit 'c1f6b4fb5226d3d2f8b2bec9e361f6b3476e03ff'
13 years ago
942896fe46
removed methods not supported by new solrj connector for httpclient 4
...
Error was:
java.lang.UnsupportedOperationException: Client was created outside of
HttpSolrServer
at
org.apache.solr.client.solrj.impl.HttpSolrServer.setDefaultMaxConnectionsPerHost(HttpSolrServer.java:614)
at
net.yacy.cora.services.federated.solr.SolrSingleConnector.<init>(SolrSingleConnector.java:128)
at
net.yacy.cora.services.federated.solr.SolrShardingConnector.<init>(SolrShardingConnector.java:55)
at net.yacy.search.Switchboard.<init>(Switchboard.java:657)
at net.yacy.yacy.startup(yacy.java:222)
at net.yacy.yacy.main(yacy.java:1018)
13 years ago
22e1f68c0b
solrj user authentication patch
13 years ago
09484955dc
added new entry class for embed tags
13 years ago
62f2554a01
- fixed build problems (deprecated methods using httpclient 3.1)
...
- removed httpclient 3.1 lib which was used by solrj (solrj now uses
httpclient 4)
13 years ago
a6d60fc21f
concurrency enhancement in ConfigurationSet
13 years ago
453010bd68
- solved problems with backpath normalization
...
- redesigned in/outbound link handover
- removed iframe links from inbound/outbound in solr scheme
13 years ago
5f5ed33ed8
patch for media search (audio, video apps)
13 years ago
7860c1df80
fix needed for new solrj library
13 years ago
0e13022147
- enhanced solr field documentation
...
- added xml api button to IndexFederated_p - the solr schema.xml file
can be generated by YaCy
13 years ago
19efbf1b0f
- apply directDocByURL to NOLOAD Queue
...
- choose pushing to NOLOAD as default for site crawl
13 years ago
659178942f
- Redesigned crawler and parser to accept embedded links from the NOLOAD
...
queue and not from virtual documents generated by the parser.
- The parser now generates nice description texts for NOLOAD entries
which shall make it possible to find media content using the search
index and not using the media prefetch algorithm during search (which
was costly)
- Removed the media-search prefetch process from image search
13 years ago
a3badd3205
changed search process for images: no more media snippet load process,
...
show only links from index which had been on the text search page
before. This creates a superfast search process for images!
13 years ago
c1f6b4fb52
lookupByIP: prevent comparing of port parameter if called with port -1 (=unknown)
13 years ago
f8cd57c92f
new indexing strategy: ALL links that appear anywhere are indexed, not
...
only links where the content can be parsed. All non-parseable links are
placed into the noload queue. The search process must therefore be able
to filter out non-text search results.
- This fixes the problem that image search results appeared in the text
search.
- The interactive search can retrieve now ALL types of links
- The p2p interface is now extended to retrieve only certain types of
links (text, image, video, apps)
- The search process has an extension to filter the right document type
according to the search query
13 years ago
14f67f217c
refactoring of ContentDomain: now subclass of Classification
13 years ago
8a08c96a82
removed dependency from logging
13 years ago
a1a5b015d8
refactoring: moved document Classification to cora package
13 years ago
33d1062c79
refactoring: the cache belongs to the crawler
13 years ago
4d5da75814
fix for parser problem if a <a>-tag is 'within' html tags with unclosed
...
tags. That prevented the <a> tags from beeing recognized. This is a fix
for http://forum.yacy-websuche.de/viewtopic.php?p=25516#p25516
13 years ago
91a86f0b06
fixed to network graph testing
13 years ago
7b5b9baee0
added citation rank to ranking profile
13 years ago
046f3a7e8d
check if httpc has decompressed the release file and rename the file
...
from .tar.gz to .tar if that happened
13 years ago
02e4dedff2
fix to url citation collection
13 years ago
e32055aa15
added stub classes for
...
- a new database for url reference data ('seen links')
- a new database extending the references to the full url metadata
attributes set which shall replace the old metadata database if it is
finished
- migration help classes stub to use old and new metadata databases
simultanously
13 years ago
ac5d124ee0
experimental implementation of a citation ranking as post-ranking
...
method. (ranking coefficient fixed, need to be made configurable)
13 years ago
8fc86fe397
added storage of full anchor link structure:
...
the links between all pages are now stored. The same index structure as
used for the word index is used to make a reverse link index.
The new file(s) in SEGMENT/default/citation.index.*.blob store the
citation index. This will be used to create much more detailed link
structures for the YaCy apis and to create a better ranking. A ranking
using the citation.index should provide better results especially for
portal indexes and initranets.
13 years ago
0b3f39136e
allow custom ppm lower than minimum button on /Crawler_p.html
...
fixes http://bugs.yacy.net/view.php?id=166
13 years ago
532c7cf827
added physics experiment to the graph plotter. not active by default
13 years ago
aba9b1bfa0
better names for elements of a linked graph
13 years ago
2fc8ecee36
ConcurrentLinkedQueue has a VERY long return time on the .size() method.
...
See
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html
and the following test programm:
public class QueueLengthTimeTest {
public static long countTest(Queue<Integer> q, int c) {
long t = System.currentTimeMillis();
for (int i = 0; i < c; i++) {
q.add(q.size());
}
return System.currentTimeMillis() - t;
}
public static void main(String[] args) {
int c = 1;
for (int i = 0; i < 100; i++) {
Runtime.getRuntime().gc();
long t1 = countTest(new ArrayBlockingQueue<Integer>(c), c);
Runtime.getRuntime().gc();
long t2 = countTest(new LinkedBlockingQueue<Integer>(), c);
Runtime.getRuntime().gc();
long t3 = countTest(new ConcurrentLinkedQueue<Integer>(),
c);
System.out.println("count = " + c + ": ArrayBlockingQueue =
" + t1 + ", LinkedBlockingQueue = " + t2 + ", ConcurrentLinkedQueue = "
+ t3);
c = c * 2;
}
}
}
13 years ago
8aba045ba1
if a new pop-up page is set in config portal, then this page applies
...
also to the default page configuration for the httpd if no path is
given.
13 years ago
8c06925984
animation of the web structure picture
13 years ago
898fa7c3f3
use tld heuristic to check if a domain is local or global
13 years ago
213c8d97f2
use less proccesses in process pool
13 years ago
c639248c23
protection against strange answers from remote peers during search
13 years ago
36e4d82b27
changed ranking
13 years ago
096c17e7cd
added test code
13 years ago
665626a51b
catch OOM errors during scanning
13 years ago
1cd711d005
added classes for citation references (for new citation ranking)
13 years ago
33a405dab8
ipv6 bugfix
13 years ago
c6c61be3f0
fix for http://bugs.yacy.net/view.php?id=148
13 years ago
e0f1e7d904
added new citation reference data structure that shall be used for a
...
citation ranking
13 years ago
e18a4f6b74
more tolerant merge iterator
13 years ago
e101c2e0e2
added changes from copperdust (submitted by email):
...
1. Improved and fixed language detection:
1.1 Identificator.java - recognition fix (improved)
1.2 DCEntry.java - fix (changed detection order due to detection from
tld in many cases is incorrect)
1.3 MultiProtocolURI.java - fixed and enhanced language from tld
detection (all currently used top-level domains; ccTLD added but not
tested).
2. Ukrainian language update.
3. Main Slavic languages langstats (tested and works fine).
13 years ago
8d63a5887c
bugfixes
13 years ago
9ad1d8dde2
complete redesign of crawl queue monitoring: do not look at a
...
ready-prepared crawl list but at the stacks of the domains that are
stored for balanced crawling. This affects also the balancer since that
does not need to prepare the pre-selected crawl list for monitoring. As
a effect:
- it is no more possible to see the correct order of next to-be-crawled
links, since that depends on the actual state of the balancer stack the
next time another url is requested for loading
- the balancer works better since the next url can be selected according
to the current situation and not according to a pre-selected order.
13 years ago
7e4e3fe5b6
free some memory after parsing html
13 years ago
4540174fe0
memory hacks
13 years ago
b4409cc803
small redesign of blob column index and usage
13 years ago
d5c1f2746e
performance hack
13 years ago
803963aebd
performance hack: better space grow in CharBuffer (speeds up html
...
parser)
13 years ago
8b0920b0b5
tried to fix the ipv6 problem as reported in bug
...
but this did not solve all problems because a bug in the apache http
client prevented that it worked. Thread dump:
Caused by: java.lang.NumberFormatException: For input string:
"1450:400c:c01:0:0:0:69"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:458)
at java.lang.Integer.parseInt(Integer.java:499)
at org.apache.http.client.utils.URIUtils.extractHost(URIUtils.java:310)
at
org.apache.http.impl.client.AbstractHttpClient.determineTarget(AbstractHttpClient.java:764)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at net.yacy.cora.protocol.http.HTTPClient.execute(HTTPClient.java:597)
at
net.yacy.cora.protocol.http.HTTPClient.getContentBytes(HTTPClient.java:558)
at net.yacy.cora.protocol.http.HTTPClient.GETbytes(HTTPClient.java:341)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:131)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:74)
at
net.yacy.repository.LoaderDispatcher.loadInternal(LoaderDispatcher.java:274)
at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:164)
at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:150)
at
net.yacy.repository.LoaderDispatcher.loadDocument(LoaderDispatcher.java:355)
at getpageinfo_p.respond(getpageinfo_p.java:97)
13 years ago
e2f8f263e8
changed storage of search words: keep order
13 years ago
ed39ef2890
changed generation of protocol information
13 years ago
0b67a0a5d8
added a column index for tables in blob files. This is heavily used
...
during receiving of DHT submissions and when answering remote search
requests. Both events together may have caused IO-deadlocking and this
commit shall fix that.
13 years ago
2e5cd6a1b2
fixed parser extension deny list generation and usage
13 years ago
8bee1472c9
there is no noindex, only nofollow in links
13 years ago
3cd6dcd352
do not add new solr fields as activated fields
13 years ago
e3bb73c3d6
serialized some database access methods
13 years ago
7e728867e5
added a synchronization around iterations to prevent IO-deadlocking
...
during concurrent remote search requests
13 years ago
355ecf330f
reduced target file site to 64mb
13 years ago
10ae6d94a1
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
2ea585d616
fix for host navigator
13 years ago
2f6dde92e2
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
c560a582ac
fix for single-word vocabulary lines
13 years ago
4c5edab1ec
added option to have exception search result windows
13 years ago
046d7de95b
Merge remote branch 'reger/master'
13 years ago
a95f645a61
Bugfix class repository.Loaddispatcher fixed download file limit of 10000
...
line 355: final Response response = this.load(request, cachePolicy, 10000, true);
13 years ago
ef78f22ee1
performance hack
13 years ago
41536eb4a2
performance hack
13 years ago
f91487fc50
added delete-button for host navigation
13 years ago
e8d24fd802
author navigator can be switched off
13 years ago
558ab7bd4e
made the protocol navigator reversible
13 years ago
96cb75f1d4
made the filetype navigator be able to deselect the search constraint
13 years ago
1f4f60654a
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/document/parser/pdfParser.java
13 years ago
32104360ce
PDFParser - return at least first 3 pages of PDF
...
fix for pdf parsing without returning parsed text due to interruption by
time out.
13 years ago
ef5192f8c9
using the generic document parser for crawl starts instead of the html
...
parser. This makes it possible that every type of document can be a
crawl start point, not only text documents or html documents. Testet
this with a pdf document.
13 years ago
a02fdf8625
better error messages
13 years ago
eadb58dd87
small enhancements in pdf parser
13 years ago
c6ba44468e
timeout = 5000 instead 3000
13 years ago
b616de5973
PDFParser - return at least first 3 pages of PDF
...
fix for pdf parsing without returning parsed text due to interruption by time out.
13 years ago
c73af39e54
refactoring of tray icon class,
...
now uses Java 6 methods natively
13 years ago
4eff0e26f1
npe bugfix
13 years ago
8776b84c10
*) small fix to make password change function of reconfigureYACY.sh work
...
again
13 years ago
1a0b6b3913
get more navigation details to search results
13 years ago
7f9b6b7a0c
added switches to ConfigParser to accept/deny documents by their
...
extension
13 years ago
4901cee3cc
suppress auto-tagged subject entries when sending out or receiving
...
metadata from other peers
13 years ago
83009d86f7
added the vocabulary navigator. It can be very simply tested by
...
switching on the locale dictionaries.
13 years ago
985b78cf89
correct 'avaiable()' to use max of young / eden
13 years ago
4da8746275
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
c9aaa9e00a
respect non-reserved Memory in GenerationMemoryStrategy
...
and enable it again
13 years ago
37f2d1b3e9
replaced Thread initialization with ExecutorService pool for delete
...
method. This is much faster and produces less blocking when using the
Compressor class which is used by the HTCache. I.e. picture search is
much faster now.
13 years ago
a58dc4a91f
added autotagging to document condenser:
...
- tags that are automatically generated now enrich the dc:subject
- auto-generated tags have a '$' at the beginning of the tag
- auto-generated tags lead the tag name with a vocabulary name
each tag has the form
$<vocabulary-name>:<tag-printname-space-replaced-by-'_'>
13 years ago
0d6176804b
emergency disabling of GenerationMemoryStrategy because of non-working
...
available-method
13 years ago
411aab02e3
Windows installer now detects reliably whether YaCy runs. A file lock on
...
the yacy.running file has been implemented.
13 years ago
87f0210480
enriched log output to find NPE in HeapReader
13 years ago
987b412491
updated solr scheme: generic declaration of solr schemes
13 years ago
254adea51c
small fixes
13 years ago
49be60a7c8
WorkflowProcess is forced to make small pauses if shortMemoryStatus is
...
reached.
13 years ago
b7bb84c0bb
set a limit to CharBuffer object size to fight against bad/too large
...
content
13 years ago
c602eaaf46
enhanced search process
13 years ago
087f97d4c0
less noise if a browser cannot be opened
13 years ago
eff966f396
fix for search process (it was aborted too early during remote search)
13 years ago
e6d51363ee
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
a231d0eeb9
Run from Java the whole app YACY
...
start for java webStart
allow for better integration with IDE
Conflicts:
source/net/yacy/gui/framework/Browser.java
13 years ago
72adbeae90
!Important: move from Hashtable to HashMap
...
Hashtable is an obsolete collection v1, now since v2 offers HashMap with same or better
functionality. Please review, almost all code was already moved, so only a few changes. That is not the issue,
but I found notices that some (ugly big) helper classes had to be created in past
to compensate missing Hashtable's functionality. I'd like input if we can remove some of them.
look for //FIX: if these commits
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
f40efb39af
Blacklist loadList() remove duplicates by using Set
...
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
f75b5e40e0
little fix in copy()
...
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
1dc5d9f0f3
make ConnectionInfo comparable and sort list of connections in Connections_p
...
ConnectionInfo compare by initTime
Connections_p implement wish to sort connections, descending
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
fa8da7f89d
vocabularies are now also used as source for a did-you-mean computation
13 years ago
eaec14ecc4
Dictionaries from words caches can now be used as autotagging vocabulary
13 years ago
91940fdf56
redesign of WordCache to be prepared to hold multiple
...
independent dictionaries. Such dictionaries can then be also used as
simplified vocabularies.
13 years ago
bd40a10230
added autotaggig stub .. only reading and parsing of vocabularies at
...
this time
13 years ago
2ee8cbeb2c
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/search/Switchboard.java
13 years ago
992dbdf4bb
added noload statistic to servlets
13 years ago
eebc02f5c1
fix
13 years ago
Michael Peter Christen