3b992e6b00
using utf8 String compression in Webstructure database
13 years ago
a1fe65b115
performance hacks
13 years ago
e0d8643226
- performance hacks
...
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
13 years ago
7c1feefb28
introduced a default 10 second time-out in rwi normalization time
...
uring search process to prevent endless deadlocks after a very long
running search
13 years ago
65d37e6a20
only ASCII needed in seed bitflags
13 years ago
0f82fb3628
using double instead float for a better release ordering
13 years ago
71c3163f3d
- fixes to node identification
...
- added link to node in network list
- added marking of portal search node peers
13 years ago
ad222be7f8
added node state icon in network list
13 years ago
3c2bec681f
added a root node flag: identifies peers with short ping time
13 years ago
f294f2e295
bugfix to http://bugs.yacy.net/view.php?id=181
...
tried to make a bit less 'noise' to dns server
also included: less processes in snippet fetch to reduce load during
search on small computers
13 years ago
89142d1e8d
removed (not all) warnings
13 years ago
15db703808
added missing serialization to remove all warnings
13 years ago
a093ccf5eb
Now used synchronization in all close() methods to make sure all objects
...
are 'closed' in an ordered way
Conflicts:
source/de/anomic/http/server/ChunkedInputStream.java
source/de/anomic/http/server/ChunkedOutputStream.java
source/de/anomic/http/server/ContentLengthInputStream.java
source/net/yacy/cora/protocol/Domains.java
source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
source/net/yacy/document/content/dao/PhpBB3Dao.java
source/net/yacy/document/parser/html/AbstractTransformer.java
source/net/yacy/kelondro/blob/BEncodedHeap.java
source/net/yacy/kelondro/blob/HeapReader.java
source/net/yacy/kelondro/index/RAMIndexCluster.java
source/net/yacy/kelondro/io/ByteCountInputStream.java
source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
source/net/yacy/kelondro/table/SQLTable.java
13 years ago
a691023d04
*) better formatting for network QPM
...
*) refactoring
13 years ago
77f8e9fb9b
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
ba6aaabc51
refactoring + parser bugfixes
13 years ago
2a0434efa4
Merge commit 'c1f6b4fb5226d3d2f8b2bec9e361f6b3476e03ff'
13 years ago
c1f6b4fb52
lookupByIP: prevent comparing of port parameter if called with port -1 (=unknown)
13 years ago
f8cd57c92f
new indexing strategy: ALL links that appear anywhere are indexed, not
...
only links where the content can be parsed. All non-parseable links are
placed into the noload queue. The search process must therefore be able
to filter out non-text search results.
- This fixes the problem that image search results appeared in the text
search.
- The interactive search can retrieve now ALL types of links
- The p2p interface is now extended to retrieve only certain types of
links (text, image, video, apps)
- The search process has an extension to filter the right document type
according to the search query
13 years ago
14f67f217c
refactoring of ContentDomain: now subclass of Classification
13 years ago
33d1062c79
refactoring: the cache belongs to the crawler
13 years ago
046f3a7e8d
check if httpc has decompressed the release file and rename the file
...
from .tar.gz to .tar if that happened
13 years ago
8c06925984
animation of the web structure picture
13 years ago
c639248c23
protection against strange answers from remote peers during search
13 years ago
7e4e3fe5b6
free some memory after parsing html
13 years ago
b4409cc803
small redesign of blob column index and usage
13 years ago
0b67a0a5d8
added a column index for tables in blob files. This is heavily used
...
during receiving of DHT submissions and when answering remote search
requests. Both events together may have caused IO-deadlocking and this
commit shall fix that.
13 years ago
7e728867e5
added a synchronization around iterations to prevent IO-deadlocking
...
during concurrent remote search requests
13 years ago
ef5192f8c9
using the generic document parser for crawl starts instead of the html
...
parser. This makes it possible that every type of document can be a
crawl start point, not only text documents or html documents. Testet
this with a pdf document.
13 years ago
72adbeae90
!Important: move from Hashtable to HashMap
...
Hashtable is an obsolete collection v1, now since v2 offers HashMap with same or better
functionality. Please review, almost all code was already moved, so only a few changes. That is not the issue,
but I found notices that some (ugly big) helper classes had to be created in past
to compensate missing Hashtable's functionality. I'd like input if we can remove some of them.
look for //FIX: if these commits
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
216a287a85
Merge commit '6d4e08ed06c5cd28c45981b2ebe31c7f7ec6fd83' into quix0r
...
Conflicts:
source/de/anomic/crawler/CrawlQueues.java
13 years ago
d18095dc48
Patch fuer Issue 0000102
...
and fixes to Patch (private peer status is a property of a peer, not a
status)
13 years ago
9f1b1b4604
Type for Robinson-Mode/Private Perr added
13 years ago
fa08ed5ae5
Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check
13 years ago
85bd4cc8bc
better lookup for peer names
13 years ago
20e3084bd4
redesign of fining of peers by ip: more leightweight method to read the
...
seed databases
13 years ago
0797b0de99
new handling of remote search processes: looking for seeds will now not
...
block the whole search process any more. A deadlock with a DHT selection
process may have been the cause for interface lockings in the past.
13 years ago
9e5894c784
Removed handling of components objects for URIMetadataRows.
...
This is a preparation to replace this rows with nodes from the node
store.
13 years ago
c04bfaa51b
refactoring
13 years ago
1f4afb4dc0
performance hacks
13 years ago
675d557e88
removed debug logging
13 years ago
e9dc99fe15
added rules to set specific RWIs as private RWIs which are not
...
transmitted to remote peers. This will be used for private index copies
and phonetic indexes.
13 years ago
044f83feed
added some pauses into the search process which shall produce
...
better-ranked search results. without that pauses the result page will
only contain links from the peer that answers first which is not a good
average picture of all the peers that provided results
13 years ago
f14faf503b
better ranking because we wait a very little time during the search
...
process more to get better remote sear results into the ranking priority
stack
13 years ago
e7e429705a
- less automatic indexing after a search (needs to reset the default
...
crawl profiles)
- fix for concurrency problem in storage of serverSwitch Properties
- markup update
13 years ago
484c4ad339
Merge branch 'master' of git://github.com/f1ori/yacy
13 years ago
402e9d71ef
changed ording on release files: main criteria is not the svn any more; releases are now ordered by
...
- release number
- date
- svn number
additionally there is a new option to remove the svn number completely
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8135 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
56ce8488e4
Merge branch 'master' of git://github.com/f1ori/yacy
13 years ago
4b8ff84705
- search bugfixes (page counter and number of results per page; recognition of new search)
...
- experiments to speed-up the network image production (commented out)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8130 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
aeeae75b8a
the timeout of httpclient is not absolut, but till a connection is
...
established or between bytes send
trying this to reduce count of client-connections to /yacy/search.html
of other peers
13 years ago
Michael Peter Christen