1c21b3256d
fix for robots.txt handling: delete old entry before starting a new
...
crawl.
11 years ago
c250fac9f4
linkstructure refactoring to get more options for clickdepth analysis
11 years ago
8068e68474
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
bd886054cb
new structure and enhancements for link graph computation:
...
- added order option to solr queries to be able to retrieve document
lists in specific order, here: link length
- added HyperlinkEdge class which manages the link structure
- integrated the HyperlinkEdge class into clickdepth computation
- extended the linkstructure.json servlet to show also the clickdepth
and other statistic information
11 years ago
f326a67561
fix: typo in default charset in metadata2solr
...
update pom and NB build to Solr 4.7.1 libs
11 years ago
df138084c0
do solr optimization independently from memory and load constraints:
...
- not doing an optimization will likely cause a too many files exception
- without optimization performance will be even worse which would
prevent optimization in the future as well (prevent a deadlock
situation)
11 years ago
ebd44a7080
replaced solr 4.6.1 with solr 4.7.1 and added index migration to
...
lucene_47
11 years ago
734778c0c8
fixed a time-out problem in the default servlet which is also a logging
...
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).
11 years ago
466d90ad42
fixed a problem with resource observer; probably coming from uncatched
...
exceptions within the apache library which appear only in concurrency
environments.
11 years ago
e8ddd415a8
enhanced the new link structure graph
11 years ago
926d28dd3f
fixed a bug which prevented crawl starts after a network switch
11 years ago
3ce8eff21b
another fix for inbound/outbound detection
11 years ago
d4b5c457e4
NPE fix
11 years ago
36a66b0704
fix for parsing of numeric value in case that boolean values are given
11 years ago
41730c8048
better logging in template engine: shows filename of servlets where
...
errors in templates occur
11 years ago
3c1274057d
fixed thread dump in case of wrong seeds
11 years ago
18f9c40302
moved Edge class out of linkstructure servlet as this does not work on
...
non-eclipse driven environments (all non-dev cases)
11 years ago
de95e5e524
reduced search activity corona strength in network image
11 years ago
da413af664
move baseurl after parsing orig source in urlproxyservlet
...
to calculate absolute href links for rewrite from unmodified source.
11 years ago
af6ad20728
fix: remove obsolete ref to yacy.home
...
(use Switchboard instead)
11 years ago
74ab094587
fix for solr query size; too many documents had been retrieved in case
...
that less than _pagesize_ had been requested.
11 years ago
c64c10ef00
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
48fbfa60c1
bugfix to inbound/outbound identification
11 years ago
227c42bc96
eleminate obsolete URIMetaDataRow class
...
by joining it with/into URIMetaDataNode.
11 years ago
cca851a417
introduced new solr field crawldepth_i which records the crawl depth of
...
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.
11 years ago
b1ba764d81
fix for first start options and added german translation for popup texts
11 years ago
429a874222
- added COLS field in GSA response (non-gsa standard by customer
...
request)
- updated document link in GSA response writer
11 years ago
1b9ec9a1c5
- added popover to p2p/stealth mode button to explain the peer mode and
...
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts
11 years ago
62a36fa584
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c9f92abddc
fix: application link count
...
(URIMetadataNode)
11 years ago
a267c46e1a
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
5b83887da8
npe fix
11 years ago
63c9fcf3e0
free configuration of postprocessing clickdepth maximum depth and time
11 years ago
39b641d6cd
added tutorial mode - some menu items will only appear if you 'qualify'
...
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.
11 years ago
f06775850f
fix receiving DHT / parse pultipart
...
+ another close to fix possible resource leak warning
11 years ago
49e76a1c55
make use of detected charset in htmlParser if none is given.
11 years ago
e11504309f
adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html)
11 years ago
b12200cafe
alternative UrlProxyServlet (for /proxy.html) using different url rewrite rules
...
- use JSoup parser for selective rewrite of html body <a href= links only,
instead of regex which rewrites also header href/src links
- this improves display of pages which use header <base> tag
- tags with src attribute are taken from original location (like css) improving display and are not routed trough the indexer
Disadvantage: scripting links will drop out of proxy
Setting of the servlet through web.xml exclusivly (in case one would like to quickly switch back to the YaCyProxyServlet,
leaving the existing code of YaCyProxyServlet untouched available)
11 years ago
2953ebe701
fix: port in local target adress
...
& button style
11 years ago
fda591695c
fixed visibility of custom icon
11 years ago
a9b9950d7f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
b488f33975
added close to fix possible resource leak warning
11 years ago
56710ecb26
prevent opening of new files as that could be a cause for the latest
...
too-many-open-files exception. The old file is just truncated if the
table is cleaned.
11 years ago
8b44fcf0f4
added missing @Override annotation
11 years ago
d7055904a6
fix: proxyservlet path header setting
11 years ago
e515dd460d
added linkscount_i and linksnofollowcount_i to the default solr schema
11 years ago
1a764135be
one more Thread Dump fix for new bootstrap css style
11 years ago
bb21d825f9
fix for thread dump line spacing
11 years ago
cbdfef7ce1
changed protocol facet to show also all other counts if one facet is
...
selected
11 years ago
b9056ef2db
remove unused private header entries (HeaderFramework)
...
X_YACY_ORIGINAL_REQUEST_LINE
X_YACY_KEEP_ALIVE_REQUEST_COUNT
CONNECTION_PROP_REQUESTLINE
11 years ago
6d16fa993d
make transparent proxy handle https-connections:
...
the implemented handle for connect did not work for me - so lets try the
connectHandler
11 years ago
61ad194065
fix for source and target clickdepth in webgraph index
11 years ago
809b4e1fd9
Team added support for URLs with unicode characters in host part to
...
blacklist. Punycode is used to handle unicode characters.
11 years ago
b126b9ba17
add some InputFileStream close at end of reads
...
to make sure file is released
11 years ago
ca7444dbdf
limit filetype nav to known extension also on image/media search
...
- on text search we limit filetype nav already to known extension, apply filter to image search
11 years ago
651d057e93
surrogate import translate dc:language 3-char codes
...
OAI records often use 3-char language codes, start converting some 3-char lang's to the internal ISO639-1 2-char code
11 years ago
22618e3ba2
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
01989f6af9
restrict write buffer size to a limit
11 years ago
d1091e79f8
- added stealth button to navigation menu
...
- more fixes to progress bar
11 years ago
c297de5145
remove check for unused virtual path /currentyacypeer/
...
- del jqueryheader.template (not used)
11 years ago
3c8d6e1eee
added adminAccount switch to ConfigAccounts_p servlet to switch on
...
protection of all pages; some refactoring as well
11 years ago
7d24bcb98d
added flag to require that all web pages, even such without a "_p"
...
extension require authorization. (default off)
11 years ago
7a6658abec
removed synchronization in embedded solr connection (that was probably
...
a mistake?)
11 years ago
a7d4379ef9
fixed shutdown of solr cores in case that more than one local core is to
...
be closed (this happens if webgraph is enabled and the index is dumped
using /IndexControlURLs_p.html
11 years ago
453bfd0f17
removed unused variables and warnings
11 years ago
05655d98df
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
9f02d2c47b
fix: remove link to triplestore in Vocabulary_p (triplestore does not longer exist)
...
- should be investigated in more detail to look for additional implications
Remove "yacyaction" from proxyservlet as it was only needed for removed interaction routines.
11 years ago
81a846ec33
fix: set YaCy CONNECTION_PROP_HOST Header in ProxyServlet to host incl. port
11 years ago
251be9ecfa
remove unused ProxySettings ref. from loader
...
clean unused whois test code
11 years ago
82dc815af9
cleanup: remove unrelated and unused code
11 years ago
85a427ec54
support for multiple sitemaps in robots.txt
11 years ago
a373fb717d
remove more unused from legacy server.http
...
- triggerOnlineAction not used
- useTemplateCache not used
11 years ago
749d020aeb
remove redundant url string manipulation in HTTPDProxyHandler
...
(still used by ProxyServlet)
11 years ago
612294cf84
use servletPath in ProxyServlet instead of fixed name
...
to allow servlet-mapping via web.xml
11 years ago
1d01672bd3
fix DCEntry.getIdentifier
...
on successful url parameter
11 years ago
b08375da33
fix for bad/missing values of size_i
11 years ago
6306d28a6a
OAI import get multivalued keywords (dc:subject)
11 years ago
0a8c8102de
allow YaCy to start w/o ssl if JKS init fails
11 years ago
0b2101c59c
Speed up the ProxyHandler:
...
simplified cache-storing and make it concurrent in order to free the
clientconnection asap
let other prozesses wait on proxy-access like it was bevore
11 years ago
516f8c2489
fix: to allow unix scripts (bin/*.sh) to allways submit http admin apicalls
...
using auth via config hash (legacy requirement)
11 years ago
ea3aa30593
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
dd5bf0b71b
cleanup old reference to HTTPDemon.setAlternativeResolver
...
optimize .yacyh check in AbstractRemoteHandler
11 years ago
51800007c4
- added concurrency to postprocessing of webgraph document
...
- bundeled separate webgraph postprocesing steps into one
11 years ago
5f4a6892c1
enhanced RowSet re-sort limit for small sets
11 years ago
351c2be68d
fix: make sure adminAccount changes made via ConfigAccounts_p are effective immediately
...
force to remove current credentials from knownuser cache
11 years ago
5c9dcc269d
improve OAI-PMH import identifier recognition
...
- find best fittng identifier (url) by checking all given dc:identifier in record (many entries proviede several identifiers)
as identifier is currently a multivalued field use "getParams" in preference of splitting the 1st string by ";"
- add resolve DOI:... identifier via http://dx.doi.org/
11 years ago
0e7d249a69
fixed another shutdown problem (only occurs if webgraph core is enabled)
11 years ago
e485fbd0ce
- let crawl loader jobs die after 10 seconds without new jobs
...
- corrected shutdown order t prevent a deadlock during shutdown
11 years ago
bcd9dd9e1d
enhanced concurrent loading by using a fixed set of concurrent loader
...
processes in favor of throwaway-processes. The control mechanism does
less often report a 'queue full' message to the busy loop which then
does not perform a long busy waiting; instead all requests are queued
and new loader processes are started if necessary up to a given limit
(as set before)
11 years ago
051328271c
bugfix-bugfix
11 years ago
eedcbcd906
bugfix to proxy handler: recognize the own yacyh-host
11 years ago
d68e5ad0c4
NPE fix for Thread name (just commited yesterday, sorry)
11 years ago
6878c90f99
fix: IPv6 INTRANET_PATTERNS for local ip (see http://bugs.yacy.net/view.php?id=378 )
...
requiring following ":" for fc and fd prefix and made pattern match case insesitive
- add some more ipv6 test cases to MultiProtocolURLTest.java
11 years ago
a2e5ea2026
status panel link to set max mem
...
+url proxy same error text as in transparent
11 years ago
6ed9c0164e
attaching names to all Threads to get a better view in profiling tools
...
like VisualVM
11 years ago
fdaeac374a
- enhanced postprocessing speed and memory footprint (by using HashMaps
...
instead of TreeMaps)
- enhanced memory footprint of database indexes (by introduction of
optimize calls)
- optimize calls shrink the amount of used memory for index sets if they
are not changed afterwards any more
11 years ago
ba49ff81ed
little more verbose proxy 403 error message
11 years ago
d325cb8912
fixes and enhancements for postprocessing
11 years ago
7c1b968378
another fix for the shutdown exceptions
11 years ago
133d41386c
(again) full redesign of ConcurrentUpdateSolrConnector to remove
...
out-of-order transactions regarding add and delete operations. Now all
operations (add and delete) are executed concurrently in-order.
11 years ago
Michael Peter Christen