e9060d31bd
update to Jetty 9
...
besides adjustments in code it makes the servlet settings in web.xml significant.
This applies to solr, gsa and proxy servlet. There is no longer a default setup in code during init (as jetty 9 checks for double definition).
11 years ago
1432a817dd
respect "index media" switched off in CrawlStartExpert.html
...
fix http://mantis.tokeek.de/view.php?id=64
11 years ago
39e1913585
next development step: migration to java 1.7
...
This includes also a small code change to test generic type inference, a
java 1.7 feature
11 years ago
4e734815e8
enhanced snippets: remove lines which are identical to the title and
...
choose longer versions if possible. Prefer the description part.
11 years ago
e84e07399a
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
89f76da24b
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
390f03e041
o not check for segments-count on optimize:
...
this is also done in Solr and our getSegmentsCount() does not return
up-to-date values
11 years ago
8a7c68e4c7
content of surrogates/out never accessed (remove)
...
After import the conent is never accessed but may take up a lot of disk space,
also the getLoadedOAIServer (which lists the files in surrogate out) is not used.
Making the surrogate.out obsolete. Removed keeping of xmls after import.
11 years ago
b8cee9b7d8
remove tables from tabletracker on close to avoid lots of dead entrys in
...
/PerformanceMemory_p.html
11 years ago
1600414450
fix NPE on continuing crawls after YaCy restart
...
(Agent is then nulll)
11 years ago
229f2248b8
added configuration option for maxmimum load and minimum ram for
...
postprocessing
11 years ago
f15c832587
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
c97da1a0d8
First draft of a blacklist API.
11 years ago
d4f65833a1
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c1c1be8f02
fix for slow crawling and better logging in balancer
11 years ago
3acf416335
npe fix
11 years ago
2eb7682772
add html5 audio/video <source> tag to html content scraper
...
- <source src=.. type=..> tag content is added to embed collection
11 years ago
0b6db04e40
fix contentscraper img height/width parsing
...
prevent numberformat exception on common "100px" property
- include in test case
11 years ago
ffc5b75c73
optimize and fix lat / lon assignment
11 years ago
9313447de2
reimplement tighter lat/lon calc in URIMetadataNode
...
from old MetadataRow, considering http://mantis.tokeek.de/view.php?id=272
11 years ago
d812f80784
add exit proxy link to UrlProxy
...
on proxied pages a link to exit proxy is added to top of page.
Link text can be configured in web.xml init-parameter (see default/web.xml). If missing no link is displayed.
11 years ago
78d08998db
throw MalformedURLException on unknown protocol
...
on other than the supported http https ftp file smb \\ mailto
11 years ago
bb8181b2be
fix: resolve url without path but searchpart
...
e.g. http://yacy.net?q=test was resolved as host "yacy.net?q=test" now host="yacy.net" path="/"
fixes http://mantis.tokeek.de/view.php?id=47
added test case for getHost
11 years ago
a3542f29b4
npe fix
11 years ago
c48d2a2a02
npe fix
11 years ago
121d25be38
recover sax fatal error on OAI-PMH import of xml with entity error
...
this allows to continue loading next resumptionToken even if import file caused sax parser error
fix http://mantis.tokeek.de/view.php?id=63
11 years ago
81dc2aa536
add current css to HTMLResponseWriter to fix metadata view
...
(using css from metas.template except js links)
11 years ago
2fd8a0ead6
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
8e5ce7cd51
fixed a situation where finished crawls had not been detected.
11 years ago
2f63bd0261
enhanced Host Balancer strategy: fair round robin
11 years ago
0c88a32c36
do not apply lazy value instantiation for numeric or boolean values
...
because that is misleading and confusing in case of 0- or false-values
and may cause NPEs in retrieval functions.
11 years ago
8e04030596
in case of short memory, do not cut down robinson peers to 1, just
...
reduce by 50%
11 years ago
86f6975edc
exclude html tags in in/outboundlinks_anchortext_txt parsed text
...
- some outboundlinks_anchortext_txt in index contain e.g. <span>text</span> or more tags,
remove all tags for text property (inline img tags are still parsed)
- added test case for above (to htmlParserTest)
- fix solr test case
11 years ago
ccb1864d55
catch IllegalArgumentException for wrong process types (that is needed
...
for migrations when new process types are introduced or disappear)
11 years ago
4ee4ba1576
fix for NPE in IndexCreateParserErrors_p.html caused by bad handling of
...
lazy value instantiation of 0-value in crawldepth_i
11 years ago
12ba890205
removed warnings
11 years ago
d51f9cc863
add custom Jetty errorhandler
...
to provide custom error page footer line
- remove redundant mime check in UrlProxyServlet
11 years ago
c193a02023
defer creation of new ArrayList after possible early return
...
(to skip not used object allocation)
11 years ago
727dfb5875
refactore URIMetadataNode to further unify interaction with index
...
- URIMetadataNode extending SolrDocument
- use language as stored (String), reducing conversion to string
- optimize debug code in transferIndex
11 years ago
79e7947442
- remove empty http0_9 status text array
...
and unused default_charset = ISO-8859-1
11 years ago
2dabe2009d
- remove unused manual http KeepAlive config
...
(reducing references to obsolete httpdemon)
- add port info to settings_http
11 years ago
5746aae3db
add canonical links to the same crawldepth, not the next crawldepth
11 years ago
74ab5ef9fa
increased runtime for postprocessing query job
11 years ago
8b32dd5f9e
special strategy for balancer: do not remove targets with zero wait time
...
from the queue
11 years ago
9c6228d948
fix for deadlocks in crawler
11 years ago
10cf8215bd
added crawl depth for failed documents
11 years ago
7fefebaeca
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c2f62e783f
- better subgraph handling, less overhead for crawls without the
...
webgraph
- usage of crawler crawldepth cache for the linkgraph target depth
computation
11 years ago
06afb568e2
new Strategies in Balancer:
...
- doublecheck cache now records the crawl depth as well
- doublecheck cache is available from the outside (made static)
- no more need to crawl hosts with lowest depth first, instead all hosts
which have only singleton entries are preferred to reduce the number of
files.
11 years ago
1aea01fe5b
fix for Table in case that requested file does not exist and paths also
...
do not exist
11 years ago
reger