f0db501630
better handling of ranking parameters and new default values for date
...
navigation which is done using ranking in solr.
11 years ago
53948da7d0
tried to make last_modified recognition smarter
11 years ago
2d03037965
'Last-Modified', not 'Last-modified' according to
...
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
11 years ago
3dc5fb0050
fix for operator precedence bug (cast binds stronger than bitwise AND)
...
in peer hash hashing. This should not change anything if java casts long
to int by masking with 0xFFFFFFFFL but you never know. The important
thing is, that the hashCode() should not return numbers that have the
same order as the hash code order because hashing of seeds is used to
remove the order in some places.
11 years ago
6634b5b737
debug code for index distribution testing
11 years ago
49e344e8d9
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
7705e36703
fix for latest generic warning fix
11 years ago
10326892a8
avoid erros from ConnectHandler, correction for #6d16fa9
11 years ago
97983ba89f
fixed generics warnings for generic array instantiation that appeared
...
after migration to Java 7
11 years ago
830057d788
lower Segment-size (hope to get Segments of 10GB)
...
see:
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5216&p=30036#p30034
11 years ago
c028ae9b09
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
e31493e139
"Use remote proxy for yacy" has no function, remove option and related config item
...
see/fix bug http://mantis.tokeek.de/view.php?id=23
http://mantis.tokeek.de/view.php?id=189
11 years ago
181784a5cb
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
0587077d06
cleanup obsolete and not used serverswitch Authentify code
...
as auth is mostly delegated to Jetty container.
11 years ago
c9f66be20b
move unnecessary nested else out of condition
11 years ago
0d8072aa99
removed warnings
11 years ago
88f4af90da
removed warnings
11 years ago
0f425e01ca
another circle computation enhancement
11 years ago
a8d162810c
Exclude = from percent-encoding in MultiProtocolURL
...
fix http://mantis.tokeek.de/view.php?id=185 and http://mantis.tokeek.de/view.php?id=280
11 years ago
024f8e9b33
fix truncated urls containing ","
...
adressing http://mantis.tokeek.de/view.php?id=58
Exclude comma from percent-encoding in MultiProtocolURL (see RFC 1738 2.2 and RFC 3986 2.2)
11 years ago
9112f0a2df
enhanced circle tool initialization
11 years ago
a1ac4c3b76
automatically clear graphics cache
11 years ago
505f58c79c
enhanced circle computation time and memory footprint
11 years ago
cd8c0dbda9
assign serialVersionUID for proxyservlet, too.
11 years ago
b300d7f4ce
set serialVersionUID on urlproxyservlet to skip compiler warning
...
- remove commented out code
11 years ago
e9060d31bd
update to Jetty 9
...
besides adjustments in code it makes the servlet settings in web.xml significant.
This applies to solr, gsa and proxy servlet. There is no longer a default setup in code during init (as jetty 9 checks for double definition).
11 years ago
1432a817dd
respect "index media" switched off in CrawlStartExpert.html
...
fix http://mantis.tokeek.de/view.php?id=64
11 years ago
39e1913585
next development step: migration to java 1.7
...
This includes also a small code change to test generic type inference, a
java 1.7 feature
11 years ago
4e734815e8
enhanced snippets: remove lines which are identical to the title and
...
choose longer versions if possible. Prefer the description part.
11 years ago
e84e07399a
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
89f76da24b
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
390f03e041
o not check for segments-count on optimize:
...
this is also done in Solr and our getSegmentsCount() does not return
up-to-date values
11 years ago
8a7c68e4c7
content of surrogates/out never accessed (remove)
...
After import the conent is never accessed but may take up a lot of disk space,
also the getLoadedOAIServer (which lists the files in surrogate out) is not used.
Making the surrogate.out obsolete. Removed keeping of xmls after import.
11 years ago
b8cee9b7d8
remove tables from tabletracker on close to avoid lots of dead entrys in
...
/PerformanceMemory_p.html
11 years ago
1600414450
fix NPE on continuing crawls after YaCy restart
...
(Agent is then nulll)
11 years ago
229f2248b8
added configuration option for maxmimum load and minimum ram for
...
postprocessing
11 years ago
f15c832587
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
c97da1a0d8
First draft of a blacklist API.
11 years ago
d4f65833a1
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c1c1be8f02
fix for slow crawling and better logging in balancer
11 years ago
3acf416335
npe fix
11 years ago
2eb7682772
add html5 audio/video <source> tag to html content scraper
...
- <source src=.. type=..> tag content is added to embed collection
11 years ago
0b6db04e40
fix contentscraper img height/width parsing
...
prevent numberformat exception on common "100px" property
- include in test case
11 years ago
ffc5b75c73
optimize and fix lat / lon assignment
11 years ago
9313447de2
reimplement tighter lat/lon calc in URIMetadataNode
...
from old MetadataRow, considering http://mantis.tokeek.de/view.php?id=272
11 years ago
d812f80784
add exit proxy link to UrlProxy
...
on proxied pages a link to exit proxy is added to top of page.
Link text can be configured in web.xml init-parameter (see default/web.xml). If missing no link is displayed.
11 years ago
78d08998db
throw MalformedURLException on unknown protocol
...
on other than the supported http https ftp file smb \\ mailto
11 years ago
bb8181b2be
fix: resolve url without path but searchpart
...
e.g. http://yacy.net?q=test was resolved as host "yacy.net?q=test" now host="yacy.net" path="/"
fixes http://mantis.tokeek.de/view.php?id=47
added test case for getHost
11 years ago
a3542f29b4
npe fix
11 years ago
c48d2a2a02
npe fix
11 years ago
121d25be38
recover sax fatal error on OAI-PMH import of xml with entity error
...
this allows to continue loading next resumptionToken even if import file caused sax parser error
fix http://mantis.tokeek.de/view.php?id=63
11 years ago
81dc2aa536
add current css to HTMLResponseWriter to fix metadata view
...
(using css from metas.template except js links)
11 years ago
2fd8a0ead6
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
8e5ce7cd51
fixed a situation where finished crawls had not been detected.
11 years ago
2f63bd0261
enhanced Host Balancer strategy: fair round robin
11 years ago
0c88a32c36
do not apply lazy value instantiation for numeric or boolean values
...
because that is misleading and confusing in case of 0- or false-values
and may cause NPEs in retrieval functions.
11 years ago
8e04030596
in case of short memory, do not cut down robinson peers to 1, just
...
reduce by 50%
11 years ago
86f6975edc
exclude html tags in in/outboundlinks_anchortext_txt parsed text
...
- some outboundlinks_anchortext_txt in index contain e.g. <span>text</span> or more tags,
remove all tags for text property (inline img tags are still parsed)
- added test case for above (to htmlParserTest)
- fix solr test case
11 years ago
ccb1864d55
catch IllegalArgumentException for wrong process types (that is needed
...
for migrations when new process types are introduced or disappear)
11 years ago
4ee4ba1576
fix for NPE in IndexCreateParserErrors_p.html caused by bad handling of
...
lazy value instantiation of 0-value in crawldepth_i
11 years ago
12ba890205
removed warnings
11 years ago
d51f9cc863
add custom Jetty errorhandler
...
to provide custom error page footer line
- remove redundant mime check in UrlProxyServlet
11 years ago
c193a02023
defer creation of new ArrayList after possible early return
...
(to skip not used object allocation)
11 years ago
727dfb5875
refactore URIMetadataNode to further unify interaction with index
...
- URIMetadataNode extending SolrDocument
- use language as stored (String), reducing conversion to string
- optimize debug code in transferIndex
11 years ago
79e7947442
- remove empty http0_9 status text array
...
and unused default_charset = ISO-8859-1
11 years ago
2dabe2009d
- remove unused manual http KeepAlive config
...
(reducing references to obsolete httpdemon)
- add port info to settings_http
11 years ago
5746aae3db
add canonical links to the same crawldepth, not the next crawldepth
11 years ago
74ab5ef9fa
increased runtime for postprocessing query job
11 years ago
8b32dd5f9e
special strategy for balancer: do not remove targets with zero wait time
...
from the queue
11 years ago
9c6228d948
fix for deadlocks in crawler
11 years ago
10cf8215bd
added crawl depth for failed documents
11 years ago
7fefebaeca
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c2f62e783f
- better subgraph handling, less overhead for crawls without the
...
webgraph
- usage of crawler crawldepth cache for the linkgraph target depth
computation
11 years ago
06afb568e2
new Strategies in Balancer:
...
- doublecheck cache now records the crawl depth as well
- doublecheck cache is available from the outside (made static)
- no more need to crawl hosts with lowest depth first, instead all hosts
which have only singleton entries are preferred to reduce the number of
files.
11 years ago
1aea01fe5b
fix for Table in case that requested file does not exist and paths also
...
do not exist
11 years ago
710054bb37
implement gzip input handling directly in defaultservlet
...
(making reference to legacy httpdemon obsolete)
11 years ago
9a5ab4e2c1
removed clickdepth_i field and related postprocessing. This information
...
is now available in the crawldepth_i field which is identical to
clickdepth_i because of a specific crawler strategy.
11 years ago
da86f150ab
- added a new Crawler Balancer: HostBalancer and HostQueues:
...
This organizes all urls to be loaded in separate queues for each host.
Each host separates the crawl depth into it's own queue. The primary
rule for urls taken from any queue is, that the crawl depth is minimal.
This produces a crawl depth which is identical to the clickdepth.
Furthermorem the crawl is able to create a much better balancing over
all hosts which is fair to all hosts that are in the queue.
This process will create a very large number of files for wide crawls in
the QUEUES folder: for each host a directory, for each crawl depth a
file inside the directory. A crawl with maxdepth = 4 will be able to
create 10.000s of files. To be able to use that many file readers, it
was necessary to implement a new index data structure which opens the
file only if an access is wanted (OnDemandOpenFileIndex). The usage of
such on-demand file reader shall prevent that the number of file
pointers is over the system limit, which is usually about 10.000 open
files. Some parts of YaCy had to be adopted to handle the crawl depth
number correctly. The logging and the IndexCreateQueues servlet had to
be adopted to show the crawl queues differently, because the host name
is attached to the port on the host to differentiate between http,
https, and ftp services.
11 years ago
075b6f9278
refactoring of the crawl balancer: the balancer is turned into an
...
interface and the old balancer class is moved into LegacyBalancer to
make room for a fresh implementation of a crawl balancer.
11 years ago
8470dfe3f8
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
46016fa153
autoupdate fails to download latest release (1.71) due to default release blacklist
...
- removed the default version blacklist regex from init (for future versions)
!!! left existing update blacklist setting untouched !!!
(existing installation wanting autoupdate for 1.71 need to change blacklist in ConfigUpdate_p.html)
- moved old blacklist patch to migration.java
11 years ago
8aeef73d49
fix for virtual root nodes
11 years ago
7c7fbb9818
find depth-matches also for edge targets
11 years ago
dd12dd392f
introduction of a data structure for HyperlinkEdges which should use
...
less memory as it does no double-storage of source links for each edge
of the graph.
11 years ago
6ea8bb7348
using MultiProtocolURL for edge data which is faster (hash computation
...
is now much easier) and smaller in size
11 years ago
b21c208b4d
enhanced hashcode computation for MultiProtocolURL
11 years ago
ce1d1b2fa0
fix for maximum tag length in parser
11 years ago
17e0956312
refactoring of SystemLoad calls (only one backend tool)
11 years ago
a37d067692
refactoring
11 years ago
95780eed32
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
67beef657f
strong redesign of html parser: object recursion is now made using a
...
stack on html tag objects, not using a recursive parse-again method
which may cause bad performance and huge memory allocation. The new
method also produced better parsed image objects with exact anchor text
references.
11 years ago
6bd8c6f195
fix for wrong status codes of error pages
11 years ago
9e503b3376
also delete the robots.txt file from the cache when a new crawl is
...
started
11 years ago
67501c9dda
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
1c21b3256d
fix for robots.txt handling: delete old entry before starting a new
...
crawl.
11 years ago
c250fac9f4
linkstructure refactoring to get more options for clickdepth analysis
11 years ago
8068e68474
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
bd886054cb
new structure and enhancements for link graph computation:
...
- added order option to solr queries to be able to retrieve document
lists in specific order, here: link length
- added HyperlinkEdge class which manages the link structure
- integrated the HyperlinkEdge class into clickdepth computation
- extended the linkstructure.json servlet to show also the clickdepth
and other statistic information
11 years ago
f326a67561
fix: typo in default charset in metadata2solr
...
update pom and NB build to Solr 4.7.1 libs
11 years ago
df138084c0
do solr optimization independently from memory and load constraints:
...
- not doing an optimization will likely cause a too many files exception
- without optimization performance will be even worse which would
prevent optimization in the future as well (prevent a deadlock
situation)
11 years ago
ebd44a7080
replaced solr 4.6.1 with solr 4.7.1 and added index migration to
...
lucene_47
11 years ago
734778c0c8
fixed a time-out problem in the default servlet which is also a logging
...
problem because the error log showed the wrong reason (file not found)
instead the actual reason (time-out).
11 years ago
466d90ad42
fixed a problem with resource observer; probably coming from uncatched
...
exceptions within the apache library which appear only in concurrency
environments.
11 years ago
e8ddd415a8
enhanced the new link structure graph
11 years ago
926d28dd3f
fixed a bug which prevented crawl starts after a network switch
11 years ago
3ce8eff21b
another fix for inbound/outbound detection
11 years ago
d4b5c457e4
NPE fix
11 years ago
36a66b0704
fix for parsing of numeric value in case that boolean values are given
11 years ago
41730c8048
better logging in template engine: shows filename of servlets where
...
errors in templates occur
11 years ago
3c1274057d
fixed thread dump in case of wrong seeds
11 years ago
18f9c40302
moved Edge class out of linkstructure servlet as this does not work on
...
non-eclipse driven environments (all non-dev cases)
11 years ago
de95e5e524
reduced search activity corona strength in network image
11 years ago
da413af664
move baseurl after parsing orig source in urlproxyservlet
...
to calculate absolute href links for rewrite from unmodified source.
11 years ago
af6ad20728
fix: remove obsolete ref to yacy.home
...
(use Switchboard instead)
11 years ago
74ab094587
fix for solr query size; too many documents had been retrieved in case
...
that less than _pagesize_ had been requested.
11 years ago
c64c10ef00
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
48fbfa60c1
bugfix to inbound/outbound identification
11 years ago
227c42bc96
eleminate obsolete URIMetaDataRow class
...
by joining it with/into URIMetaDataNode.
11 years ago
cca851a417
introduced new solr field crawldepth_i which records the crawl depth of
...
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.
11 years ago
b1ba764d81
fix for first start options and added german translation for popup texts
11 years ago
429a874222
- added COLS field in GSA response (non-gsa standard by customer
...
request)
- updated document link in GSA response writer
11 years ago
1b9ec9a1c5
- added popover to p2p/stealth mode button to explain the peer mode and
...
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts
11 years ago
62a36fa584
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c9f92abddc
fix: application link count
...
(URIMetadataNode)
11 years ago
a267c46e1a
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
5b83887da8
npe fix
11 years ago
63c9fcf3e0
free configuration of postprocessing clickdepth maximum depth and time
11 years ago
39b641d6cd
added tutorial mode - some menu items will only appear if you 'qualify'
...
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.
11 years ago
f06775850f
fix receiving DHT / parse pultipart
...
+ another close to fix possible resource leak warning
11 years ago
49e76a1c55
make use of detected charset in htmlParser if none is given.
11 years ago
e11504309f
adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html)
11 years ago
b12200cafe
alternative UrlProxyServlet (for /proxy.html) using different url rewrite rules
...
- use JSoup parser for selective rewrite of html body <a href= links only,
instead of regex which rewrites also header href/src links
- this improves display of pages which use header <base> tag
- tags with src attribute are taken from original location (like css) improving display and are not routed trough the indexer
Disadvantage: scripting links will drop out of proxy
Setting of the servlet through web.xml exclusivly (in case one would like to quickly switch back to the YaCyProxyServlet,
leaving the existing code of YaCyProxyServlet untouched available)
11 years ago
2953ebe701
fix: port in local target adress
...
& button style
11 years ago
fda591695c
fixed visibility of custom icon
11 years ago
a9b9950d7f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
b488f33975
added close to fix possible resource leak warning
11 years ago
56710ecb26
prevent opening of new files as that could be a cause for the latest
...
too-many-open-files exception. The old file is just truncated if the
table is cleaned.
11 years ago
8b44fcf0f4
added missing @Override annotation
11 years ago
d7055904a6
fix: proxyservlet path header setting
11 years ago
e515dd460d
added linkscount_i and linksnofollowcount_i to the default solr schema
11 years ago
1a764135be
one more Thread Dump fix for new bootstrap css style
11 years ago
bb21d825f9
fix for thread dump line spacing
11 years ago
cbdfef7ce1
changed protocol facet to show also all other counts if one facet is
...
selected
11 years ago
b9056ef2db
remove unused private header entries (HeaderFramework)
...
X_YACY_ORIGINAL_REQUEST_LINE
X_YACY_KEEP_ALIVE_REQUEST_COUNT
CONNECTION_PROP_REQUESTLINE
11 years ago
6d16fa993d
make transparent proxy handle https-connections:
...
the implemented handle for connect did not work for me - so lets try the
connectHandler
11 years ago
61ad194065
fix for source and target clickdepth in webgraph index
11 years ago
809b4e1fd9
Team added support for URLs with unicode characters in host part to
...
blacklist. Punycode is used to handle unicode characters.
11 years ago
b126b9ba17
add some InputFileStream close at end of reads
...
to make sure file is released
11 years ago
ca7444dbdf
limit filetype nav to known extension also on image/media search
...
- on text search we limit filetype nav already to known extension, apply filter to image search
11 years ago
651d057e93
surrogate import translate dc:language 3-char codes
...
OAI records often use 3-char language codes, start converting some 3-char lang's to the internal ISO639-1 2-char code
11 years ago
Michael Peter Christen