e9eae45b55
simplify rssreader and improve atom feed link extraction
...
- type detection (rss/atom)
- init type parameter overwritten during parse, parameter obsolete
- detection by endtag changed to simpler first-tag evaluation
- channel image not used, removed related extra parser handling
- remove unused code (set/getImage) in rssfeed
- atom link extraction to account for possible multipe link tags
- spec limits link to one with rel="alternate" or one without rel attribute
not accounting for the follwing type & hreflang exception yet:
o atom:entry elements MUST NOT contain more than one atom:link
element with a rel attribute value of "alternate" that has the
same combination of type and hreflang attribute values.
11 years ago
a8508417d1
catch NPE during crawl (OAI import)
...
- condenseDocument mime=null (allowed)
- collectionconfiguration responseheader = null (allowed)
11 years ago
3dde94422f
center searchevent lines on network graph
...
(PerformanceSearch_p.html)
11 years ago
3860711aef
fix for possible interruption of concurrent queries
11 years ago
6344718f8b
reducing the concurrent query stack size and reduced concurrency of
...
postprocessing to avoid OOM situations
11 years ago
eca9380e3d
bugfix for crawler double-check: if an url is redirected, the
...
redirect-target was not double-checked. This is now done by replacing
the redirect-URL on the crawl queue again (where it is double-checked)
11 years ago
9ac0c93f17
fix for subpath crawl filter
11 years ago
66106bdaf0
fix for crawler attribute maxdompages
11 years ago
49d91b94c3
npe fix in crawler
11 years ago
b7183a7321
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
ea2e627662
fix ConfigAccounts del user with uppercase letter in name
...
(usernames are case sensitive, userdb.delete used toLower)
11 years ago
c465b791af
typo
11 years ago
191ec8c82a
added concurrency to postprocess rewrite process
11 years ago
a1e8bdd5e9
log ppm instead of docs/second
11 years ago
cc0ded7abd
set process type of web graph according to fields as defined in the
...
schema
11 years ago
12fb9d7cd1
log postprocessing constraints in case that postprocessing is not
...
performed
11 years ago
3c23b89823
less logging
11 years ago
a0c53174c5
better solr query logging to detect unnecessary sort requests for more
...
performance profiling
11 years ago
338f574bdc
no sorting if http/www unique fields are not demanded (makes query
...
faster) and some code restrucuring
11 years ago
1609763be5
toString fix
11 years ago
b983e68254
more retries, less sleep
11 years ago
1503ba7794
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
8f77719091
fix "Ljava.lang.String" in crawl queue anchor name
...
(e.g. IndexCreateQueues_p.html?stack=LOCAL with images in queue)
11 years ago
0ceeceb35e
more logic on Solr queries; usage of the query terms in posprocessing,
...
saving one query for double document detection now per document
11 years ago
38864ae004
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
4099296b45
added new classes which shall reduce call overhead to Solr (stub)
11 years ago
d0c02e1de7
adjust rss lat/lon to double
...
(common format across other classes)
11 years ago
3491ab4c38
removed unused images from webgraph edge computation
11 years ago
2371d6b8db
target linktexts must be string to enable search facets on these fields
11 years ago
001e05bb80
do not store failure of loading of robots.txt into the index as a fail
...
document
11 years ago
05d58e4df0
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
98f45c9032
fix for image alt attachment to AnchorURLs in html parser.
11 years ago
22ce4fb4dd
better error handling for remote solr queries and exists-checks
11 years ago
9df14fc126
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
477be17c51
Replaced old UPNP library with Weupnp. UPNP should
...
work now, at least it does on my network. UPNP code in YaCy can still
be improved though (see TODO comment: make port on gateway configurable
or find free one).
*) removed old code
*) added new lib
*) changed code to work with new lib
11 years ago
738989aab7
reverted commit f94c91315b because the
...
webgraph has not enough performance for that
11 years ago
e9163e7e10
fix for malformed hostpath names in crawl balancer
11 years ago
c115f3869c
enhanced snippet computation and test method in ViewFile
11 years ago
6c10b59f3e
move bootstrap peers test systems to its test class
...
var assignment not needed elsewhere.
11 years ago
1027f3d04a
fix for the usage of ready-prepared solr queries, some queries are
...
formulated as edismax query but this was not set as query attribut. The
defType=edismax property needs a qf-field, so this was added as well. Do
not remove that field again! This fixes also a problem with title-unique
computation.
11 years ago
f94c91315b
if the webgraph is used, then use it also for reference computation to
...
avoid contradictions with references_i in the collection index.
11 years ago
6e1dc444c3
added a snippet test function in ViewFile: you can now search for a
...
specific word on the document; the servlet returns the snippet in the
same way as it would be shown in a search result.
11 years ago
4b06adb751
fix for file urls
11 years ago
08409ec680
no idea why the words max was an ordered one. This change increaes speed
...
dunring document processin a bit
11 years ago
e5854a5cdb
fix localhost link to opensearchdescription.xml
11 years ago
b44626e55b
fixed target_alt_t in webgraph
11 years ago
504327b15c
fix for condition for writing the webgraph
11 years ago
542c20a597
changed handling of crawl profile field crawlingIfOlder: this should be
...
filled with the date, when the url is recognized as to be outdated. That
field was partly misinterpreted and the time interval was filled in. In
case that all the urls which are in the index shall be treated as
outdated, the field is filled now with Long.MAX_VALUE because then all
crawl dates are before that date and therefore outdated.
11 years ago
4eec1a7452
refactoring (change Metadata name of load time data structure to avoid
...
confusion with Node data which is also called metadata)
11 years ago
c95ba52cf0
improve logexception info
...
- log a message or class name insted of msgtxt "null"
11 years ago
reger