1fdcc2d67b
change seedfile upload ip check to allow intranet ip in intranet mode
...
- this allows to setup a principal peer in intranet environment
11 years ago
e31b0e6d67
- update javadoc Seed.getIP
...
- default mySeed.ip to hostip in SeedDB.initMySeed() if Intranetmode
this allows to become senior status in intranet hosted search network with view peers,
otherwise peer would stay junior because of default init with loopback ip as public (dna) ip.
11 years ago
350c6b8250
in IntranetMode allow intranet hosted seedlist with Network_Domain "any"
...
- so far intranet seedlist hosts are always denied but need to be allowed in intranet mode
11 years ago
d68438c3d9
make sure that the postprocessing background thread never dies by any
...
exception
11 years ago
b4f2a1db6e
added a unlock icon for all protected pages that are unlocked because
...
the administrator is logged in.
11 years ago
ea6c9e9b07
reduce mem buffer overhead for gap files during r/w
...
(they are typically small compared to idx allowing to use smaller buffersize -> set to 16k records)
11 years ago
e88537522d
allow single quote " ' " in query
...
see http://mantis.tokeek.de/view.php?id=379
-add QueryGoal test case for this
11 years ago
487021fb0a
snippet computation update
11 years ago
1c2f1f233a
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
5a4995ded3
fill solr rss writer dc:subject tag with keyword content
11 years ago
927aaa95a6
concurrency bugfix
11 years ago
c9e593cf78
removed warnings
11 years ago
7584352e7b
use more predefined Solr query parameter constants
...
- use CommonParams and DisMaxParams constants
- fix typo in get sort parameter
- getDocumentCountByParams redundant implementation and risk of not optimized call (row parameter unspecified) -> as only used from getCountByQuery removed from interface
11 years ago
f9db5dd6c5
reduce doublecontent check document (prevent out of memory)
...
see http://mantis.tokeek.de/view.php?id=437
test result (concurrency=7)
2000 docs = eom always
1000 docs = eom always
100 docs = eom never
chosen -> 200 docs (eom not encountered during test with 1GB mem setting)
11 years ago
e9eae45b55
simplify rssreader and improve atom feed link extraction
...
- type detection (rss/atom)
- init type parameter overwritten during parse, parameter obsolete
- detection by endtag changed to simpler first-tag evaluation
- channel image not used, removed related extra parser handling
- remove unused code (set/getImage) in rssfeed
- atom link extraction to account for possible multipe link tags
- spec limits link to one with rel="alternate" or one without rel attribute
not accounting for the follwing type & hreflang exception yet:
o atom:entry elements MUST NOT contain more than one atom:link
element with a rel attribute value of "alternate" that has the
same combination of type and hreflang attribute values.
11 years ago
a8508417d1
catch NPE during crawl (OAI import)
...
- condenseDocument mime=null (allowed)
- collectionconfiguration responseheader = null (allowed)
11 years ago
3dde94422f
center searchevent lines on network graph
...
(PerformanceSearch_p.html)
11 years ago
3860711aef
fix for possible interruption of concurrent queries
11 years ago
6344718f8b
reducing the concurrent query stack size and reduced concurrency of
...
postprocessing to avoid OOM situations
11 years ago
eca9380e3d
bugfix for crawler double-check: if an url is redirected, the
...
redirect-target was not double-checked. This is now done by replacing
the redirect-URL on the crawl queue again (where it is double-checked)
11 years ago
9ac0c93f17
fix for subpath crawl filter
11 years ago
66106bdaf0
fix for crawler attribute maxdompages
11 years ago
49d91b94c3
npe fix in crawler
11 years ago
b7183a7321
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
ea2e627662
fix ConfigAccounts del user with uppercase letter in name
...
(usernames are case sensitive, userdb.delete used toLower)
11 years ago
c465b791af
typo
11 years ago
191ec8c82a
added concurrency to postprocess rewrite process
11 years ago
a1e8bdd5e9
log ppm instead of docs/second
11 years ago
cc0ded7abd
set process type of web graph according to fields as defined in the
...
schema
11 years ago
12fb9d7cd1
log postprocessing constraints in case that postprocessing is not
...
performed
11 years ago
3c23b89823
less logging
11 years ago
a0c53174c5
better solr query logging to detect unnecessary sort requests for more
...
performance profiling
11 years ago
338f574bdc
no sorting if http/www unique fields are not demanded (makes query
...
faster) and some code restrucuring
11 years ago
1609763be5
toString fix
11 years ago
b983e68254
more retries, less sleep
11 years ago
1503ba7794
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
8f77719091
fix "Ljava.lang.String" in crawl queue anchor name
...
(e.g. IndexCreateQueues_p.html?stack=LOCAL with images in queue)
11 years ago
0ceeceb35e
more logic on Solr queries; usage of the query terms in posprocessing,
...
saving one query for double document detection now per document
11 years ago
38864ae004
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
4099296b45
added new classes which shall reduce call overhead to Solr (stub)
11 years ago
d0c02e1de7
adjust rss lat/lon to double
...
(common format across other classes)
11 years ago
3491ab4c38
removed unused images from webgraph edge computation
11 years ago
2371d6b8db
target linktexts must be string to enable search facets on these fields
11 years ago
001e05bb80
do not store failure of loading of robots.txt into the index as a fail
...
document
11 years ago
05d58e4df0
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
98f45c9032
fix for image alt attachment to AnchorURLs in html parser.
11 years ago
22ce4fb4dd
better error handling for remote solr queries and exists-checks
11 years ago
9df14fc126
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
477be17c51
Replaced old UPNP library with Weupnp. UPNP should
...
work now, at least it does on my network. UPNP code in YaCy can still
be improved though (see TODO comment: make port on gateway configurable
or find free one).
*) removed old code
*) added new lib
*) changed code to work with new lib
11 years ago
738989aab7
reverted commit f94c91315b because the
...
webgraph has not enough performance for that
11 years ago
reger