ff8fe7b6a4
fix for ',' or '.' appearing within a word or number. This will not
...
tokenize the query into parts around that character to make it possible
to search for numbers or version numbers.
2 years ago
88cd17ea57
migrated solr from 8.9.0 to 8.11.2; activated also migration script. A YaCy index with solr 8.9.0 will automatically be migrated to 8.11.2. This is a preparation step to migrate to 9.0.0 soon.
2 years ago
1c0f50985c
fixed documentation and some details of handling of keywords
2 years ago
9fcd8f1bda
added canonical filter
...
attention: this is on by default!
(it should do the right thing)
2 years ago
4304e07e6f
crawl profile adoption to new tag valency attribute
2 years ago
309adb814e
fixed import of jsonlist imort from searchlab.eu using a direct URL
2 years ago
62d177bf59
stub for jsonlist index importer web page
2 years ago
efa0425f00
refactoring: moved jsonlist importer to importer class
2 years ago
49daa32a88
yacy can now read searchlab export dump files
...
using the surrogate input process:
- copy the searchlab export file to DATA/SURROGATE/in
- the file is processed automatically and then moved to
DATA/SURROGATE/OUT
2 years ago
99174282d8
try to shut down in a bit more ordered way
...
inspired by https://github.com/yacy/yacy_search_server/issues/518
2 years ago
482f507e65
upgraded solr from 8.8.1 to 8.9.0
...
should hopefully fix
https://github.com/yacy/yacy_search_server/issues/496
because it includes https://issues.apache.org/jira/browse/SOLR-13034
2 years ago
60c9986a0e
new release file names with date and git hash
...
...without reference to 9000ish SVN
2 years ago
9c1bc533fa
removed hazelcast because it is phoning home, see also:
...
https://github.com/yacy/yacy_search_server/issues/504
2 years ago
fc98ca7a9c
removed ContentControl servlet and functinality
...
This was not used at all (as I know) and was blocking a smooth
integration of ivy in the context of an existing JSON parser.
2 years ago
3d138d3fdd
catch error when initializing hazelcast
...
should fix https://github.com/yacy/yacy_search_server/issues/468
3 years ago
a6a9828181
Merge pull request #440 from lfuelling/master
...
Add setting for public facing port
3 years ago
3ced06c731
Various javadoc fixes
3 years ago
6a1e259fd0
Fix NPE in Switchboard . getURL https://github.com/yacy/yacy_search_server/issues/441
3 years ago
e8a00007f6
add setting for public facing port
3 years ago
bd3f2483a1
replaced url and date retrieval by only url retrieval
...
This should prevent that the search index is used for freshnes of the
index entry.
3 years ago
163ba26d90
replaced check for load time method
...
instead of loading the solr document, an index only for the last loading
time was created. This prevents that solr has to fetch from its index
while the index is created. Excessive re-loading of documents while
indexing has shown to produce deadlocks, so this should now be
prevented.
3 years ago
be0aebad84
fixes https://github.com/yacy/yacy_search_server/issues/424
3 years ago
63ad8ce6b2
removed ymarks
...
had not been used since a long time
4 years ago
ef5a71a592
enhanced crawl start response time
...
for very very large crawl start lists
4 years ago
e9c5e78868
replaced new Number(Number) with Number.instanceOf
...
to remove deprecation warnings for Java 9
4 years ago
e81b770f79
enabled crawl starts with very large sets of start urls
...
i.e. 10MB large url list with approx 0.5 million start points
4 years ago
1cdb21592b
added hazelcast and some modifications to align legacy YaCy with
...
YaCyGrid
4 years ago
8f876a8c72
added concurrency to enhance indexing speed during json surrogate import
4 years ago
f8cbaeef93
Merge branch 'master' of git@github.com:yacy/yacy_search_server.git
4 years ago
a857e3d3d5
fix for json importer
4 years ago
c69c462a15
replaces a expensive getLoadTimeURL() by exists()
...
refactors urlExists to getHarvestProcess as that is what it does
4 years ago
26223dc25a
replaces getLoadTime() by exists() with a simpler query
...
since solr-8.8.1 getLoadTime() causes a high cpu usage
4 years ago
8b4394a6c5
fixes for solr 8.8.1 migration
...
- replace new guava 30 with older 25 because that is the correct
dependency for solr 8.8.1. The newer one did actually not work!
- index will be crated in a DATA/INDEX/freeworld/SEGMENTS/solr_8_8_1
subfolder. The older solr_6_6 index is not touched but also not
migrated. The index starts with fresh (empty) content.
- Older indexes must be migrated by hand (export/import) so far until a
better solution is found.
- Large schema adoptions for lucene 8.8.1
4 years ago
69014a701e
Update API Usage
4 years ago
13a2e6dc6e
Merge branch 'master' of https://github.com/yacy/yacy_search_server.git
4 years ago
0ae8ccf657
Make it possible to set an empty password disabling the authentication
...
protocol completely
If you set now an empty password, then the http server will not ask to
authentify. This is required for environment where we attach an outside
authentification service like keycloak or similar using authentication
in an ingress proxy.
This change is part of the approach to run YaCy inside of a kubernetes
cluster where we do not want individual authentication of peers and want
to apply a ingress authentication.
4 years ago
96592a10cf
added option to set yacy configuration values using environment
...
variables
To use that feature, set an environment variable with prefix "yacy." and
suffix identical to the yacy configuration attribute name.
Additionaly we implemented a way to set a peer name using the setting
"network.unit.agent". This can therefore now be used to set a peer name
with the java call parameter
-Dyacy.network.unit.agent=anonymous
The purpose for this feature is the ability to set peer names in
mass-deployed kubernetes clusters to the same name to prevent that we
are flooding peer name statistics with auto-deployment-generated names.
4 years ago
198826c362
added network scanner process to discover all YaCy peers in the intranet
...
this will be used to wire YaCy peers in a kubernetes cluster
4 years ago
907f121d0c
do not overwrite PW with random PW
4 years ago
3e6a1e0a49
fixed surrogate process counter
4 years ago
baad56d83d
beautified default peer names
4 years ago
43a9f4f574
updated solr 6.6.6 -> 7.7.3
...
dropped GSA support (GSA API is still in YaCy Grid)
The 6.6.6 solr index works without migration also with 7.7.3
4 years ago
c0d9a3e9a7
turned HostBrowser into a admin-only page, now called IndexBrowser
...
This was required because spiders and bots crawled through this page and
created load on the peer without use for the user or the YaCy network.
4 years ago
6271e9122c
javadoc fix
4 years ago
52228cb6be
added a gc to cleanup process (once every 10 minutes)
4 years ago
22841ffbf1
creating a threaddump during every cleanup process
...
to be able to find out what a peer did (not) last time before a crash
4 years ago
3431f91db9
removes unused 'unused' tokens
5 years ago
fc03c4b4fe
removes some warning and unused objects
5 years ago
4a495df63a
removes some deprecation-warnings
5 years ago
dd9d4b1188
replace org.junit.Assert.assertThat by
...
org.hamcrest.MatcherAssert.assertThat from hamcrest 2.2 to avoid
deprecation-warning
5 years ago
Michael Peter Christen