d097a642c2
Merge pull request #615 from okybaca/logging2
...
Logging unclutter
12 months ago
6d5e9ff53f
Merge pull request #616 from okybaca/logging3
...
changed the log entry REJECTED to CRAWLER * REJECTED, loglevel fine
12 months ago
dfb2b79609
Add setting for DHT receive loadprereq insted of hardcoded load < 2.0
12 months ago
5dee8dbcbd
changed the log entry REJECTED to CRAWLER * REJECTED, loglevel fine
12 months ago
4c603e23f0
Merge pull request #610 from okybaca/cr-text
...
UI: added a more descriptive message, CitationRank instead of cr
12 months ago
7831f294a9
changed regular peerping messages to level fine
12 months ago
553c859703
logging: moved some log-cluttering DHT messages to level 'fine'
12 months ago
1c5fca9a58
changed network operation log category from YACY to NETWORK
12 months ago
3d3bdb0f5f
added zim importer rule for mdwiki
1 year ago
4a611ac6a3
another possible fix for
...
https://github.com/yacy/yacy_search_server/issues/500
1 year ago
d72cd7916c
Merge branch 'master' of https://github.com/yacy/yacy_search_server
1 year ago
0663ae3c99
adds synchornized dumplog
1 year ago
cba84632ee
UI: added a more descriptive message, CitationRank instead of cr
1 year ago
cff0991d85
test if this is helpful for https://github.com/yacy/yacy_search_server/issues/500
1 year ago
ceb07a5218
fixed problem with zim importer which crashed when non-valid urls appeared
1 year ago
3268a93019
added a 'minified' option to YaCy dumps
1 year ago
c20c4b8a21
modified export: added maximum number of docs per chunk
...
The export file can now be many files, called chunks.
By default still only one chunk is exported.
This function is required in case that the exported files shall be
imported to an elasticsearch/opensearch index. The bulk import function
of elasticsearch/opensearch is limited to 100MB. To make it possible to
import YaCy files, those must be splitted into chunks. Right now we
cannot estimate the chunk size as bytes, only as number of documents.
The user must do experiments to find out the optimum chunk max size,
like 50000 docs per chunk. Try this as first attempt.
1 year ago
24011dcbcc
more file name extensions for json list surrogate files
1 year ago
34a9fc1a07
bugfixes to zim reader:
1 year ago
7db0534d8a
Added a zim parser to the surrogate import option.
...
You can now import zim files into YaCy by simply moving them
to the DATA/SURROGATE/IN folder. They will be fetched and after
parsing moved to DATA/SURROGATE/OUT.
There are exceptions where the parser is not able to identify the
original URL of the documents in the zim file. In that case the file
is simply ignored.
This commit also carries an important fix to the pdf parser and an
increase of the maximum parsing speed to 60000 PPM which should make it
possible to index up to 1000 files in one second.
1 year ago
70e29937ef
added a check in zim importer which tests if import URLs actually exist
1 year ago
fdc6311dc7
added parsing rules for wikibooks and wikinews in zim reader
1 year ago
53b01dbf2e
Merge branch 'master' of https://github.com/yacy/yacy_search_server.git
1 year ago
1c0df28bfb
added a zim importer that can be used for surrogate imports.
...
Can not be used yet because it requires some security additions
to verify that the given urls actually work.
1 year ago
4add1f6bc7
replaced all the links to legacy legacy wiki to legacy wiki
1 year ago
4a54b24703
fix for "negative seek offset" error during extension of heap files.
...
This would have always happend when a heap file exceeds 2GB.
should fix https://github.com/yacy/yacy_search_server/issues/372
1 year ago
5ba5fb5d23
upgraded pdfbox to 3.0.0
1 year ago
4308aa5415
removed concept of empty passwords as "no passwords used",
...
because we now start YaCy with a default password (yacy).
This has impact of all function that check the current state of
password-protection that included the empty password situation,
including the warnings to set a password in case that none is set (which
cannot be the case any more).
1 year ago
2c60ff14bb
fixed default pw comparison
1 year ago
4da320bebf
added a warning message in ConfigBasic in case that the default password
...
was not changed.
1 year ago
7830268be1
fix 756c817b5a
...
must be applied to all code where a transaction token is generated.
1 year ago
756c817b5a
fix for https://github.com/yacy/yacy_search_server/issues/544
1 year ago
03bf259601
fix for https://github.com/yacy/yacy_search_server/issues/363
...
We still need to set the load in the process because a demand for higher
crawl speed may require to increase the maximum load limit. However,
following the criticism in the bug, we do never reduce the load limit
again.
1 year ago
8fc51f66c6
fixed a test class which prevented compilation on latest jvm
1 year ago
53bafa1544
consistent formatting in string concatenation
1 year ago
22c4188001
additionally match release stub for YaCy version
1 year ago
ff8fe7b6a4
fix for ',' or '.' appearing within a word or number. This will not
...
tokenize the query into parts around that character to make it possible
to search for numbers or version numbers.
1 year ago
0689f4f0ae
Check if the character is a minus sign and is followed by a letter or a
...
digit. Treat it as part of the word/number.
1 year ago
5db97a8928
parser can now separate numbers from words also when they are not
...
separated by space, i.e. 4.7Ohm
1 year ago
e3797de7de
enhanced the word tokenizer to recognize numbers in a proper way
1 year ago
88cd17ea57
migrated solr from 8.9.0 to 8.11.2; activated also migration script. A YaCy index with solr 8.9.0 will automatically be migrated to 8.11.2. This is a preparation step to migrate to 9.0.0 soon.
1 year ago
0089f234f4
added npe protection
1 year ago
8285fe715a
tab to spaces for classes supporting the condenser.
...
This is a preparation step to make changes in condenser and parser more
visible; no functional changes so far.
1 year ago
195bd2e444
extended the maximum header size to 16k to prevent http error 431
1 year ago
92dad3ed49
removed 7Zip parser because the old library could not be replaced by a maven repository
1 year ago
5afcba162b
updated libraries
1 year ago
a348146d8f
setting connect host to 0.0.0.0
1 year ago
1c0f50985c
fixed documentation and some details of handling of keywords
2 years ago
3472bcb4d3
patched a 'java.lang.NoSuchMethodError: com.twelvemonkeys.imageio.util.IIOUtil.lookupProviderByName' problem which occurred only on ARM
2 years ago
f7b6e98ed7
Merge pull request #562 from thkoch2001/fix-warnings
...
Fix warnings
2 years ago
Michael Christen