1cdc55a425
lets SOLR merge bigger segments (up to 50GB)
...
+ some setting to reduce caches
3 years ago
e7d3a363f2
refactor to use finish()
3 years ago
4fc876f4a3
revert back to use EntityUtils.consumeQuietly - as it simply closes the
...
underlying stream
3 years ago
4f0392e93e
refactor use of AuthSchemeProvider
3 years ago
b74f337859
removes double setting of UserAgent
3 years ago
965748fefb
some refactoring using try with resources
3 years ago
f4834e8e31
link fix
3 years ago
7f5d3e3a12
fixed name
3 years ago
552ab7051b
fix for warc importer
3 years ago
3c86b7b780
attempt to make a Mac Release using gradle
...
This is almost working with many workarounds:
- run rm lib/yacycore.jar
- run ./gradlew clean build bundleNative
- run ant clean all
- run again rm lib/yacycore.jar
- run ./fixMacBuild.sh
The build is then inside build/mac/YaCy.app
Right now this works so far but it does not have the correct release
number inside.
Target is to make this working for Windows releases and to embedd jre
entirely.
3 years ago
49cae8ca62
network bootstraping addresses update
3 years ago
8e4383c49e
downgrading gradle to 6.9
...
to be able to support org.mini2Dx.parcl
3 years ago
999c819e3e
Merge branch 'master' of https://github.com/yacy/yacy_search_server.git
3 years ago
fd770e90e2
spike to identify paths for YaCy within mac application bundles
3 years ago
d19872fd26
making sure that crawl queues are closed correctly to prevent data loss
3 years ago
90507c0fdc
comments out printing query params to std.out
3 years ago
be0aebad84
fixes https://github.com/yacy/yacy_search_server/issues/424
3 years ago
63ad8ce6b2
removed ymarks
...
had not been used since a long time
3 years ago
ef5a71a592
enhanced crawl start response time
...
for very very large crawl start lists
3 years ago
1bab4ffe20
calculating the correct size of an export.
...
This can be seen as a fix for
https://github.com/yacy/yacy_search_server/issues/343
however, the export was not flawed, it is just the impression that
something is wrong, but the export size must be smaller than the index
size because the index also containers error documents.
Now an information line is presented that shows i.e.:
"The local index currently contains 181,319 documents, only 106,887
exportable with status code 200 - the remaining are error documents."
3 years ago
4cadd557dc
removed synchronization in table creation
...
to avoid possible deadlocks when handling OnDemandOpenFileIndex
which happens quite often during wide crawling
3 years ago
8084960392
disabled citation index
...
that was created but never used
3 years ago
9b7668fa58
reduced memory footprint during indexing/crawling
3 years ago
fbf8ddd32d
upgrade of jsoup 1.12.1 -> 1.14.2
3 years ago
53518a91ab
In case of reload404, load only failed documents
3 years ago
4c889b7ff9
fixed build paths
3 years ago
683cac125f
updated bouncy castle 1.60 -> 1.69
3 years ago
e6a87e0426
enhanced crawler
...
a main problem when crawling is long waiting time cuased by crawl-delay
values from robots.txt entries. that attribute is not supported by
google and interpreted by yandex and bing in different ways. In large
crawls there is always one host which blocks the whole crawl with
extreme large values. YaCy now still obeys crawl-delay but limits them
to 10 seconds.
Additionally the blocking logic when loading new robots.txt was analyzed
and a deadlock was removed. Furthermore the construction of new queue
lists was redesigned and it was ensured that always a large list of
different hosts for host-balancing is provided for the loader.
3 years ago
e9c5e78868
replaced new Number(Number) with Number.instanceOf
...
to remove deprecation warnings for Java 9
3 years ago
9e13d77de4
removed call to class.finalize() because of deprecation in java 9
...
next: removal of finalize() implementation
after testing with assert false
3 years ago
9ef4503672
fixed some newInstance() warnings
...
.. by adding .getDeclaredConstructor()
3 years ago
82df012442
removed old lib
3 years ago
8a2adb2b15
upgraded commons-compress lib
...
cause: alert in
https://github.com/yacy/yacy_search_server/security/dependabot/pom.xml/org.apache.commons:commons-compress/open
3 years ago
9182b3dfca
enhanced default value
3 years ago
294d56d4a2
addressing better GC behavior after removing Xms with earlier heap increase strategy
3 years ago
3959d43a5c
fixed doku link
3 years ago
c4659f0fb0
removed Debian and Red Hat build process
...
as announced in
https://twitter.com/yacy_search/status/1414608643241152516
because of lack of community support for these kind of
distributions. We will still support
tarball, Windows, Mac and Docker releases.
3 years ago
73360ed52b
add gradle to gitignore
3 years ago
15b7461bc7
removed Xms java memory startup parameter
...
We will use the default value for now on.
This is much better for resource economy and fits better into a
container/docker/kubernetes strategy.
Furthermore, a small memory footprint is essential for the usage on
small devices like RaspberryPi.
3 years ago
c3b3087077
gradle cleanup
3 years ago
a13986d659
replaced maven with gradle
3 years ago
1d41380f0a
better support for mac-specific tray functions in java 9
3 years ago
4377bd2b70
fix for wrong crawlName construction
3 years ago
e81b770f79
enabled crawl starts with very large sets of start urls
...
i.e. 10MB large url list with approx 0.5 million start points
3 years ago
4b73b3f9f2
docker has no latest-alpine
...
There is no yacy/yacy_search_server:latest-alpine on docker hub
3 years ago
c623a3252e
fix for jdk 14 bug
4 years ago
dbd211a1ad
removed/replaced reflection in memory tool
4 years ago
160f00e59e
removed reconfigure script which is seven years old any may not up to
...
standards of current password implementation.
See https://github.com/yacy/yacy_search_server/issues/409 as hint
4 years ago
1cdb21592b
added hazelcast and some modifications to align legacy YaCy with
...
YaCyGrid
4 years ago
42ea2a1c6f
Merge pull request #405 from jfhs/jfhs/support-all-html-entities
...
Improve HTML entities support
4 years ago
sgaebel