Michael Peter Christen
8dbc80da70
redesign of index.exist-test: this shall now not be done using a single
...
id to be tested, but with a collection of ids. This will cause only a
single call to solr instead of many. The result is a much better
performace when testing the existence of many urls. The effect should
cause very much less IO during index transmission, both on sender and
receiver side.
12 years ago
Michael Peter Christen
44e363f37f
refactoring of WorkflowProcessor, added process counter, update of
...
process counter if an blocking thread dies. Added also a new column in
PerformanceConcurrency_p servlet to show the actual number of concurrent
processes.
12 years ago
orbiter
aeff31cd44
fix for workflow processor (cause: latest redesign for less threads)
12 years ago
orbiter
e1bfe9d07a
- reduction of the concurrently running processes to make YaCy more
...
adjusted to smaller and 1-core devices.
- the workflow processor now starts no process at all. these are started
as soon as parser/condenser/indexing queues are filled.
- better abstraction
12 years ago
Michael Peter Christen
24c9bb35f7
extended the Scheduler: introduced scheduled events
...
- an event type (once, regular) can be selected
- for this event type, a fixed time can be selected. This may be either
directly after startup or at one of the full hours at a day (==25
options)
The main point about this feature is the opportunity to start an action
directly after startup. That makes it possible to create YaCy
distributions which, after started at the first time, start to index
parts of the intranet/internet by itself.
12 years ago
Michael Peter Christen
a33e2742cb
- removed unnecessary synchronized and deadlock in crawler
...
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings
12 years ago
orbiter
276dd6452b
removed warnings
12 years ago
Michael Peter Christen
a8167e6e5b
clean-up: removed unused methods in kelondro
12 years ago
Michael Peter Christen
94a334f128
another fix to the Solr metadata reading process and to the shutdown
...
process
12 years ago
Michael Peter Christen
b51df6c7e8
- added coordinate storage in solr schema
...
- fixed shutdown process
- fixed some solr-to-metadata reading
- added a large number of metadata attributes in ViewFile.html
12 years ago
Michael Peter Christen
8efc1c1078
- fixed a memory leak (or bad usage) during parsing/snippet fetch
...
- more logging for errors
13 years ago
Michael Peter Christen
5bd3c90907
- removed unnecessary semicolons
...
- added default case for switch
13 years ago
Michael Peter Christen
0301aba1e9
removed unused method parameters
13 years ago
Michael Peter Christen
ea10766bfd
cleaned unnecessary nested code
13 years ago
Michael Peter Christen
1481037820
replaced non-generic array with collection
13 years ago
Michael Peter Christen
43c2c6e588
better logging
13 years ago
Roland 'Quix0r' Haeder
d10627d591
More sync in close() methods
...
Conflicts:
source/net/yacy/kelondro/logging/GuiHandler.java
source/net/yacy/kelondro/workflow/InstantBusyThread.java
13 years ago
Michael Peter Christen
49be60a7c8
WorkflowProcess is forced to make small pauses if shortMemoryStatus is
...
reached.
13 years ago
orbiter
8895d8c1cd
removed unnecessary log entries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8117 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
5a55397f99
some last-minute performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
sixcooler
411ed159f8
do some extra sleep while running low on memory
...
(1 sec. per outofmemoryCycle)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7879 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
31283ecd07
- added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search.
...
for example search for "passwd /ftp". This can also be done with /http /https and /smb
- fixed some search throttling processes that should protect your peer against search DoS or strong search load
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7794 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
bd55dcee50
- commented out experimental distributed ranking loading
...
- less threads for blocking threads
- disable all threads for DHT transmission for networks with zero peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7737 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
039126cfaf
better handling of on/off switched solr indexing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7709 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e1b6916423
always try to guess the size of a StringBuilder to prevent too many memory re-allocations
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7572 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
3b40b98256
*) set SVN properties
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7567 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
dc5f0e357c
*) fixed SVN properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6972 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
01d6b952f0
*) minor changes for easier to read code, no functional changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6971 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
55d8e686ea
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6807 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
90dd197ae7
- no latency for local crawls
...
- catch interrupted exception during 'fast' crawls in workflow processor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6759 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
748abfcffa
added patches to prevent yacy-protocol DoS settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6751 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dd459281c8
applied code changes that are recommended by PMD
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6563 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
82d740050f
*) adding more SVN properties
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6541 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8281e29963
- more configuration for profiling graph (number of events)
...
- more logging for a shutdown: print reason and accessing IP into log
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6520 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4a5100789f
replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
491ba6a1ba
- some refactoring in workflow
...
- some refactoring in search process
- fixed image search for json and rss output
- search navigation on bottom of search result page in cases where there are more than 6 results on page
- fixes for number of displayed documents
- disabled pseudostemming
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4431b9767e
added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ce8dc575ca
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6398 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago