orbiter
65f383e70b
some adjustments to the httpc after testing with a very slow httpd
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6831 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5ab5ac80fe
fix for NPE in TextParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6830 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b95ae2518b
fix for assert
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6829 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3247f0e901
fix for deadlocks caused by self-blocking access to TreeMap in concurrent environments. The TreeMap was replaced by a ConcurrentHashMap and additional care that the strings are compared all in lowercase
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6828 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
027b971bde
fix for concurrent quicksort: catch jobs from ThreadPoolExecutor that had been rejected because of full processing queues.
...
Non-catched jobs may have been the cause for blockings and freezes in case of overloading during strong processing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6827 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8c40f1cb8e
self-healing for broken table files (may cause other problems, but better than nothing)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6826 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
13f5b8e7ba
fix for storing/getting bookmark-folders
...
called by Quix0r
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6825 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7b69d79727
enhanced remove() operation: in many cases it is not necessary to return the removed object to the called.
...
for such cases the delete() operation was introduced which is sometimes much cheaper in operation since it does not need to create objects to hold the removed content and it does not need to read those objects.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6824 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
93ea0a4789
enhanced remove operation in search consequences (which are triggered when the snippet fetch proves that the word has disappeared from the page that was stored in the index)
...
- no direct deletion of referenced during search (shifted to time after search)
- bundling of all deletions for the references of a single word into one remove operation
- enhanced remove operation by caring that the collection is stored sorted (experimental)
- more String -> byte[] transition for search word lists
- clean up of unused code
- enhanced memory allocation of RowSet Objects (will use a little bit less memory which was wasted before)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6823 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7a59012632
fix for NPE
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6822 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1a6c2f77b4
fix for NPE in statistic servlet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6821 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
64f29f990e
a collection of performance hacks and code cleanup:
...
- removed usage of URL-Caches which could have been a memory leak
- removed unused classes and methods
- removed not necessary synchronizations
- added synchronization hacks where possible
- fine-tuned crawling speed to prevent IO of balancer
- fixed a bug in IODispatcher that may have caused that no merges were done
- reduced number of parameters in very often called methods (compare methods)
- reduced complexity of data structures of now massively used HandleSet class
- reduction of new String() and getBytes() usage / new methods to support this transition
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6820 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8b8107b2a3
reduced IO-load and synchronization/blocking
...
- enhanced the Balancer performance when building new domain stacks using a new Table buffer
- added the new Table buffer BufferedObjectIndex class
- changed order of access to LURL-read (prefereing segment over Crawl Queues) will reduced blocking time on balancer
- fixed PPM setting in Crawler_p servlet (had doubled values)
- reduced synchronization in IndexCell because it is not necessary: reduced blocking during indexing/merging/dumping
- removed did-you-mean cache in IndexCell because that caused too much overhead and more memory usage but was not very useful. This reduced also deadlocks that could be causes when searched are performed during indexing.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6819 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
mikeworks
7417425e6a
Added (b)log notice in header of table in Network.html
...
Added some missing translations for German de.lng in Network.html page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6818 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ed07046870
flush only when > 3000 RWIs present + code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6817 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3a50b5aa04
enhanced object hash computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6816 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1a8a134e0c
continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775 and continued in SVN 6790
...
The result should be a less usage of new String() and less memory usage (since a String-encapsulated byte[] has 40 bytes overhead)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6815 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dde394a977
- shifted some computation out of synchronization to allow more concurrency
...
- removed synchronization where not necessary
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6814 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f204076d25
removed usage of temporary files: causes too much IO
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6813 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
48b9371735
changed balancer re-load counter. causes less blocking here doing intranet indexing.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6812 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
650be3599f
added a time-out to the RWI cache to flush the cache if it has not been written for ten minutes. This additional dump criteria is necessary because some data sources repeat their vocabulary and may cause that the number of words in a RWI does not increase while the number of references in the RWI set increases. Now the RWI Buffer is flushed every 10 minutes or later if at that time already a dump is ongoing.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6811 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ff6cf24b80
replaced RowSetArray in ObjectIndexCache with RowSet to reduce complexity in MergeIterator. This complexity caused too much computing overhead when the RowSetArray had become very large.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6810 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0d04ab1422
new access tracking data type strategy; previous data types may have caused deadlocks of httpd when performing statistic cleanups
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6809 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
fc43f3028e
*) hopefully fixing NPE issue introduced in r6797
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6808 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
55d8e686ea
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6807 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2f181d0027
introduced concurrency in HTCACHE storage compression
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6806 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2e26744f4e
more concurrency when normalizing RWI entries + cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6805 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
555b333041
fix for wrong count of server processes. may fix non-access problems in some cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6804 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5d8238680f
better thread dump statistic evaluation in threaddump servlet when creating multiple dumps for profiling: no distinguishing between thread states
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6803 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
aa083fc45c
try to get a fix for OOM problem in case that there is no real problem with missing memory.
...
See also http://forum.yacy-websuche.de/viewtopic.php?p=19835#p19835
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6802 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
70e6222978
more concurrency during search requests
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6801 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4917f96729
fixes for some changes in SVN 6797 that caused NPEs when the bookmarks initialized
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6800 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
dff660441a
*) changes for better code readability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6799 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
15d9ea8375
*) changes for better code readability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6798 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
2bc459252e
*) changes for better code readability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6797 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
dc93cec3a8
*) Java 1.5 compatibility (see http://forum.yacy-websuche.de/viewtopic.php?f=8&t=2764 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6796 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
67ec58d8e7
search performance enhancement
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6795 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
4ec0092677
more null == proxy fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6794 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
2f90f0ad56
Remove asserts blocking proxy use cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6793 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
ef467a0303
Another workaround for the second part of http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770
...
This should prevent URLs with bad referrer entries from being dropped by transferURL or even crashing the whole Transmission$Chunk
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6792 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
eb2a4bb555
workaround(?) for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770&start=0&st=0&sk=t&sd=a&hilit=DefaultCharsetStringPart
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6791 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
25aef069a6
continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6790 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
mikeworks
aad0a3045a
Fixed CSS file yacyui-portalsearch.cs
...
border-width: small; --> border-width: thin;
margin-botton --> margin-bottom
Forum Thread: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2767
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6789 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
mikeworks
6ef4900ec5
Changed function name status() -> statuscheck() in yacyui-portalsearch.js fixing the "status is not a function" bug visible in Firefox error console:
...
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2625
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6788 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
b97ad0f380
*) some minor changes for better code readability
...
*) added more SVN properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6787 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
mikeworks
7a3c19846f
Updated German translation de.lng: added new Table_RobotsTxt_p.html and some other changes
...
Changed 'Sprache' -> 'Language' in yacydoc.html and added translation in de.lng
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6783 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ba51d140e1
added more info in assert in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6782 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
000dad034b
*) better usage of JS timers (intervals)
...
*) added messages when status is checked
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6781 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a85c5bb8a7
added support for multiple (fail-over) network definition locations when http-locations are given. multiple locations can be given with a comma-separated list of urls pointing to the network definition file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6780 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f83b1b91b9
increased dht busy sleep time to 10 seconds to reduce TCP/IP traffic for default settings. 2 seconds had been too much traffic for home-use routers.
...
Please try to set your dht busy sleep time in existing installations also to 10 seconds.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6779 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago