orbiter
ed07046870
flush only when > 3000 RWIs present + code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6817 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3a50b5aa04
enhanced object hash computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6816 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1a8a134e0c
continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775 and continued in SVN 6790
...
The result should be a less usage of new String() and less memory usage (since a String-encapsulated byte[] has 40 bytes overhead)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6815 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dde394a977
- shifted some computation out of synchronization to allow more concurrency
...
- removed synchronization where not necessary
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6814 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f204076d25
removed usage of temporary files: causes too much IO
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6813 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
48b9371735
changed balancer re-load counter. causes less blocking here doing intranet indexing.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6812 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
650be3599f
added a time-out to the RWI cache to flush the cache if it has not been written for ten minutes. This additional dump criteria is necessary because some data sources repeat their vocabulary and may cause that the number of words in a RWI does not increase while the number of references in the RWI set increases. Now the RWI Buffer is flushed every 10 minutes or later if at that time already a dump is ongoing.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6811 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ff6cf24b80
replaced RowSetArray in ObjectIndexCache with RowSet to reduce complexity in MergeIterator. This complexity caused too much computing overhead when the RowSetArray had become very large.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6810 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0d04ab1422
new access tracking data type strategy; previous data types may have caused deadlocks of httpd when performing statistic cleanups
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6809 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
fc43f3028e
*) hopefully fixing NPE issue introduced in r6797
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6808 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
55d8e686ea
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6807 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2f181d0027
introduced concurrency in HTCACHE storage compression
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6806 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2e26744f4e
more concurrency when normalizing RWI entries + cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6805 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
555b333041
fix for wrong count of server processes. may fix non-access problems in some cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6804 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
aa083fc45c
try to get a fix for OOM problem in case that there is no real problem with missing memory.
...
See also http://forum.yacy-websuche.de/viewtopic.php?p=19835#p19835
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6802 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
70e6222978
more concurrency during search requests
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6801 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4917f96729
fixes for some changes in SVN 6797 that caused NPEs when the bookmarks initialized
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6800 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
dff660441a
*) changes for better code readability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6799 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
15d9ea8375
*) changes for better code readability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6798 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
2bc459252e
*) changes for better code readability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6797 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
dc93cec3a8
*) Java 1.5 compatibility (see http://forum.yacy-websuche.de/viewtopic.php?f=8&t=2764 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6796 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
67ec58d8e7
search performance enhancement
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6795 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
4ec0092677
more null == proxy fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6794 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
2f90f0ad56
Remove asserts blocking proxy use cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6793 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
ef467a0303
Another workaround for the second part of http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770
...
This should prevent URLs with bad referrer entries from being dropped by transferURL or even crashing the whole Transmission$Chunk
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6792 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
eb2a4bb555
workaround(?) for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770&start=0&st=0&sk=t&sd=a&hilit=DefaultCharsetStringPart
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6791 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
25aef069a6
continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6790 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
b97ad0f380
*) some minor changes for better code readability
...
*) added more SVN properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6787 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ba51d140e1
added more info in assert in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6782 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a85c5bb8a7
added support for multiple (fail-over) network definition locations when http-locations are given. multiple locations can be given with a comma-separated list of urls pointing to the network definition file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6780 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9b3840cb66
performance hacks for the template engine + cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6778 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5c10f8bc5f
enhanced latest hack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6777 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b3238bec83
performance hack for httpd
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6776 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1e8e79b9ef
redesign of reference hash (URL-hash) parameter hand-over:
...
pass value as byte[], not as String. This should cause that less
byte[] <-> String conversions are made during time-critical tasks.
This redesign is not yet complete, more to come ..
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6775 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
72d8e9897b
removed unnecessary cache flush call in backend of BufferedRecords
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6774 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
749ffbd642
- added another catch case for the index dump and index merge process that should cause non-blocking behavior in case that index dump and/or index merge caused any unexpected exception.
...
- reverted SVN 6766, this is too dangerous (may cause unexpected memory usage) and should not be necessary
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6773 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9ddb8e4a43
set an option for the java-internal image parser that prevents that the image is cached using the file-system in a temporary file. This should speed up image parsing during image indexing dramatically and should also cause better performance when showing the yacy banner and OSM tiles.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6772 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
312ca5d917
removed flush at end of every rwi entry since this reduces the write performance.
...
This should speed up RWI cache dump and RWI merge operations and should cause less blocking time during these processes for the indexer.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6771 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0018163c07
moved table row/column matching method from front-end to back-end
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6770 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e12f1fd821
- added setting of access rights for executable scripts after auto-installation
...
The correct access right was missing expecially for bin/apicall.sh
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6769 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
31e29a8831
- removed synchronization during index dump and index cleaning
...
- added semaphores to synchronize index dump and index cleaning for each process separately
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6767 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
95f31da8da
increase dump cache queue length from 1 to 2
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6766 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
6c093d6aed
- enhanced domain navigator computation
...
- fixed domain navigator content in case that a mustmatch constraint was given
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6763 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bb63c5d075
using a Pattern object with precompiled regular expressions to apply must-match constraints to search results: should speed up pre-sorting of search results and should cause richer search result sets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6762 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e0da0a84b0
performance fix in http parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6760 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
90dd197ae7
- no latency for local crawls
...
- catch interrupted exception during 'fast' crawls in workflow processor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6759 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bfb518cd47
some refactoring to get the LoaderDispatcher a little bit more independent from the switchboard
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6755 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
36bd843ece
for for RFC5322 comformance as suggested by Quix0r in http://forum.yacy-websuche.de/viewtopic.php?p=19585#p19585
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6754 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c855fc48c6
only load robots.txt for http and http protocol
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6753 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
748abfcffa
added patches to prevent yacy-protocol DoS settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6751 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago