Commit Graph

541 Commits (2280a7b276df243b687fb7a3ed10c939fd25c9c4)

Author SHA1 Message Date
orbiter 51cf697acd refactoring: moved all score-related classes to new ranking package
13 years ago
sixcooler 4fec99115b Implementation of strategies for controlling memory resources.
13 years ago
orbiter 2c58af6874 - added a short memory status simulation mode
13 years ago
orbiter c64faf41e2 addon to svn 7880
13 years ago
sixcooler 411ed159f8 do some extra sleep while running low on memory
13 years ago
sixcooler 07f5954570 try better handling of corrupt blobs
13 years ago
orbiter 0a3ab7da1b do not sort concrrently the same array
13 years ago
orbiter 44d74f8f89 performance hacks for seed generation (because thread dumps showed multiple occurrences at these code points)
13 years ago
sixcooler 5cd07d7f84 early freeing resources on deleting index reference if search-verification fails (aka Switchboard.cleanupJob)
13 years ago
sixcooler 9170a434ed throwing an exception again in FileUtils.copy(reader, writer)
13 years ago
sixcooler 916d79111e Runtime.maxMemory() DOES change @ runtime:
14 years ago
orbiter 1f300217f8 more protection for the cleanup thread
14 years ago
orbiter d13103a0a7 changed the way how the index cache is flushed: do not flush when a put was made because that could cause that many put calls synchronize for a long time when the dump or a merge is performed. Instead a watchdog thread is doing the dump and therefore puts cannot block any more which is good when a put happens during a search result preparation.
14 years ago
orbiter 6a6f27eaf3 do not sort arrays again if arrays are already sorted
14 years ago
orbiter 3d043ce9d6 - refactoring
14 years ago
orbiter 48b78e9ff4 disabling concurrency in new sort since that is not working yet correctly
14 years ago
orbiter 62ac73a108 fixed bugs and deadlocks in core database indexing structures:
14 years ago
orbiter 1912d0cccc changed handling of RowSet element retrieval: until today all elements had been copied from the underlying byte[] arrays into a new Entry object that again had a copy of a portion of that byte[] in its own bye[]. There was an option to just refer to the underlying byte[] with a pointer but that was almost never used. This commit now changes an interface to the Row class where it is now necessary to tell if a copy is always required. Fortunately the copy is only needed in very rare cases. That means that this change should cause much less memory allocation; it is expected that this happens especially during search situations.
14 years ago
orbiter bb8e3f8523 code cleanup
14 years ago
orbiter 11dc653de3 added a visualization of peer pings to the performance graphic
14 years ago
orbiter 6d2e252bcf fix for:
14 years ago
orbiter b666a929e7 fixed Semaphore handling in case of interruptions
14 years ago
orbiter 267290a821 removed the semaphores from the cache dump process because I believe some of the semaphores may be lost somewhere which then causes that the cache is never flushed and then the peer dies from a OOM. The re-introduced synchronization may not be the best solution but should ensure that the caches are flushed.
14 years ago
orbiter f803da8aae code cleanup
14 years ago
orbiter 31283ecd07 - added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search.
14 years ago
orbiter 7db208c992 performance hacks: more pre-allocated StringBuilder
14 years ago
orbiter 996f0a8764 disabled assert in Base64Order which eats away too much performance during testing with -l
14 years ago
orbiter f30d36b101 enhanced template engine
14 years ago
orbiter 0c1b29f3c9 - applied many small performance hacks
14 years ago
orbiter fe0c08455b more concurrency (enhancement) hacks
14 years ago
orbiter 87082f407e less String object creation during search
14 years ago
orbiter a36fda991e hack to increase speed of url hash computation
14 years ago
orbiter dbea40d536 - changed snippet fetch strategy logic: do not check if entry is in cache. This should reduce IO load on the HTCACHE which is a showstopper during large number of search requests
14 years ago
orbiter 4bea3f9714 hack to reduce resource contention caused by massive UTF8 decodings which use java.nio resources:
14 years ago
orbiter 746e3c3b06 Replaced a widely-used Property Object in the httpd with HashMap<String, Object> which is not synchronized like Properties
14 years ago
orbiter e28bd0d038 fix for some possible causes of memory leaks
14 years ago
orbiter 09ba6814c0 - non-blocking word hash computation with dynamic digest object generation (this was important!)
14 years ago
orbiter 10e2f588f8 - enhanced ybr ranking computation
14 years ago
orbiter bd55dcee50 - commented out experimental distributed ranking loading
14 years ago
orbiter 3ed4a09368 small features, some bug fixes and performance hacks
14 years ago
orbiter b45701d20f this is a re-implementation of the YaCy Block Rank feature
14 years ago
orbiter d27a0a67ff fix in log initialization according to hint from Dominic
14 years ago
orbiter 123375bfba added a new yacy protocol servlet 'idx'. This returns an index to one of the data entities that is stored in YaCy.
14 years ago
orbiter 5b579e21a3 code cleanup
14 years ago
orbiter 039126cfaf better handling of on/off switched solr indexing
14 years ago
orbiter dc54915df4 fix for very bad compare
14 years ago
orbiter deda54d684 - relaxed matching of string-search (this is now case-insensitive)
14 years ago
orbiter b77b8cac0c - enhanced html parser: recognized much more details in the content
14 years ago
orbiter 17530ca7b5 fix for bug http://bugs.yacy.net/view.php?id=10
14 years ago
orbiter 0430a94eaa the location search shows now not re-evaluated locations but only such locations that are attached as metadata to web pages
14 years ago
orbiter 8412f8787d fix for http://bugs.yacy.net/view.php?id=8
14 years ago
orbiter 9b25d07295 - added geo information parsing to html parser
14 years ago
orbiter b1a8d0c020 enhancements to web cache and less strict caching rules
14 years ago
f1ori df71776929 * fix bug #7
14 years ago
orbiter 78d4c45d09 enhancement during search process: fast fail of search in case that all index feeder have terminated.
14 years ago
orbiter 2b5f8585bf performance hack for Balancer and ip address parsing
14 years ago
orbiter b1d133b69f another anhancement to the ThreadDump function: better multiple dumps and filtering out of not interesting dump parts
14 years ago
orbiter a35d513bd8 fix for not-deleted .gap and .idx files
14 years ago
orbiter 859c99886c fix for multiple thread dump
14 years ago
orbiter 61acf55da4 avoided using a synchronized(this) for the hash computation to prevent that the lock on the object is (accidently) stolen by another thread and replaced this synchronization using the protocol object. Made also the protocol object final.
14 years ago
orbiter c2a968c23f fix for bug in formatting in ThreadDump
14 years ago
orbiter 078ecacf61 avoid synchronization in DigestURI hash requests
14 years ago
orbiter 1989ebc24b removed more warnings
14 years ago
orbiter 0324de1467 removed debug line
14 years ago
orbiter 1aba7869bf patch for Windows: do not use the thread lock feature from previous commit if used on Windows
14 years ago
orbiter 0a11727374 added new feature for Thread dump:
14 years ago
orbiter a07a1a8b1e removed type cast warnings
14 years ago
orbiter e6c3507b17 disabled some of the previous changes (did not work in openjdk)
14 years ago
orbiter f9e5c21083 update to thread dump logs
14 years ago
orbiter 8f11d3a5bb redesigned the ScoreMap classes:
14 years ago
orbiter a564230c48 more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases)
14 years ago
orbiter dc0db3550e avoid string conversion
14 years ago
orbiter 694fa3a2a5 - replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion
14 years ago
orbiter 30aed9824a moved getBytes() to UTF8.getBytes() to use a default String encoding
14 years ago
orbiter 3820525464 more memory protection: auto-flush of caches in case of memory shortage
14 years ago
orbiter 96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...)
14 years ago
orbiter e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations
14 years ago
low012 3b40b98256 *) set SVN properties
14 years ago
orbiter 619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word
14 years ago
low012 bf27a72d53 *) set SVN properties
14 years ago
low012 b649ce2dd7 *) minor changes
14 years ago
orbiter 70a996a06c reverted SVN 7557 because these classes are called using reflection. The class declaration is in the log configuration. Without these classes you get errors during runtime and a non-formatted log output, i.e.:
14 years ago
orbiter cb1f49d0f2 replaced all 'new String' with default encoding (missing) or UTF-8 encoding with a String generation method that uses a pre-defined Charset constant for UTF-8. This avoids a cache-lookup for the Charset object using String hashing of the String 'UTF-8'.
14 years ago
low012 9d366ee9d7 *) removed unused code (I assume that most of the code was really dead, but if you need any of the classes, tell me and I will put it back in.)
14 years ago
orbiter 7138f4036b less synchronization, better thread dump tool
14 years ago
orbiter 8d14916c74 more patches for a better out-of-memory management
14 years ago
orbiter ce0c8247fc removed (most probably!?!) superfluos System.err output
14 years ago
orbiter 799c534935 one more patch again OOM during secondary remote search
14 years ago
orbiter f8d0454c53 small bug fixes and experiments with search speed enhancement
14 years ago
orbiter 993b9bc1a8 memory/performance hacks, less synchronization, better concurrency
14 years ago
orbiter 42d90664f3 - fixed a memory leak in the httpc.post method (no finish)
14 years ago
orbiter 38dce547c0 better concurrency (less locking on date formatting) more logging and minor bug fixes
14 years ago
orbiter b1781d7aae some more performance hacks
14 years ago
orbiter b2f147d28e performance hack: excluded map encoding in many cases from synchronization block, especially when doing an iteration
14 years ago
orbiter 5e186e0122 continuing the fight against deadlocks during time formatting: better caching.
14 years ago
orbiter 1110d16af9 performance hack: replaced generic row.getColBytes() call with row.getPrimaryKeyBytes() where the column is 0
14 years ago
orbiter 19b2a50578 - enhanced date formatter cache
14 years ago
orbiter 48a61c39a3 speed hacks in BLOB ArrayStack:
14 years ago
orbiter ad7fcb9d61 Enhanced Base64Order transformation: less overhead (transformation between StringBuilder and byte[])
14 years ago
orbiter 0ce17d823a - fixed bug in ordering
14 years ago
orbiter dec4f36700 - fix for missing favicons in search widgets
14 years ago
orbiter 804ae2275b - do not delete idx and gap files if the heap is not modified
14 years ago
orbiter 5e45ded8e2 - removed locks from WordReference
14 years ago
orbiter cd19d0517e added dns resolve to HTTPClient POST using a dns cache to prevent that that not-thread-safe built-in dns cache inside apache http client is used
14 years ago
orbiter af87af0d4c - removed synchronization in serverSwitch which should improve speed
14 years ago
orbiter d84b4a072e healing for some OOM problems
14 years ago
orbiter 6083f2f171 fix for (false) oom
14 years ago
orbiter fe93caac5a added flags and administration options to show advanced search and to show search result attributes (for each search result)
14 years ago
orbiter 431f780f41 patch for bad data in url metadata
14 years ago
orbiter 0cdfb82963 replaced more appearance of double values by float values
14 years ago
orbiter eb12e15738 moved all Double values to Float values because of
14 years ago
f1ori 982aa689ef * fix StringIndexOutOfBoundException in WebStructureGraph
14 years ago
orbiter 090c73e32e catch a OOM in HeapReader iteration
14 years ago
orbiter feefe17568 npe assert fix
14 years ago
orbiter 733903f2c9 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21489#p21489
14 years ago
orbiter 10ae8d961b - cora package has now no dependencies to other yacy packages and becomes a 'base' package (refactoring)
14 years ago
lotus b1484299b2 same units for memory observer configuration (MiB)
14 years ago
orbiter 387db84087 maybe found bug in non-working index dumper
14 years ago
orbiter a4c9d27287 - moved some variables from Stwitchboard to new class AccessTracker
14 years ago
orbiter cdfe8afe3f fix for really bad table iteration implementation: reduction of IO
14 years ago
orbiter b2ed4cfaf8 more small bugfixes and light refactoring
14 years ago
orbiter e753027c43 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21439#p21439
14 years ago
orbiter bf4ef1513e - fix for map view
14 years ago
orbiter 56264dcc17 - added CamelCase parser to MultiProtocolURI: generate better to-be-indexed words from urls
14 years ago
orbiter 99a7fe87f9 - removed old intranet scanner (the generic scanner now completely subsumes the old one)
14 years ago
orbiter a563b05b60 enhanced crawler:
14 years ago
orbiter db99db4be9 some redesign of the search-fail-response mechanism:
14 years ago
f1ori 4915d1781a * use local backup-file, if remote network-definition is not availible
14 years ago
orbiter f0651e5f2f added image search to yacyinteractive.html
14 years ago
orbiter a9f754c45f removed unused CR accumulation and distribution process
14 years ago
low012 9b3fae9496 *) cleaning up the code a little bit
14 years ago
orbiter 321eb012fe removed two warnings and reverted one change
14 years ago
low012 eb79b952ef *) cleaner code
14 years ago
sixcooler b87bf88ac8 using less memory on merging and rewriting blobs
14 years ago
orbiter 4c50d3428e smaller file size for array stacks to support smaller deletion sizes
14 years ago
orbiter becc463d8a enhanced did-you-mean
14 years ago
orbiter 93c535d111 fixed http://forum.yacy-websuche.de/viewtopic.php?p=21113#p21113
14 years ago
orbiter 445619f3ec added a submenu ConfigHTCache_p.html to set the size of the HTCache separately from the proxy configuration.
14 years ago
f1ori 7d8de34778 * add a bit documentation to DigestURI, use DigestURI(string) instead of DigestURI(string, null)
14 years ago
orbiter ed4371dcf3 enhanced navigation implementation and enhanced tag cloud computation
14 years ago
orbiter ca738ac924 - added a tag cloud to search results (using the topics)
14 years ago
orbiter e4d561971e added more score cluster options and made score cluster usage more transparent
14 years ago
orbiter 7cd9d9d22a - enhanced DidYouMean computation using a faster count on index entries; this causes that results can be ranked better
14 years ago
orbiter 24f1cba7b2 performance hacks:
14 years ago
orbiter d607b30b6a performance enhancements for search and code review for database functions
14 years ago
orbiter 0d363a94d7 more performance hacks
14 years ago
orbiter 091dd3f6ec - enhanced intranet search speed
14 years ago
orbiter aacf572a26 - enhancements for search speed
14 years ago
orbiter 2c549ae341 fixed a number of small bugs:
14 years ago
orbiter e54cb7fb0c more bugfixes (also for latest commit)
14 years ago
orbiter be6b48311c misc bugfixes
14 years ago
orbiter 48c0d508ac fixes for crawling of smb links (file length not always available)
14 years ago
orbiter 09c208a3ab patch for corrupted database files (just work on and forget key)
14 years ago
orbiter 97ee278931 enhanced search speed:
14 years ago
orbiter 8da4eb5de6 addition to patch in SVN 7111
14 years ago
orbiter 37baa8bae3 - fixes for concurrency exceptions and failed database integrity verification
14 years ago
orbiter 461a2a6ec7 enhanced remote crawling:
14 years ago
orbiter 0cf006865e refactoring and enhanced concurrency
14 years ago
orbiter 83ac07874f - corrected return value of put() methods (not used anywhere, so it did not harm before)
14 years ago
orbiter 14c843d364 more performance hacks
14 years ago
orbiter 39f409a7bb performance hacks
14 years ago
orbiter 906c572621 - enhanced index create menu structure
14 years ago
orbiter 64860dc1bb enhanced search event logging (to be used for further improvements)
14 years ago
orbiter 7dbc357593 patch to identify corrupted database files
14 years ago
sixcooler 17eebd4ef8 counting crawler traffic again:
14 years ago
orbiter 32f73d1aaa added copy for Info.plist for Mac application release updates (this file contains class paths and start parameters)
14 years ago
orbiter 570ca577c6 performance hacks
14 years ago
orbiter 348dece62f redesign of the SortStack and SortStore classes:
14 years ago
orbiter 5fe828fa06 - replaced pdfbox and fontbox version 1.1.0 with 1.2.1
14 years ago
orbiter 24502fe3de performance hacks
14 years ago
orbiter d865ef77a8 removed re-read of index in case of a bad index. This may not solve the problem but it applies a 100% CPU problem on the peer. I'm afraid bad index files must be abandoned, and cannot be fixed this way.
14 years ago
orbiter b2c9db48ea Performance enhancement
14 years ago
orbiter e8228fba09 less locking in time format computation, caching and during secondary (remote) search evaluation
14 years ago
orbiter 9c0c94683c because of a bug in search result caching count search results had not been generated as fast as possible.
14 years ago
orbiter 9d080f387e change in handling of the all-visible home path for storage in YaCy:
14 years ago
orbiter 65eaf30f77 redesign of crawl profiles data structure. target will be:
14 years ago
orbiter 4f22e2df41 bugfixes for
14 years ago
orbiter 42414a6ae3 added two more tables in rss reader interface:
14 years ago
orbiter 0010cd9db1 Support for indexing of RSS feeds!
14 years ago
orbiter 0f276dd63f - MapHeap now implements Map<byte[], Map<String, String>>
14 years ago
orbiter cf07b34c2d implemented the Map interface in the ARC classes so it will be possible to instantiate ARCs as
14 years ago
orbiter c60d0282fd more abstraction for tables stored in heaps:
14 years ago
orbiter d1be64d491 removed wrong assert
14 years ago
orbiter 3197ca42ed preparations to move the HTCache into cora:
14 years ago
orbiter 844f158686 - removed dependencies in header framework:
14 years ago
orbiter 5e7081cd19 refactoring towards a unified loading mechanism for MultiProtocolURIs
14 years ago
orbiter 7aa860c505 - more logging
14 years ago
orbiter 66ac3a7d9d corrected database row iteration
14 years ago
orbiter 70dd26ec95 added the new crawl scheduling function to the crawl start menu:
14 years ago
orbiter 5a994c9796 added a scheduler based on API actions
14 years ago
orbiter 189a986ebd - modified api-call interface to record api calls with references to api-call database (carries pk)
14 years ago
orbiter 054c22e2c6 added TLDs from http://www.opennicproject.org
14 years ago
orbiter 7fdb17bb96 redirect uncaught exceptions to logging + small other changes
14 years ago
orbiter a82a93f2fc - better url double check in crawler
14 years ago
orbiter a835a22b32 fixed isLocal() property (better recognition of intranet hosts)
14 years ago
orbiter 301a59e07f moved browser access method from kelondro/util/OS to gui/framework/Browser
14 years ago
orbiter 6388a58fc7 better memory management and slightly less (in total and temporary) RAM allocation:
14 years ago
orbiter 5924a0d851 - enhanced concurrency in database index access for multicore
14 years ago
orbiter 9ab06bc333 enhancement in sorting efficiency (database root operation): less object allocation
14 years ago
mikeworks b12db14b9f Added Generics to new net.yacy.upnp.* classes to eliminate compiler warnings
15 years ago