Commit Graph

4617 Commits (e02bfbde569117645c730968d6fa4e4c1ba33b41)

Author SHA1 Message Date
f1ori 41e146116a fixes size of document in case the server doesn't give the size in the header
14 years ago
orbiter e1a3d609aa moved merger object from Segment to IndexCell to enable a correct shutdown sequence. This solves a bug where yacy cannot be shut down during an index merge that appears during the shutdown phase.
14 years ago
sixcooler 2cf61a40ce fixed a bug from 7856, where Snippet returned an error by mistake when Metadata was found
14 years ago
orbiter 610b01e1c3 - added a 'add every media object linked in a html document as a new document' to the html parser. This causes that all image, app, video or audio file that is linked in a html file is added as document. In fact that means that parsing a single html document may cause that a number of documents is inserted into the search index.
14 years ago
orbiter 3da21c4266 protection against starting of a (second) yacy peer while another one is already running on the same port
14 years ago
orbiter 3e6767d66c limitation of reference evaluation (protection against crawler pits)
14 years ago
orbiter 2c595a6a47 added new methods to count the number of objects in RWIs. lots of refactoring was necessary to introduce new Rating class and to unify naming of methods
14 years ago
orbiter 9f9f634de2 fix in search
14 years ago
sixcooler 5f8a5ca32d - not doing merge-jobs while short on Memory
14 years ago
orbiter 22d69a6368 refactoring in cora: added sorting package
14 years ago
orbiter 51cf697acd refactoring: moved all score-related classes to new ranking package
14 years ago
sixcooler 169236c6d9 almost revert changes in this class of 7880 and 7882
14 years ago
sixcooler 4fec99115b Implementation of strategies for controlling memory resources.
14 years ago
orbiter c64faf41e2 addon to svn 7880
14 years ago
sixcooler 06408a9428 since many POST-requests come as gzip they report a contentlength of -1
14 years ago
orbiter 594d8f546a #cccamp11 maintenance fix: anons may find up to 1000 items in interactive search (was: 100)
14 years ago
sixcooler eb14111200 encapsulate potential expensive objects in TextSnippet to allow GC them asap
14 years ago
orbiter e3fc1efbef performance hack and ensuring termination in serverAccessTracker. cause:
14 years ago
orbiter 44d74f8f89 performance hacks for seed generation (because thread dumps showed multiple occurrences at these code points)
14 years ago
sixcooler 5cd07d7f84 early freeing resources on deleting index reference if search-verification fails (aka Switchboard.cleanupJob)
14 years ago
sixcooler a311596881 finishing up my commits (7855-7858) which could be helpful for
14 years ago
sixcooler c0caca57e3 stoping thread for fetching searchresults if running short on memory
14 years ago
sixcooler ce248cc8dd less byte-arrays of response-content, less byte-array <-> stream conversation
14 years ago
sixcooler 59b767eebd stop loading via http at defined maximum of bytes - even size is unknown before loading
14 years ago
f1ori 3a5fa73008 * revert parts of previous commit, because it breaks the trickle-feature
14 years ago
f1ori 6e79675ff3 * use gzip-encoding in more cases
14 years ago
sixcooler aff875baef smaler ping-entry @ ProfilingGraph
14 years ago
orbiter 1912d0cccc changed handling of RowSet element retrieval: until today all elements had been copied from the underlying byte[] arrays into a new Entry object that again had a copy of a portion of that byte[] in its own bye[]. There was an option to just refer to the underlying byte[] with a pointer but that was almost never used. This commit now changes an interface to the Row class where it is now necessary to tell if a copy is always required. Fortunately the copy is only needed in very rare cases. That means that this change should cause much less memory allocation; it is expected that this happens especially during search situations.
14 years ago
orbiter be15874be1 added request line in http which can support better debugging
14 years ago
orbiter 11dc653de3 added a visualization of peer pings to the performance graphic
14 years ago
orbiter 3a191cdf14 because newbies are scared about the memory consumption in the performance graph and arguments about high memory consumption according to bad knowledge about java garbage collection techniques, the memory display had been removed from the performance graph shown on the Status.html page. The memory graph can still be seen on the Performance page where the memory graph is just like it was.
14 years ago
cominch 09bb7a390c do not replace malformed or invalid URLs in urlproxy
14 years ago
orbiter 768c59740c - replaced solrj 3.1 with solrj 3.3
14 years ago
low012 c7b95e8c81 *) Invalid crawl profiles (containing invalid mustmatch/mustnotmatch filters) will be moved from active crawls to invalid crawls (new file: DATA/INDEX/freeworld/QUEUES/crawlProfilesInvalid.heap). This file can not be edited yet, but it shoudl be easy to extend the CrawlProfileEditor accordingly.
14 years ago
orbiter 719777b2a7 replaced method to call getUsableSpace using reflection with direct call since we now use java 1.6
14 years ago
orbiter 2d4bb139d3 - added counting of links with noindex tag for solr index
14 years ago
orbiter 892caccdca added default configuration in ConfigurationSet in case of new values
14 years ago
orbiter bda3eec0ff added parsing of canonical link element to html parser
14 years ago
orbiter b6f09a475d - added an index profile editor in the /indexFederated_p.html servlet for solr indexes
14 years ago
f1ori a17351dcfe * navigation bar for filetype constraints
14 years ago
f1ori 96957375cc * fix url proxy for relative links and chromium
14 years ago
orbiter 9ebc75db4b fix for channel authorization
14 years ago
orbiter 6d9e5865ee faster appearance of search result page (but complete search time is the same)
14 years ago
orbiter f7ca84cfc0 enhanced template engine
14 years ago
orbiter 84c9658644 added a file type navigator
14 years ago
orbiter 31283ecd07 - added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search.
14 years ago
orbiter 4b425ffdd2 fix for http://bugs.yacy.net/view.php?id=41
14 years ago
orbiter 7db208c992 performance hacks: more pre-allocated StringBuilder
14 years ago
orbiter 87bd559c42 fixed warning
14 years ago
orbiter f30d36b101 enhanced template engine
14 years ago