Commit Graph

499 Commits (2be327b5ab79f72cf411e3b2bcf1add08b1002cd)

Author SHA1 Message Date
orbiter 123375bfba added a new yacy protocol servlet 'idx'. This returns an index to one of the data entities that is stored in YaCy.
14 years ago
orbiter 5b579e21a3 code cleanup
14 years ago
orbiter 039126cfaf better handling of on/off switched solr indexing
14 years ago
orbiter dc54915df4 fix for very bad compare
14 years ago
orbiter deda54d684 - relaxed matching of string-search (this is now case-insensitive)
14 years ago
orbiter b77b8cac0c - enhanced html parser: recognized much more details in the content
14 years ago
orbiter 17530ca7b5 fix for bug http://bugs.yacy.net/view.php?id=10
14 years ago
orbiter 0430a94eaa the location search shows now not re-evaluated locations but only such locations that are attached as metadata to web pages
14 years ago
orbiter 8412f8787d fix for http://bugs.yacy.net/view.php?id=8
14 years ago
orbiter 9b25d07295 - added geo information parsing to html parser
14 years ago
orbiter b1a8d0c020 enhancements to web cache and less strict caching rules
14 years ago
f1ori df71776929 * fix bug #7
14 years ago
orbiter 78d4c45d09 enhancement during search process: fast fail of search in case that all index feeder have terminated.
14 years ago
orbiter 2b5f8585bf performance hack for Balancer and ip address parsing
14 years ago
orbiter b1d133b69f another anhancement to the ThreadDump function: better multiple dumps and filtering out of not interesting dump parts
14 years ago
orbiter a35d513bd8 fix for not-deleted .gap and .idx files
14 years ago
orbiter 859c99886c fix for multiple thread dump
14 years ago
orbiter 61acf55da4 avoided using a synchronized(this) for the hash computation to prevent that the lock on the object is (accidently) stolen by another thread and replaced this synchronization using the protocol object. Made also the protocol object final.
14 years ago
orbiter c2a968c23f fix for bug in formatting in ThreadDump
14 years ago
orbiter 078ecacf61 avoid synchronization in DigestURI hash requests
14 years ago
orbiter 1989ebc24b removed more warnings
14 years ago
orbiter 0324de1467 removed debug line
14 years ago
orbiter 1aba7869bf patch for Windows: do not use the thread lock feature from previous commit if used on Windows
14 years ago
orbiter 0a11727374 added new feature for Thread dump:
14 years ago
orbiter a07a1a8b1e removed type cast warnings
14 years ago
orbiter e6c3507b17 disabled some of the previous changes (did not work in openjdk)
14 years ago
orbiter f9e5c21083 update to thread dump logs
14 years ago
orbiter 8f11d3a5bb redesigned the ScoreMap classes:
14 years ago
orbiter a564230c48 more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases)
14 years ago
orbiter dc0db3550e avoid string conversion
14 years ago
orbiter 694fa3a2a5 - replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion
14 years ago
orbiter 30aed9824a moved getBytes() to UTF8.getBytes() to use a default String encoding
14 years ago
orbiter 3820525464 more memory protection: auto-flush of caches in case of memory shortage
14 years ago
orbiter 96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...)
14 years ago
orbiter e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations
14 years ago
low012 3b40b98256 *) set SVN properties
14 years ago
orbiter 619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word
14 years ago
low012 bf27a72d53 *) set SVN properties
14 years ago
low012 b649ce2dd7 *) minor changes
14 years ago
orbiter 70a996a06c reverted SVN 7557 because these classes are called using reflection. The class declaration is in the log configuration. Without these classes you get errors during runtime and a non-formatted log output, i.e.:
14 years ago
orbiter cb1f49d0f2 replaced all 'new String' with default encoding (missing) or UTF-8 encoding with a String generation method that uses a pre-defined Charset constant for UTF-8. This avoids a cache-lookup for the Charset object using String hashing of the String 'UTF-8'.
14 years ago
low012 9d366ee9d7 *) removed unused code (I assume that most of the code was really dead, but if you need any of the classes, tell me and I will put it back in.)
14 years ago
orbiter 7138f4036b less synchronization, better thread dump tool
14 years ago
orbiter 8d14916c74 more patches for a better out-of-memory management
14 years ago
orbiter ce0c8247fc removed (most probably!?!) superfluos System.err output
14 years ago
orbiter 799c534935 one more patch again OOM during secondary remote search
14 years ago
orbiter f8d0454c53 small bug fixes and experiments with search speed enhancement
14 years ago
orbiter 993b9bc1a8 memory/performance hacks, less synchronization, better concurrency
14 years ago
orbiter 42d90664f3 - fixed a memory leak in the httpc.post method (no finish)
14 years ago
orbiter 38dce547c0 better concurrency (less locking on date formatting) more logging and minor bug fixes
14 years ago
orbiter b1781d7aae some more performance hacks
14 years ago
orbiter b2f147d28e performance hack: excluded map encoding in many cases from synchronization block, especially when doing an iteration
14 years ago
orbiter 5e186e0122 continuing the fight against deadlocks during time formatting: better caching.
14 years ago
orbiter 1110d16af9 performance hack: replaced generic row.getColBytes() call with row.getPrimaryKeyBytes() where the column is 0
14 years ago
orbiter 19b2a50578 - enhanced date formatter cache
14 years ago
orbiter 48a61c39a3 speed hacks in BLOB ArrayStack:
14 years ago
orbiter ad7fcb9d61 Enhanced Base64Order transformation: less overhead (transformation between StringBuilder and byte[])
14 years ago
orbiter 0ce17d823a - fixed bug in ordering
14 years ago
orbiter dec4f36700 - fix for missing favicons in search widgets
14 years ago
orbiter 804ae2275b - do not delete idx and gap files if the heap is not modified
14 years ago
orbiter 5e45ded8e2 - removed locks from WordReference
14 years ago
orbiter cd19d0517e added dns resolve to HTTPClient POST using a dns cache to prevent that that not-thread-safe built-in dns cache inside apache http client is used
14 years ago
orbiter af87af0d4c - removed synchronization in serverSwitch which should improve speed
14 years ago
orbiter d84b4a072e healing for some OOM problems
14 years ago
orbiter 6083f2f171 fix for (false) oom
14 years ago
orbiter fe93caac5a added flags and administration options to show advanced search and to show search result attributes (for each search result)
14 years ago
orbiter 431f780f41 patch for bad data in url metadata
14 years ago
orbiter 0cdfb82963 replaced more appearance of double values by float values
14 years ago
orbiter eb12e15738 moved all Double values to Float values because of
14 years ago
f1ori 982aa689ef * fix StringIndexOutOfBoundException in WebStructureGraph
14 years ago
orbiter 090c73e32e catch a OOM in HeapReader iteration
14 years ago
orbiter feefe17568 npe assert fix
14 years ago
orbiter 733903f2c9 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21489#p21489
14 years ago
orbiter 10ae8d961b - cora package has now no dependencies to other yacy packages and becomes a 'base' package (refactoring)
14 years ago
lotus b1484299b2 same units for memory observer configuration (MiB)
14 years ago
orbiter 387db84087 maybe found bug in non-working index dumper
14 years ago
orbiter a4c9d27287 - moved some variables from Stwitchboard to new class AccessTracker
14 years ago
orbiter cdfe8afe3f fix for really bad table iteration implementation: reduction of IO
14 years ago
orbiter b2ed4cfaf8 more small bugfixes and light refactoring
14 years ago
orbiter e753027c43 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21439#p21439
14 years ago
orbiter bf4ef1513e - fix for map view
14 years ago
orbiter 56264dcc17 - added CamelCase parser to MultiProtocolURI: generate better to-be-indexed words from urls
14 years ago
orbiter 99a7fe87f9 - removed old intranet scanner (the generic scanner now completely subsumes the old one)
14 years ago
orbiter a563b05b60 enhanced crawler:
14 years ago
orbiter db99db4be9 some redesign of the search-fail-response mechanism:
14 years ago
f1ori 4915d1781a * use local backup-file, if remote network-definition is not availible
14 years ago
orbiter f0651e5f2f added image search to yacyinteractive.html
14 years ago
orbiter a9f754c45f removed unused CR accumulation and distribution process
14 years ago
low012 9b3fae9496 *) cleaning up the code a little bit
14 years ago
orbiter 321eb012fe removed two warnings and reverted one change
14 years ago
low012 eb79b952ef *) cleaner code
14 years ago
sixcooler b87bf88ac8 using less memory on merging and rewriting blobs
14 years ago
orbiter 4c50d3428e smaller file size for array stacks to support smaller deletion sizes
14 years ago
orbiter becc463d8a enhanced did-you-mean
14 years ago
orbiter 93c535d111 fixed http://forum.yacy-websuche.de/viewtopic.php?p=21113#p21113
14 years ago
orbiter 445619f3ec added a submenu ConfigHTCache_p.html to set the size of the HTCache separately from the proxy configuration.
14 years ago
f1ori 7d8de34778 * add a bit documentation to DigestURI, use DigestURI(string) instead of DigestURI(string, null)
14 years ago
orbiter ed4371dcf3 enhanced navigation implementation and enhanced tag cloud computation
14 years ago
orbiter ca738ac924 - added a tag cloud to search results (using the topics)
14 years ago
orbiter e4d561971e added more score cluster options and made score cluster usage more transparent
14 years ago
orbiter 7cd9d9d22a - enhanced DidYouMean computation using a faster count on index entries; this causes that results can be ranked better
14 years ago
orbiter 24f1cba7b2 performance hacks:
14 years ago
orbiter d607b30b6a performance enhancements for search and code review for database functions
14 years ago
orbiter 0d363a94d7 more performance hacks
14 years ago
orbiter 091dd3f6ec - enhanced intranet search speed
14 years ago
orbiter aacf572a26 - enhancements for search speed
14 years ago
orbiter 2c549ae341 fixed a number of small bugs:
14 years ago
orbiter e54cb7fb0c more bugfixes (also for latest commit)
14 years ago
orbiter be6b48311c misc bugfixes
14 years ago
orbiter 48c0d508ac fixes for crawling of smb links (file length not always available)
14 years ago
orbiter 09c208a3ab patch for corrupted database files (just work on and forget key)
14 years ago
orbiter 97ee278931 enhanced search speed:
14 years ago
orbiter 8da4eb5de6 addition to patch in SVN 7111
14 years ago
orbiter 37baa8bae3 - fixes for concurrency exceptions and failed database integrity verification
14 years ago
orbiter 461a2a6ec7 enhanced remote crawling:
14 years ago
orbiter 0cf006865e refactoring and enhanced concurrency
14 years ago
orbiter 83ac07874f - corrected return value of put() methods (not used anywhere, so it did not harm before)
14 years ago
orbiter 14c843d364 more performance hacks
14 years ago
orbiter 39f409a7bb performance hacks
14 years ago
orbiter 906c572621 - enhanced index create menu structure
14 years ago
orbiter 64860dc1bb enhanced search event logging (to be used for further improvements)
14 years ago
orbiter 7dbc357593 patch to identify corrupted database files
14 years ago
sixcooler 17eebd4ef8 counting crawler traffic again:
14 years ago
orbiter 32f73d1aaa added copy for Info.plist for Mac application release updates (this file contains class paths and start parameters)
14 years ago
orbiter 570ca577c6 performance hacks
14 years ago
orbiter 348dece62f redesign of the SortStack and SortStore classes:
14 years ago
orbiter 5fe828fa06 - replaced pdfbox and fontbox version 1.1.0 with 1.2.1
14 years ago
orbiter 24502fe3de performance hacks
14 years ago
orbiter d865ef77a8 removed re-read of index in case of a bad index. This may not solve the problem but it applies a 100% CPU problem on the peer. I'm afraid bad index files must be abandoned, and cannot be fixed this way.
14 years ago
orbiter b2c9db48ea Performance enhancement
14 years ago
orbiter e8228fba09 less locking in time format computation, caching and during secondary (remote) search evaluation
14 years ago
orbiter 9c0c94683c because of a bug in search result caching count search results had not been generated as fast as possible.
14 years ago
orbiter 9d080f387e change in handling of the all-visible home path for storage in YaCy:
14 years ago
orbiter 65eaf30f77 redesign of crawl profiles data structure. target will be:
14 years ago
orbiter 4f22e2df41 bugfixes for
14 years ago
orbiter 42414a6ae3 added two more tables in rss reader interface:
14 years ago
orbiter 0010cd9db1 Support for indexing of RSS feeds!
14 years ago
orbiter 0f276dd63f - MapHeap now implements Map<byte[], Map<String, String>>
14 years ago
orbiter cf07b34c2d implemented the Map interface in the ARC classes so it will be possible to instantiate ARCs as
14 years ago
orbiter c60d0282fd more abstraction for tables stored in heaps:
14 years ago
orbiter d1be64d491 removed wrong assert
14 years ago
orbiter 3197ca42ed preparations to move the HTCache into cora:
14 years ago
orbiter 844f158686 - removed dependencies in header framework:
14 years ago
orbiter 5e7081cd19 refactoring towards a unified loading mechanism for MultiProtocolURIs
14 years ago
orbiter 7aa860c505 - more logging
14 years ago
orbiter 66ac3a7d9d corrected database row iteration
14 years ago
orbiter 70dd26ec95 added the new crawl scheduling function to the crawl start menu:
14 years ago
orbiter 5a994c9796 added a scheduler based on API actions
14 years ago
orbiter 189a986ebd - modified api-call interface to record api calls with references to api-call database (carries pk)
14 years ago
orbiter 054c22e2c6 added TLDs from http://www.opennicproject.org
14 years ago
orbiter 7fdb17bb96 redirect uncaught exceptions to logging + small other changes
14 years ago
orbiter a82a93f2fc - better url double check in crawler
14 years ago
orbiter a835a22b32 fixed isLocal() property (better recognition of intranet hosts)
14 years ago
orbiter 301a59e07f moved browser access method from kelondro/util/OS to gui/framework/Browser
14 years ago
orbiter 6388a58fc7 better memory management and slightly less (in total and temporary) RAM allocation:
14 years ago
orbiter 5924a0d851 - enhanced concurrency in database index access for multicore
14 years ago
orbiter 9ab06bc333 enhancement in sorting efficiency (database root operation): less object allocation
14 years ago
mikeworks b12db14b9f Added Generics to new net.yacy.upnp.* classes to eliminate compiler warnings
15 years ago
orbiter 60caade056 removed debug output
15 years ago
orbiter dec1419bc3 ;-)
15 years ago
orbiter 22dbbcfa56 better (and corrected) recognition of intranet and internet-addresses. This corrects the isLocal property that is used by network definitions to restrict index ranges to local and global addresses. Address locations (intranet or internet) had been partly identified by the top level domain of the host address. Since intranet addresses can also be addressed using a host name that is in a country domain it is necessary to do a dns resolving for each check. The check is supported by a local dns cache so the intranet/internet check should not affect network traffic too much. To ensure that the cache works properly the cache class was upgraded to better concurrency data structures.
15 years ago
low012 dc5f0e357c *) fixed SVN properties
15 years ago
low012 01d6b952f0 *) minor changes for easier to read code, no functional changes
15 years ago
orbiter 25024d6ab2 fix for problen when accessing the metadata index. The index was not available for all peers with no RAM table copy.
15 years ago
orbiter b6fb239e74 redesign of parser interface:
15 years ago
orbiter 150cf42a1b migrated all my LGPL 3 -licensed files to the LGPL 2.1 because LGPL 3 is not compatible to the GPL 2
15 years ago
orbiter 37b8827a7a - removed the UPnP library sources from sbbi and added the jar library again. The library was included to get support for fedora releases, but after this time the fact that the sbbi cannot be part of fedora should be re-discussed. If this will still not be possible, then we may integrate the sbbi UPnP package using reflection.
15 years ago
orbiter 777195e8d1 more abstraction for access of LoaderDispatcher and cache
15 years ago
orbiter 7e2d6fac12 patch for bad values during local search join
15 years ago
orbiter 986d4f34d9 added a consistency check for new queues
15 years ago
orbiter fbf021bb50 redesign of index abstract processing - currently disabled until enough peers have fix in SVN 6928
15 years ago
orbiter 87087f12fe - scanned remote search process and enhanced some data structure and synchronizations here and there
15 years ago
orbiter de4f30bb2e UTF-8 fix
15 years ago
orbiter 3a1cebb598 bugfixes
15 years ago
orbiter 51332b787d reverted SVN 6869 as discussed with dulcedo in car after LinuxTag:
15 years ago
orbiter b03caaa57a better handling of OOM situations
15 years ago
orbiter 60e71876ad - more abstraction (HashMap -> Map)
15 years ago
orbiter a83772c71b fixes and enhancements for balancer:
15 years ago
orbiter 9cde05418f fixed url crawl list display
15 years ago
orbiter 11639aef35 - added new protocol loader for 'file'-type URLs
15 years ago
orbiter 6950d8a33d fixes to SMB crawler
15 years ago
orbiter 1defd580bc - added option to localization search to distinguish between a search for a location according to the search word only or for the relation between a web search results and locations found in the metadata fields
15 years ago
orbiter 118d589eff replaced the very very old data structure 'Records' with a simple table to fix the problem from
15 years ago
orbiter 2a8f70f0ca - fix for caching of OSM tiles. if you want that this fix applies to your peer, please delete the crawl profiles
15 years ago
orbiter 439b44be9e removed exit from computation in ReferenceContainerArray.get merge method
15 years ago
orbiter 789c6b26ce added a location search service: using the following servlet/example:
15 years ago
orbiter f23cbd2dab more bugfixes to date parser
15 years ago
orbiter cf43bdc87e This is a large bugfix and enhancement commit to support a better location detection for data
15 years ago
orbiter c45117f81f fixed dates in metadata
15 years ago
orbiter 0a5fd15703 :-(
15 years ago
orbiter ac16f582aa fix for http://forum.yacy-websuche.de/viewtopic.php?p=20017#p20017
15 years ago
orbiter fc5efcc05a enhanced and fixed OAI-PMH import
15 years ago
orbiter 455a763d7c performance hacks
15 years ago
orbiter b6cce08019 fixed a bug in rwi storage data size allocation
15 years ago
orbiter 90c3e5d6f6 - cleanup, removed unused imports
15 years ago
orbiter b18a7606a0 some performance hacks and fixed after reading dump in
15 years ago
orbiter 4cd5418963 removed finalize methods because of a hint in
15 years ago
orbiter cff8ed134f added index check to prevent blocking in synchronization
15 years ago
orbiter b95ae2518b fix for assert
15 years ago
orbiter 027b971bde fix for concurrent quicksort: catch jobs from ThreadPoolExecutor that had been rejected because of full processing queues.
15 years ago