boxtec
/
yacy_search_server
mirror of https://github.com/yacy/yacy_search_server
4
1
Fork 0
Code Issues Projects Releases Wiki Activity
8762 Commits
1 Branch
20 Tags
1023 MiB
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '755f5e76cf'
${ noResults }
Commit Graph

1337 Commits (755f5e76cfe9414272d84a12d0c3f8dc68636f29)

Author SHA1 Message Date
Michael Christen 20962a4ed7 added metadata node stub for metadata from blobs
13 years ago
Michael Christen 575dbbaa93 enhancements in Blob retrieval: try to use less CPU resources by testing
13 years ago
Michael Christen 585a8f3c44 fixed a bug in search sequence (caused emtpy results)
13 years ago
Michael Christen 361146dd7a better error handling for file loader
13 years ago
Roland 'Quix0r' Haeder 6d4e08ed06 Rewrote filesize() to (hopefully) avoid a NPE, rewrote Blacklist class to concurrent classes to avoid a CME
13 years ago
Roland 'Quix0r' Haeder fa08ed5ae5 Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check
13 years ago
Roland Haeder 319fd1f4aa A concurrent access can happen on the blacklist (with latest introduced blacklist check in media snippet computation)
13 years ago
Roland 'Quix0r' Haeder a3083d13bf Blacklist checks are now always turned on, in media searches (e.g. image search) images matching blacklist entries are no longer shown to the user
13 years ago
Michael Christen 52184a1170 fix for search process
13 years ago
Michael Christen 85bd4cc8bc better lookup for peer names
13 years ago
Michael Christen 20e3084bd4 redesign of fining of peers by ip: more leightweight method to read the
13 years ago
Michael Christen 0797b0de99 new handling of remote search processes: looking for seeds will now not
13 years ago
Michael Christen ee9aae5cc0 more about CreativeCommons license vocabulary
13 years ago
Michael Christen ecd74fe34f less dramatic upnp failures
13 years ago
Michael Christen c75e1a3125 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
Michael Christen 13f5b5f80d the component part in the YaCy Metadata is filled using the Dubling Core
13 years ago
Michael Peter Christen 8d2cbfb685 more vocabularies and more semantics for lod data structures
13 years ago
Michael Christen 9cd36b4c44 added vocabulary for geolocalization as used in georss
13 years ago
Michael Christen 9e5894c784 Removed handling of components objects for URIMetadataRows.
13 years ago
Michael Christen 66ab51f89d added rdf vocabulary
13 years ago
Michael Christen c04bfaa51b refactoring
13 years ago
Michael Peter Christen 136b514f52 added a Triple Store based on Nodes that fit to the new storage classes.
13 years ago
Michael Peter Christen 613ab6a69d added BEncodedHeapBag and BEncodedHeapShard which are storage container
13 years ago
Michael Christen 6fecd0db88 one more performance hack to prevent costly md5 computation
13 years ago
Michael Christen e13441b069 better digest pool size (smaller by default but unlimited)
13 years ago
Michael Christen 1f4afb4dc0 performance hacks
13 years ago
Michael Christen 675d557e88 removed debug logging
13 years ago
Michael Christen e9dc99fe15 added rules to set specific RWIs as private RWIs which are not
13 years ago
Michael Peter Christen 4243ace863 added phonetic classes
13 years ago
Michael Peter Christen 0bcef2d156 added feature as requested in
13 years ago
Michael Christen 204c29f010 small bugfixes for search result display and cache display
13 years ago
Michael Christen 17f962fceb translator updates:
13 years ago
Michael Christen 078fcde0dd bad initialization
13 years ago
Michael Christen 14e45e90fd patch for a bug that I don't understand by now.
13 years ago
Michael Christen 3eccdca63c protection against too long running snippet fetch processes
13 years ago
Michael Christen 86b3385847 fixed a deadlock during secondary remote search
13 years ago
Michael Christen c715d19c09 fixes for dependency on svn
13 years ago
Michael Christen 404758698a less io operations
13 years ago
Michael Christen 0bc5d76bee ups
13 years ago
Michael Christen 044f83feed added some pauses into the search process which shall produce
13 years ago
Michael Christen 943b670738 less terrible warning if uPnP fails
13 years ago
sixcooler 448656087a probably fix for http://bugs.yacy.net/view.php?id=94
13 years ago
Michael Christen f14faf503b better ranking because we wait a very little time during the search
13 years ago
Michael Christen 762e0ecfb6 fixed localization dictionaries, see
13 years ago
Michael Christen d35bdc2df6 removed npe
13 years ago
Michael Christen e7e429705a - less automatic indexing after a search (needs to reset the default
13 years ago
Michael Christen 9cd469e6d6 added pull request from als plus an NPE fix
13 years ago
admin 484c4ad339 Merge branch 'master' of git://github.com/f1ori/yacy
13 years ago
orbiter 402e9d71ef changed ording on release files: main criteria is not the svn any more; releases are now ordered by
13 years ago
admin 56ce8488e4 Merge branch 'master' of git://github.com/f1ori/yacy
13 years ago
orbiter 4b8ff84705 - search bugfixes (page counter and number of results per page; recognition of new search)
13 years ago
sixcooler aeeae75b8a the timeout of httpclient is not absolut, but till a connection is
13 years ago
hermens 2ac272cfbf Fix for PeerSelection.seedsByAge() for big networks (>1000 Peers)
13 years ago
orbiter 83335c3b09 fix for http://bugs.yacy.net/view.php?id=78
13 years ago
Al Sutton 39898cb94a Added try/finally protection to ensure streams are closed. Added initial size guess for the CharBuffer
13 years ago
Al Sutton 4c67a964a1 Added try/finally protection to ensure streams are closed. Added initial size guess for the CharBuffer
13 years ago
Al Sutton 3f9b9f953f Added close() to ensure buffer close actions are invoked
13 years ago
Al Sutton d73c84f9a0 Allow initial buffer size definition in TransformWriter, and use available() method to set it in htmlParser. In this situation a ByteArrayInputStream is used so the available() method gives a good size estimation and avoid the buffer needing to be continually grown
13 years ago
Al Sutton f02ea27b31 Added missing closure of ByteArrayInputSteam
13 years ago
orbiter 0796b54601 - some speed hacks for network image
13 years ago
orbiter f9216e388c - faster ping to clean up old peers faster
13 years ago
orbiter 35a9e8f307 - fixed network graphic
13 years ago
Al Sutton 8993cac4d8 Initial performance improvements
13 years ago
orbiter d9c066227a fix for npe
13 years ago
orbiter 8895d8c1cd removed unnecessary log entries
13 years ago
orbiter 550c881d80 remove more news (all older than one day) because they can be a performance problem if we have too many peers sending news
13 years ago
orbiter ebd840ebf6 - enhanced description on search front page
13 years ago
orbiter e22f8497c9 - tested the ARC methods
13 years ago
orbiter bc5df0eef5 updated ranking tables (fresh computation)
13 years ago
orbiter 5a55397f99 some last-minute performance hacks
13 years ago
orbiter c9216d5adf fixed secondary remote search (the process that finds distributed join situations)
13 years ago
orbiter 64fd20b857 new default ranking profile
13 years ago
orbiter 0cf9ebc3b0 speed enhancements when parsing RWI rows (makes search slightly faster)
13 years ago
orbiter c9a0dbd25a added a security check
13 years ago
orbiter ee8b1d4de1 fixed unresolved pattern and unwanted local/global switch when using votes on search results
13 years ago
orbiter c584db991f creating a bookmark from the search results now works again .. with new YMarks
13 years ago
orbiter 1120f0c93c update to network graphics: slightly less crawling activity, slightly stronger color for query activity
13 years ago
orbiter 6cd27473f5 - better default values for caching and cache usage
13 years ago
orbiter 709013385a fix for language fix
13 years ago
orbiter 1019c36dad bug fixes and speed enhancements for search
13 years ago
orbiter 507c9d478d much better timing when search globally; less blocking; more results earlier!
13 years ago
orbiter 8e0b2c5832 fixed cluster search
13 years ago
orbiter c0c6e9e7a5 fix for bad language encoding
13 years ago
apfelmaennchen 564374d1fe - included YMarks in addition to old bookmarks in yacysearchitem.html; don't get confused by the old bookmark dialog, the ymark is automatically added silently beforehand.
13 years ago
orbiter 05f34a3fa7 added a full, complete, database insert, update and delete API for the tables.
13 years ago
lotus 3cc93325f0 temporary remove compare search from tray
13 years ago
orbiter 804e48888b smaller bug fixes for search behavior; should produce less unnecessary removals and an exact number of results as shown in counter
13 years ago
orbiter 84c3fc9d97 local/global fixes in search, better abstraction
13 years ago
orbiter 06352b8d6b more logging
13 years ago
orbiter 017a01714d - enhanced logging in robots.txt parser for remote debugging
13 years ago
orbiter 3a15e58e28 - increased stability when opening the robots table
13 years ago
orbiter 775b44017e refactoring
13 years ago
orbiter e914a30099 fix for npe
13 years ago
orbiter 78ce3b13be typo
13 years ago
orbiter 85d6bf4ac4 fixed urls to media content during indexing
13 years ago
orbiter 0d858d48ec replaced String with StringBuilder in suggestion process
13 years ago
orbiter 3a807e10cf - added a cache for active crawl profiles to the crawl switchboard
13 years ago
orbiter 37e35f2741 normalization of url using urlencoding/decoding
13 years ago
orbiter e58438c01c - added a new retry connector for solr (for cases where solr responses are slow)
13 years ago
orbiter d8d9735b4f stability bugfix
13 years ago
orbiter c31564ef08 stability bugfixes
13 years ago
orbiter f121f4bb45 fix for link in Supporter and Suftipps page
13 years ago
orbiter 94eab08794 - updated opensearchdescription text and icon
13 years ago
orbiter 279482a76d fix for npe
13 years ago
orbiter 1b86d06d1e fix for http://bugs.yacy.net/view.php?id=62
13 years ago
orbiter eb9c9edb01 enhanced table method (used by almost all yacy api interfaces)
13 years ago
orbiter 4ad9fc2bff new snippet strategy for search hits in metadata: show beginning of text instead of hit position
14 years ago
orbiter a9838f8b99 fix for http://bugs.yacy.net/view.php?id=59
14 years ago
hermens d3df03838a make sure myself-target is always inserted at its appropriate position
14 years ago
hermens c3e7efa846 added sender side prevention of rwi flooding as mentioned in SVN 7993
14 years ago
orbiter 5af9598bd1 enhanced exported row parsing during row import
14 years ago
orbiter 7598a9e26b fix for thread dump
14 years ago
orbiter 8eef8722d1 update to ThreadDump analysis: freerunner and thread state recognition
14 years ago
orbiter 1df43b137d another performance hack
14 years ago
orbiter 7df0643f0e performance hacks
14 years ago
orbiter a7df70221e refactoring
14 years ago
orbiter 1b45e33f04 added robots tag parser to solr scheme
14 years ago
orbiter cf4fd525ee added directDocByURL attribute in crawl profile
14 years ago
orbiter 813f297a95 another performance hack: re-use of known host addresses for isLocal property; avoids look-up in local hash
14 years ago
orbiter 035ebfbf3b - performance hacks (should affect the crawl balancer and reduce CPU load during crawl stack re-fill)
14 years ago
orbiter b250e6466d implemented crawl restrictions for IP pattern and country lists
14 years ago
orbiter 57d5529a01 performance hacks
14 years ago
orbiter 47a8c69745 added a new feature to MultiProtocolURIs to get the locale for each url:
14 years ago
orbiter 2c3161b4ac refactoring:
14 years ago
orbiter d2ea250d99 refactoring:
14 years ago
low012 277b454a62 *) added comments
14 years ago
orbiter 6b22865dbc - removed some warinings
14 years ago
orbiter 0c6d95e57b - more tolerance against failure of table opening
14 years ago
orbiter 6b02b696b0 - add number of search results to end of rss and json output to reflect latest status of retrieval
14 years ago
orbiter ce2a76d603 performance hack for search process
14 years ago
orbiter aaf7a0feaa yet another cache strategy
14 years ago
orbiter 8a428d3e77 ensure termination of pdf parser to avoid deadlocking of other processes during search result preparation
14 years ago
orbiter 2c4a672fe2 bugfixes and performance hacks for tabe index
14 years ago
orbiter dad5b586a4 added a concurrent warmin-up of Table data structures. that should speed-up the start-up process but may also cause stronger CPU load at that time.
14 years ago
orbiter 734059d33e performance hacks
14 years ago
orbiter 23e81b28b2 synchronization enhancements
14 years ago
orbiter dd4635e323 patches
14 years ago
orbiter 85a5487d6d YaCy can now use the solr index to compute text snippets. This makes search result preparation MUCH faster because no document fetching and parsing is necessary any more.
14 years ago
orbiter 0819e1d397 protection against OOM cases in image parser. See also bugs.yacy.net/view.php?id=54
14 years ago
orbiter 2cba860693 - fix for wrong entries in NOLOAD indexing queue (that caused that urls had been only indexed based on their url and not loaded)
14 years ago
orbiter 2842ce30d6 added synchronization in ReferenceContainer and logging for shrinking
14 years ago
orbiter cec3836e73 added reference limitation to IndexControlRWIs_p.html servlet
14 years ago
sixcooler ecb4986b38 refactored stuff from last commit to ReferenceContainer
14 years ago
sixcooler f7c4abfdd7 limit references per blob & term to the 100.000 youngest
14 years ago
orbiter 28f5b79deb added a fast mass-deletion method
14 years ago
orbiter a70dbce41c added another file tool class to yacy-cora
14 years ago
orbiter 49e5ca579f added new configuration property "crawler.embedLinksAsDocuments". If this is switched on (this is default now), the all embedded image, audio and video links from all parsed documents are added to the search index as individual document. This will increase the search index size dramatically but will also enable us to create a much faster image, audio and video search. If the flag is switched on, the index entries are also stored to a solr index, if this is also enabled.
14 years ago
orbiter e02bfbde56 fix for solr url
14 years ago
orbiter 580beb12a5 reverting SVN 7863; the synchronization was needed and no synchronization causes repeated DNS lookup for the same hosts
14 years ago
orbiter 44d6416e2d ensure termination of shrink()
14 years ago
orbiter 52230a6864 replaced catching of Exception with Throwable, which catches also Errors
14 years ago
orbiter 877eaf6bcb switched off logging of org.apache.http which was suddenly switched on by default (??)
14 years ago
orbiter e1a3d609aa moved merger object from Segment to IndexCell to enable a correct shutdown sequence. This solves a bug where yacy cannot be shut down during an index merge that appears during the shutdown phase.
14 years ago
orbiter 610b01e1c3 - added a 'add every media object linked in a html document as a new document' to the html parser. This causes that all image, app, video or audio file that is linked in a html file is added as document. In fact that means that parsing a single html document may cause that a number of documents is inserted into the search index.
14 years ago
orbiter 3da21c4266 protection against starting of a (second) yacy peer while another one is already running on the same port
14 years ago
orbiter b5252ef91f added new word recommendation library in DictionaryLoader_p.html
14 years ago
orbiter 1c007188ad bugfixes in html parser
14 years ago
orbiter 231074bf0a fixed a parsing bug by reverting SVN 7766
14 years ago
low012 30a8a2f76b *) replacing one ugly hack with an extended ugly hack ;-)
14 years ago
low012 95379ce0b1 *) should fix some problems with RSS Importer (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3253)
14 years ago
low012 24e76a7b69 *) Replaced occurrences of "Wikimedia" with "MediaWiki" where applicable. (Thanks to the folks of 0x20.be for pointing this out.)
14 years ago
sixcooler d40a177c05 Generation Memory Strategy fine tuning
14 years ago
sixcooler 839f407fe4 Generation Memory Strategy fine tuning:
14 years ago
orbiter a5541751a8 - added memory computation to termlist_p.xml
14 years ago
orbiter 45e497a9bd fix for term iteration
14 years ago
orbiter 5dd2efc9a2 - bugfixes in html parser
14 years ago
orbiter 2c595a6a47 added new methods to count the number of objects in RWIs. lots of refactoring was necessary to introduce new Rating class and to unify naming of methods
14 years ago
orbiter 75df87832c refactoring/better naming of methods and classes
14 years ago
sixcooler 5f8a5ca32d - not doing merge-jobs while short on Memory
14 years ago
orbiter 965fabfb87 enhanced sorting speed (affects all DB operations)
14 years ago
orbiter 41a8ee4569 added iterable implementation in KeyList
14 years ago
orbiter 22d69a6368 refactoring in cora: added sorting package
14 years ago
orbiter 51cf697acd refactoring: moved all score-related classes to new ranking package
14 years ago
orbiter a0d5e7b6e6 added new score comparator
14 years ago
sixcooler 4fec99115b Implementation of strategies for controlling memory resources.
14 years ago
sixcooler 63a375b801 do not look at external dtd, cause this make this reader stay forewer(?) on on faulty dtd-locations
14 years ago
orbiter 2c58af6874 - added a short memory status simulation mode
14 years ago
orbiter c64faf41e2 addon to svn 7880
14 years ago
sixcooler 7b7a196243 ignore cookies in httpclient per default
14 years ago
sixcooler 411ed159f8 do some extra sleep while running low on memory
14 years ago
sixcooler 9ab0ba41e2 using GzipDecompressingEntity from httpclient instead of our own
14 years ago
sixcooler 07f5954570 try better handling of corrupt blobs
14 years ago
orbiter f970670a7c - bugfix in ServerScannerList
14 years ago
orbiter 8e03b8ee8b better integration of server list in interactive search
14 years ago
orbiter 0a3ab7da1b do not sort concrrently the same array
14 years ago
sixcooler eb14111200 encapsulate potential expensive objects in TextSnippet to allow GC them asap
14 years ago
orbiter 0d33cf352b removed synchronization in DNS resolve (solves a problem when loading snippets but in the past concurrent dns requests also caused deadlocks. but this is many years ago and we will give it another try)
14 years ago
orbiter 44d74f8f89 performance hacks for seed generation (because thread dumps showed multiple occurrences at these code points)
14 years ago
sixcooler 5cd07d7f84 early freeing resources on deleting index reference if search-verification fails (aka Switchboard.cleanupJob)
14 years ago
sixcooler a311596881 finishing up my commits (7855-7858) which could be helpful for
14 years ago
sixcooler 9170a434ed throwing an exception again in FileUtils.copy(reader, writer)
14 years ago
sixcooler ce248cc8dd less byte-arrays of response-content, less byte-array <-> stream conversation
14 years ago
sixcooler 59b767eebd stop loading via http at defined maximum of bytes - even size is unknown before loading
14 years ago
sixcooler 916d79111e Runtime.maxMemory() DOES change @ runtime:
14 years ago
orbiter 299af4943c added another memory protection hack
14 years ago
orbiter 1f300217f8 more protection for the cleanup thread
14 years ago
orbiter d13103a0a7 changed the way how the index cache is flushed: do not flush when a put was made because that could cause that many put calls synchronize for a long time when the dump or a merge is performed. Instead a watchdog thread is doing the dump and therefore puts cannot block any more which is good when a put happens during a search result preparation.
14 years ago
orbiter b06faab9d3 do not allocate a StringBuilder object in case that there is not enough memory for that
14 years ago
orbiter 6a6f27eaf3 do not sort arrays again if arrays are already sorted
14 years ago
orbiter 3d043ce9d6 - refactoring
14 years ago
orbiter 48b78e9ff4 disabling concurrency in new sort since that is not working yet correctly
14 years ago
orbiter 62ac73a108 fixed bugs and deadlocks in core database indexing structures:
14 years ago
orbiter 1912d0cccc changed handling of RowSet element retrieval: until today all elements had been copied from the underlying byte[] arrays into a new Entry object that again had a copy of a portion of that byte[] in its own bye[]. There was an option to just refer to the underlying byte[] with a pointer but that was almost never used. This commit now changes an interface to the Row class where it is now necessary to tell if a copy is always required. Fortunately the copy is only needed in very rare cases. That means that this change should cause much less memory allocation; it is expected that this happens especially during search situations.
14 years ago
orbiter bb8e3f8523 code cleanup
14 years ago
orbiter 11dc653de3 added a visualization of peer pings to the performance graphic
14 years ago
orbiter 3a191cdf14 because newbies are scared about the memory consumption in the performance graph and arguments about high memory consumption according to bad knowledge about java garbage collection techniques, the memory display had been removed from the performance graph shown on the Status.html page. The memory graph can still be seen on the Performance page where the memory graph is just like it was.
14 years ago
orbiter 52d799e7c8 fix for solr auth
14 years ago
orbiter 9eb8e9acd9 no error message about missing browser in headless environments
14 years ago
orbiter d3c89b90ce temporary adding the old httpclient-3.1 again because the solrj classes need them. should be removed as soon solrj supports httpclient-4
14 years ago
orbiter bd99969758 fixed bad query
14 years ago
orbiter 768c59740c - replaced solrj 3.1 with solrj 3.3
14 years ago
low012 c7b95e8c81 *) Invalid crawl profiles (containing invalid mustmatch/mustnotmatch filters) will be moved from active crawls to invalid crawls (new file: DATA/INDEX/freeworld/QUEUES/crawlProfilesInvalid.heap). This file can not be edited yet, but it shoudl be easy to extend the CrawlProfileEditor accordingly.
14 years ago
orbiter 6d2e252bcf fix for:
14 years ago
orbiter 2d4bb139d3 - added counting of links with noindex tag for solr index
14 years ago
orbiter 892caccdca added default configuration in ConfigurationSet in case of new values
14 years ago
orbiter bda3eec0ff added parsing of canonical link element to html parser
14 years ago
orbiter b6f09a475d - added an index profile editor in the /indexFederated_p.html servlet for solr indexes
14 years ago
orbiter b666a929e7 fixed Semaphore handling in case of interruptions
14 years ago
orbiter de7a054d77 added parser for such files like the new solr.key.list
14 years ago
orbiter 267290a821 removed the semaphores from the cache dump process because I believe some of the semaphores may be lost somewhere which then causes that the cache is never flushed and then the peer dies from a OOM. The re-introduced synchronization may not be the best solution but should ensure that the caches are flushed.
14 years ago
orbiter d8072d1866 added more info to DNS cache in /PerformanceMemory_p.html
14 years ago
orbiter f803da8aae code cleanup
14 years ago
orbiter 84c9658644 added a file type navigator
14 years ago
orbiter 31283ecd07 - added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search.
14 years ago
orbiter 7db208c992 performance hacks: more pre-allocated StringBuilder
14 years ago
orbiter 07e89a7ae5 added @Deprecated
14 years ago
orbiter 9706fc55aa enhanced content scraper (should discover urls much faster in case of very large plain texts)
14 years ago
orbiter 996f0a8764 disabled assert in Base64Order which eats away too much performance during testing with -l
14 years ago
orbiter f667b9c289 enhanced identificator: using AtomicInteger for counter
14 years ago
orbiter 16327d1cbe unwrapping of call depth (one call less for UTF8.String)
14 years ago
orbiter f30d36b101 enhanced template engine
14 years ago
orbiter aa6c32d753 enhanced UTCDiffString
14 years ago
f1ori f87865a50b always shutdown log, fixes zombie processes in init stop script
14 years ago
orbiter 115abc8917 - more attributes for search progress bar
14 years ago
orbiter 77fe69395d added jempbox-1.5.0.jar which is required by pdfbox-1.5 as stated in http://pdfbox.apache.org/dependencies.html
14 years ago
sixcooler df1725ef43 re-enable POST over proxy, which didn't work since update to httpcore-4.1.1
14 years ago
orbiter 2683162ec5 - added more options to access grid picture, web structure picture and network graphics
14 years ago
orbiter 0c1b29f3c9 - applied many small performance hacks
14 years ago
orbiter fe0c08455b more concurrency (enhancement) hacks
14 years ago
orbiter 87082f407e less String object creation during search
14 years ago
orbiter 3c2b994bd6 write access/load time to solr index
14 years ago
orbiter a36fda991e hack to increase speed of url hash computation
14 years ago
orbiter dbea40d536 - changed snippet fetch strategy logic: do not check if entry is in cache. This should reduce IO load on the HTCACHE which is a showstopper during large number of search requests
14 years ago
orbiter 4bea3f9714 hack to reduce resource contention caused by massive UTF8 decodings which use java.nio resources:
14 years ago
orbiter 746e3c3b06 Replaced a widely-used Property Object in the httpd with HashMap<String, Object> which is not synchronized like Properties
14 years ago
orbiter e28bd0d038 fix for some possible causes of memory leaks
14 years ago
orbiter 09ba6814c0 - non-blocking word hash computation with dynamic digest object generation (this was important!)
14 years ago
orbiter 10e2f588f8 - enhanced ybr ranking computation
14 years ago
orbiter bd55dcee50 - commented out experimental distributed ranking loading
14 years ago
orbiter 98c4d25185 fix for endless loop in FTP crawling, see http://bugs.yacy.net/view.php?id=32
14 years ago
orbiter 3ed4a09368 small features, some bug fixes and performance hacks
14 years ago
orbiter b45701d20f this is a re-implementation of the YaCy Block Rank feature
14 years ago
orbiter d27a0a67ff fix in log initialization according to hint from Dominic
14 years ago
orbiter 205cc75157 abstraction of surrogate main element (xmlns:geo was missing for wiki extracts)
14 years ago
orbiter 021840e5ba removed (almost) deadlocks and unnecessary CPU load
14 years ago
orbiter 123375bfba added a new yacy protocol servlet 'idx'. This returns an index to one of the data entities that is stored in YaCy.
14 years ago
orbiter 5c981762c6 added bigrange option for network scan
14 years ago
orbiter bade61696f speed-up of network port scanner
14 years ago
orbiter 1d8b0f74f4 one more fix for SVN 7713
14 years ago
orbiter 0960261769 fix for svn 7713
14 years ago
orbiter 5b579e21a3 code cleanup
14 years ago
orbiter 039126cfaf better handling of on/off switched solr indexing
14 years ago
orbiter dc54915df4 fix for very bad compare
14 years ago
orbiter 9248a4eef4 reduce teh effect of 'Bildersuche findet generierte HTML-Seiten als Bilder'
14 years ago
orbiter 76f2817e00 a fix for the snippet computation and hopefully better snippets
14 years ago
orbiter deda54d684 - relaxed matching of string-search (this is now case-insensitive)
14 years ago
orbiter 15e3a57b4e removed unused functions in condenser
14 years ago
orbiter 6e42d4de88 - added full-String search function: find things that match exactly what is quoted in the query
14 years ago
orbiter 8e10b82280 small fix for solr export
14 years ago
orbiter 6fa439c82b - refactoring of robots
14 years ago
orbiter e3d19d0a90 fix in Document inboundlinks/outboundlinks sorting
14 years ago
orbiter 4e8fa03514 added more attributes to html evaluation
14 years ago
orbiter 528da7c9ea removed unused class and added license header for new class
14 years ago
orbiter f6077b3cc0 added more attributes for html parser and enhanced data structures
14 years ago
sixcooler 4eb9c1e7c3 not setting userAgent from Constructor as default for following calls
14 years ago
orbiter d8e934c085 better abstraction of http client identification
14 years ago
sixcooler a3e707283d not using HTTPConnector anymore
14 years ago
orbiter 9f1f47ec67 added some comments to explain the isLocal patch
14 years ago
orbiter b77b8cac0c - enhanced html parser: recognized much more details in the content
14 years ago
orbiter 3d5104d357 - fixed a bug in crawl start with file name (npe in new url)
14 years ago
orbiter 958ff4778e enhanced location search:
14 years ago
sixcooler 8d63f3b70f just cosmetics - keeping my baby clean :-)
14 years ago
orbiter e402622584 removed httpclient-3.1 (this was added with last commit which was a mistake)
14 years ago
orbiter 19fd13d3bc Added federated index storage to solr.
14 years ago
orbiter c17d102bd8 enhanced speed for OrderedScoreMap inc method and size comparisment in concurrent environments
14 years ago
orbiter b788182954 some enhancements to scoring speed
14 years ago
orbiter 01690eab86 fix for mediawiki importer and wikicode parser
14 years ago
orbiter 4c013d9088 more UTF8 getBytes() performance hacks
14 years ago
cominch 9ac02caf00 different initialization of empty variables in alternative constructor. This leads to wrong interpretation of user credentials, resulting in unnecessary "@" in front of host, and different urlhash values.
14 years ago
orbiter 57ce1fb491 reverted synchronization from SVN 7641
14 years ago
orbiter 17530ca7b5 fix for bug http://bugs.yacy.net/view.php?id=10
14 years ago
orbiter 7c8e764201 removed synchronization again...
14 years ago
orbiter 96c32e87b0 fixes to crawler and new user-agent crawl-delay handling
14 years ago
orbiter cb6f709a16 - enhancements in surrogate reading
14 years ago
low012 1ff9947f91 *) added new user right: extended search right (allows to define users who can query more results than anonymous users)
14 years ago
orbiter 564184909a enhanced the surrogate parser: better reading of UTF-8 characters
14 years ago
orbiter 156cf02703 - added an index constraint 'has location' to the condenser
14 years ago
orbiter 41b8d7f655 fix for url normalization (no backpath resolving in post parameters)
14 years ago
orbiter 0430a94eaa the location search shows now not re-evaluated locations but only such locations that are attached as metadata to web pages
14 years ago
orbiter 8412f8787d fix for http://bugs.yacy.net/view.php?id=8
14 years ago
orbiter 9b25d07295 - added geo information parsing to html parser
14 years ago
lotus cbf87fe72f write PID to yacy.running
14 years ago
orbiter b1a8d0c020 enhancements to web cache and less strict caching rules
14 years ago
orbiter f3baaca920 - enhancements to DNS IP caching and crawler speed
14 years ago
f1ori df71776929 * fix bug #7
14 years ago
orbiter 78d4c45d09 enhancement during search process: fast fail of search in case that all index feeder have terminated.
14 years ago
orbiter a50f28e6e7 - fixed missing save operation for peer name change
14 years ago
orbiter 2b5f8585bf performance hack for Balancer and ip address parsing
14 years ago
orbiter b1d133b69f another anhancement to the ThreadDump function: better multiple dumps and filtering out of not interesting dump parts
14 years ago
orbiter a35d513bd8 fix for not-deleted .gap and .idx files
14 years ago
orbiter a6935e7dc8 fix for active dns resolving: do not resolve in case that the dns server is not available (offline mode)
14 years ago
orbiter 859c99886c fix for multiple thread dump
14 years ago
orbiter 61acf55da4 avoided using a synchronized(this) for the hash computation to prevent that the lock on the object is (accidently) stolen by another thread and replaced this synchronization using the protocol object. Made also the protocol object final.
14 years ago
orbiter c2a968c23f fix for bug in formatting in ThreadDump
14 years ago
orbiter 078ecacf61 avoid synchronization in DigestURI hash requests
14 years ago
orbiter 1989ebc24b removed more warnings
14 years ago
orbiter 0324de1467 removed debug line
14 years ago
orbiter 1aba7869bf patch for Windows: do not use the thread lock feature from previous commit if used on Windows
14 years ago
orbiter 0a11727374 added new feature for Thread dump:
14 years ago
orbiter b62b79675b removed type cast warnings
14 years ago
orbiter a07a1a8b1e removed type cast warnings
14 years ago
orbiter 8edaccfedf removed unused variables
14 years ago
orbiter e6c3507b17 disabled some of the previous changes (did not work in openjdk)
14 years ago
orbiter f9e5c21083 update to thread dump logs
14 years ago
orbiter 8f11d3a5bb redesigned the ScoreMap classes:
14 years ago
orbiter a564230c48 more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases)
14 years ago
orbiter dc0db3550e avoid string conversion
14 years ago
orbiter 694fa3a2a5 - replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion
14 years ago
orbiter 30aed9824a moved getBytes() to UTF8.getBytes() to use a default String encoding
14 years ago
lotus cb6d307bba adding extension for parser
14 years ago
orbiter 3820525464 more memory protection: auto-flush of caches in case of memory shortage
14 years ago
orbiter 7962d35425 - removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons:
14 years ago
orbiter 96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...)
14 years ago
orbiter e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations
14 years ago
low012 3b40b98256 *) set SVN properties
14 years ago
orbiter 619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word
14 years ago
low012 bf27a72d53 *) set SVN properties
14 years ago
low012 b649ce2dd7 *) minor changes
14 years ago
orbiter 70a996a06c reverted SVN 7557 because these classes are called using reflection. The class declaration is in the log configuration. Without these classes you get errors during runtime and a non-formatted log output, i.e.:
14 years ago
orbiter cb1f49d0f2 replaced all 'new String' with default encoding (missing) or UTF-8 encoding with a String generation method that uses a pre-defined Charset constant for UTF-8. This avoids a cache-lookup for the Charset object using String hashing of the String 'UTF-8'.
14 years ago
low012 9d366ee9d7 *) removed unused code (I assume that most of the code was really dead, but if you need any of the classes, tell me and I will put it back in.)
14 years ago
orbiter 7138f4036b less synchronization, better thread dump tool
14 years ago
orbiter 8d14916c74 more patches for a better out-of-memory management
14 years ago
orbiter c2c5b12882 - even less memory for circle tool
14 years ago
orbiter 6badc5e558 reduce size of static memory usage: use short instead of int in circle coordinates cache
14 years ago
orbiter ce0c8247fc removed (most probably!?!) superfluos System.err output
14 years ago
orbiter 799c534935 one more patch again OOM during secondary remote search
14 years ago
orbiter f8d0454c53 small bug fixes and experiments with search speed enhancement
14 years ago
orbiter 993b9bc1a8 memory/performance hacks, less synchronization, better concurrency
14 years ago
sixcooler 65bcc60808 stupid me: revert placement of closing connection which caused unclosed connections
14 years ago
sixcooler e3d75d6cd5 Not storing external header in an Header-Array and reduce a loop for its conversion.
14 years ago
orbiter 42d90664f3 - fixed a memory leak in the httpc.post method (no finish)
14 years ago
orbiter 38dce547c0 better concurrency (less locking on date formatting) more logging and minor bug fixes
14 years ago
orbiter 89d337841c more logging for OOMs
14 years ago
orbiter b1781d7aae some more performance hacks
14 years ago
orbiter b2f147d28e performance hack: excluded map encoding in many cases from synchronization block, especially when doing an iteration
14 years ago
orbiter 5e186e0122 continuing the fight against deadlocks during time formatting: better caching.
14 years ago
orbiter dec24244cf added convenience class to generate UTF StringBody objects with a default UTF8 charset.
14 years ago
orbiter 1110d16af9 performance hack: replaced generic row.getColBytes() call with row.getPrimaryKeyBytes() where the column is 0
14 years ago
orbiter 19b2a50578 - enhanced date formatter cache
14 years ago
orbiter 48a61c39a3 speed hacks in BLOB ArrayStack:
14 years ago
orbiter a92d80a545 performance enhancements using an alternative to a insensitive collator (a complex string compare):
14 years ago
sixcooler bcea497644 next try to fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3193&start=0&sid=b98aa9a7466397602b436eb45f4a9d39
14 years ago
orbiter ad7fcb9d61 Enhanced Base64Order transformation: less overhead (transformation between StringBuilder and byte[])
14 years ago
orbiter f95e50ec3d more explanation
14 years ago
orbiter bb36bf841a emergency commit (sorry sixcooler for not waiting) because without that automatic updating peers would not be able to do the next update.
14 years ago
sixcooler 8ad4e10491 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3193&start=0&sid=b98aa9a7466397602b436eb45f4a9d39
14 years ago
orbiter 0ce17d823a - fixed bug in ordering
14 years ago
orbiter dec4f36700 - fix for missing favicons in search widgets
14 years ago
orbiter 804ae2275b - do not delete idx and gap files if the heap is not modified
14 years ago
orbiter e3ef4e3021 - increased default peer ping time from 2 minutes to 1 minute
14 years ago
orbiter 5e45ded8e2 - removed locks from WordReference
14 years ago
orbiter cd19d0517e added dns resolve to HTTPClient POST using a dns cache to prevent that that not-thread-safe built-in dns cache inside apache http client is used
14 years ago
orbiter af87af0d4c - removed synchronization in serverSwitch which should improve speed
14 years ago
orbiter 57e6728cb7 - removed usage of /etc/alternatives/www-browser because of problems with lynx, see:
14 years ago
orbiter 91eeaf2cff fix in ftp client
14 years ago
orbiter e717bf74ba more logging, more care about OOMs
14 years ago
orbiter d84b4a072e healing for some OOM problems
14 years ago
orbiter 4aa406fb0f added log output to find bug in url parser for short hosts
14 years ago
orbiter 82f262f685 - enhanced circle drawing speed
14 years ago
orbiter 29dc416ac6 more animations in graphics. See network and access picture.
14 years ago
orbiter 93b9c4fbc9 added missing file for latest commit
14 years ago
orbiter 3e380c51b6 update to browser start with linux
14 years ago
orbiter 6083f2f171 fix for (false) oom
14 years ago
orbiter b35fda43ea more changes to headless mode; now non-headless mode is used when:
14 years ago
orbiter 6c52e31993 new methods to open a browser
14 years ago
orbiter 5892fff51f introduction of dht-burst modes: this can expand the number of target peers in some cases where a better heuristic is needed. The problematic cases are either when a muti-word search is made (still a hard case for our term-oriented DHT) or when a network operator wants that all robinson peers are asked. We therefore introduced two new network steering values that switch on more peers during the peer selection. Because the number of peers can now be very large, the number of maximum httpc connections was also increased.
14 years ago
orbiter 4588b5a291 - fixed document number limitation for crawls that restrict the number of documents per domain
14 years ago
low012 64f32e8f00 *) replaced all IPs in IP filters for proxy with the proper regular expression
14 years ago
sixcooler 3e8b72be50 update to httpclient-4.1 - sorry forgot some
14 years ago
low012 74b22dfa24 *) fixed bug which affected blacklist entries which consisted of domain _and_ path parts
14 years ago
orbiter fe93caac5a added flags and administration options to show advanced search and to show search result attributes (for each search result)
14 years ago
orbiter 431f780f41 patch for bad data in url metadata
14 years ago
orbiter 5905f912c5 replaced more double types with float
14 years ago
orbiter 0cdfb82963 replaced more appearance of double values by float values
14 years ago
orbiter eb12e15738 moved all Double values to Float values because of
14 years ago
f1ori 982aa689ef * fix StringIndexOutOfBoundException in WebStructureGraph
14 years ago
orbiter 88773e4daa changed the default port from 8080 to 8090
14 years ago
orbiter 991b92f4ae enhanced network graphic
14 years ago
f1ori a321c7673d * adminAccountForLocalhost only for localhost
14 years ago
hermens 930cb412dd Let SHORT_MILSEC_FORMATTER make a new formatted String every millisecond
14 years ago
orbiter 090c73e32e catch a OOM in HeapReader iteration
14 years ago
low012 48463c4507 *) General private License? ;-)
14 years ago
orbiter 6c1b14c8e1 - more control in access tracker: count number of returned search results (not only info how much is in the index)
14 years ago
low012 9f38c0023d *) Minor changes, mainly cleaning up a little bit, no functional changes.
14 years ago
orbiter 54e77e6255 refactoring
14 years ago
orbiter feefe17568 npe assert fix
14 years ago
orbiter 733903f2c9 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21489#p21489
14 years ago
orbiter 24e4126eee added JSON parser code from json.org (added generics to it)
14 years ago
orbiter 10ae8d961b - cora package has now no dependencies to other yacy packages and becomes a 'base' package (refactoring)
14 years ago
lotus 0e54233408 UPnP: map port again if we are not reachable (e.g. when router rebooted)
14 years ago
lotus b1484299b2 same units for memory observer configuration (MiB)
14 years ago
orbiter 387db84087 maybe found bug in non-working index dumper
14 years ago
orbiter a4c9d27287 - moved some variables from Stwitchboard to new class AccessTracker
14 years ago
f1ori e4aabaa1c3 * fix negative filelength for files >2G
14 years ago
orbiter cdfe8afe3f fix for really bad table iteration implementation: reduction of IO
14 years ago
low012 9eae33f886 *) Ooops...
14 years ago
low012 a001e8075c *) minor enhancements
14 years ago
low012 11ea966f9e *) added SID file (Commodore 64) sound file parser
14 years ago
orbiter b2ed4cfaf8 more small bugfixes and light refactoring
14 years ago
orbiter 3ca06d6290 patch for http://forum.yacy-websuche.de/viewtopic.php?p=21460#p21460
14 years ago
orbiter 903c824c2c - allow only scanned resourced with granted status
14 years ago
low012 936e976c23 *) added FreeMind (http://freemind.sourceforge.net/) mindmap parser
14 years ago
low012 3d95981f7d *) cleaning up the code a little bit
14 years ago
low012 2a6499364d *) minor changes
14 years ago
low012 c0274bd123 *) minor changes
14 years ago
orbiter fe46536f6e enhanced network scanner (less name resolving during scanning and no name resolving during search)
14 years ago
orbiter e753027c43 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21439#p21439
14 years ago
orbiter bf4ef1513e - fix for map view
14 years ago
orbiter 6b70393d1d - new java version 1.6
14 years ago
orbiter e88c428008 fix to ftp loader
14 years ago
orbiter 59b70a5a92 another fix to the ftp crawler: now correct directory listings according to rfc2640 (path with spaces) and better title names for such files
14 years ago
orbiter 9b25a33fd9 - fixed numerous bugs
14 years ago
orbiter 7bdb13bf7f more fixes to smb crawling: better file names
14 years ago
orbiter 94c48500cc several fixes
14 years ago
orbiter 0ac7311a62 fix for token parser
14 years ago
orbiter 58b59f9bc8 - a collection of bug fixes and some redesign of the Scanner class
14 years ago
orbiter c288fcf634 redesigned CrawlStartScanner user interface and added more features:
14 years ago
f1ori 9d2159582f * fix system update if urls are in blacklist (for example for very general blacklists like *.de)
14 years ago
orbiter 56264dcc17 - added CamelCase parser to MultiProtocolURI: generate better to-be-indexed words from urls
14 years ago
orbiter 99a7fe87f9 - removed old intranet scanner (the generic scanner now completely subsumes the old one)
14 years ago
orbiter acab6801d9 added new network scanner
14 years ago
orbiter 14e4fae8e9 fixes to ftp client
14 years ago
orbiter a563b05b60 enhanced crawler:
14 years ago
orbiter c36da90261 added a very fast ftp file list generator to site crawler:
14 years ago
orbiter db99db4be9 some redesign of the search-fail-response mechanism:
14 years ago
f1ori 4915d1781a * use local backup-file, if remote network-definition is not availible
14 years ago
orbiter 4e2c14efbb fixed bugs in parser and ftp client
14 years ago
orbiter d78e322e84 added a directory-structure reader to ftp client
14 years ago
orbiter f0651e5f2f added image search to yacyinteractive.html
14 years ago
orbiter b769cce433 - added a catch-all parser for all documents that cannot be parsed: they will contributed with their document url for the search index only
14 years ago
orbiter 21e84539e8 one more fix to Domains
14 years ago
orbiter e192d61972 fix for latest commit
14 years ago
orbiter 22453b13ad implemented local host address discovery as posted in
14 years ago
orbiter cc6499bf8d - added http://blekko.com as search heuristic (like scroogle). This was easy since they deliver their search results also as rss feed
14 years ago
orbiter a9f754c45f removed unused CR accumulation and distribution process
14 years ago
orbiter 3d945bb442 fix for ftp client: suppress bad directory listing time-out
14 years ago
orbiter d4a1a1850b removed warnings
14 years ago
low012 9b3fae9496 *) cleaning up the code a little bit
14 years ago
orbiter 321eb012fe removed two warnings and reverted one change
14 years ago
f1ori fd74bc388c * fix small bug in sessionid-removal
14 years ago
low012 eb79b952ef *) cleaner code
14 years ago
low012 38fdf43587 *) renamed classes according to standard Java coding conventions
14 years ago
low012 025e3f4790 *) renamed classes according to standard Java coding conventions
14 years ago
f1ori a025b1da89 * fix bug when browsing local filesystem (e. g. repository) with yacy
14 years ago
sixcooler b87bf88ac8 using less memory on merging and rewriting blobs
14 years ago
f1ori d62e449a11 * fix FilterEngine, forgot comparision-operator
14 years ago
orbiter 441fbc26e2 security patch for WeakPriorityBlockingQueue (produced a deadlock)
14 years ago
orbiter 5dcb838293 - removed thread overhead when calling dns services
14 years ago
orbiter 4c50d3428e smaller file size for array stacks to support smaller deletion sizes
14 years ago
orbiter becc463d8a enhanced did-you-mean
14 years ago
orbiter 93c535d111 fixed http://forum.yacy-websuche.de/viewtopic.php?p=21113#p21113
14 years ago
orbiter 04932dc268 added rdf data structure for rss feeds
14 years ago
orbiter 84f2953cd8 fix for rss loader / rss type recognition
14 years ago
orbiter 4c72885cba added a sitemap entry parser and loader for sitemaps
14 years ago
orbiter 445619f3ec added a submenu ConfigHTCache_p.html to set the size of the HTCache separately from the proxy configuration.
14 years ago
sixcooler 85c65475fa smal but important correction of last commit @ HTTPClient
14 years ago
f1ori acd93b1b31 * add failsafe mechanisme to domainlist retrieval
14 years ago
orbiter 70c95608d4 Added CORS Access header for yacysearch.rss output
14 years ago
f1ori def4253555 * add option to network definition to provide a domainlist (syntax like in blacklists)
14 years ago
orbiter fb92f9ae8e added mime type image/jpeg (image/jpg is wrong but it is left here because it does not harm and this error also exists in configuration of web servers)
14 years ago
orbiter 155d556568 - better memory protection
14 years ago
f1ori 7d8de34778 * add a bit documentation to DigestURI, use DigestURI(string) instead of DigestURI(string, null)
14 years ago
orbiter e3e3b49d52 - enhanced main release recognition
14 years ago
orbiter 58e74282af added a word counter statistic in condenser which is used by the did-you-mean to calculate best matches for given search words.
15 years ago
orbiter 863065abc4 added user agent logging to access tracker
15 years ago
orbiter ed4371dcf3 enhanced navigation implementation and enhanced tag cloud computation
15 years ago
orbiter ca738ac924 - added a tag cloud to search results (using the topics)
15 years ago
orbiter e4d561971e added more score cluster options and made score cluster usage more transparent
15 years ago
orbiter 7cd9d9d22a - enhanced DidYouMean computation using a faster count on index entries; this causes that results can be ranked better
15 years ago
orbiter de722090b5 enhancements in did-you-mean guessing
15 years ago
orbiter 24f1cba7b2 performance hacks:
15 years ago
orbiter d607b30b6a performance enhancements for search and code review for database functions
15 years ago
orbiter fcd40cd30f - disabled domZones (buggy, must think about better solution)
15 years ago
orbiter ec38eca278 fix for new URI equal method
15 years ago
orbiter 0d363a94d7 more performance hacks
15 years ago
orbiter b8aee6d402 performance hacks for better search performance
15 years ago
orbiter 091dd3f6ec - enhanced intranet search speed
15 years ago
orbiter aacf572a26 - enhancements for search speed
15 years ago
sixcooler 61c82f3105 gzip-compresson @ transferRWI & transferURL back again
15 years ago