Commit Graph

4257 Commits (36bd843ece3cae998991a025a16029646daccc62)

Author SHA1 Message Date
orbiter 36bd843ece for for RFC5322 comformance as suggested by Quix0r in http://forum.yacy-websuche.de/viewtopic.php?p=19585#p19585 15 years ago
orbiter c855fc48c6 only load robots.txt for http and http protocol 15 years ago
orbiter 748abfcffa added patches to prevent yacy-protocol DoS settings 15 years ago
orbiter e820ed061a avoiding excessive DNS lookups to determine localhost 15 years ago
orbiter 11983bc936 redesigned some parts of the parser entry point: 15 years ago
orbiter de88200e11 - added Byte Order Mark recognition to serverObjects 15 years ago
orbiter 89b4fff1c2 adopted ant script for new exif library 15 years ago
orbiter 24e5faee75 added exif parsing for jpg images 15 years ago
orbiter 82f76e1296 removed log line 15 years ago
orbiter 0f8004f9da enhanced html parser to recognize a href tags inside header tags 15 years ago
orbiter 3300930fc5 - (almost) fixed FTP crawler 15 years ago
orbiter 1198b9989d bugfixes, more sorttable 15 years ago
orbiter 9623d9e6d2 added a smb loader component for the YaCy crawler 15 years ago
orbiter ae2f3f000f better handling of table copy abandon .. prevent memory leak 15 years ago
orbiter 0769517129 added a robots.txt monitor in the crawler monitor submenu 15 years ago
orbiter 48995e71c4 added soft-auth to general authentication scheme 15 years ago
orbiter 72f00dee59 removed never-used server access account function 15 years ago
orbiter 57e1eae95e longer time-out for url fetching .. may help to show all that links that the statistic say for a search result 15 years ago
orbiter 9e639603e3 after frequent occurrences of 100% CPU usages and permanent blockings I try to disable a function in a method that may cause the problem when calling an external library (apache http client 3.x). The thread dump that shows the problem is attached here. 15 years ago
orbiter 4144927d94 show less errors 15 years ago
orbiter b88f5fbb4b slightly changed crawling policy 15 years ago
orbiter de01fe0e6d fix for bug in url parser 15 years ago
orbiter 7684a575c4 fix for deletion of error database each time when YaCy starts up 15 years ago
orbiter f561e340c6 show more results of single domains when not authorized fully (up to 100) 15 years ago
orbiter c4bdb1e7f2 added one more option in ViewFile to show an iframe like for the orginal web page content but using the cache than the direct link to the content in the web. Upgraded the very old and previously not any more used CacheResource_p servlet to a new and working version. 15 years ago
orbiter c09a995930 better logging of double occurrences of urls in the crawler 15 years ago
orbiter 1bbe14d23f SVN 6716 unfortunately contained parts of the unfinished SMB integration. To fix compile errors the remaining parts of the SMB implementation stub is added with this commit. 15 years ago
orbiter 884b262130 - added a new Wiki Namespace Navigator 15 years ago
orbiter 617dfbbd06 allo 'authorization by encoded password' also if requesting client is not from localhost but from the same host as yacy is running on. 15 years ago
orbiter 270fb38674 - fixed some bugs in Table viewer 15 years ago
orbiter 599c3766c4 added authentication to automated API call 15 years ago
orbiter 727dd9b193 - fixed a bug in robots.txt parser 15 years ago
orbiter 54af9e6b49 - added parsing of robots meta-tag in html headers to detect a noindexing request 15 years ago
orbiter 46c4f8b68a better look-ahead into the crawl queue: show more on crawl monitor 15 years ago
lotus 7b546415dc added svn6695 for windows 15 years ago
orbiter f175f9a2d3 changed way how number of search requests are counted: 15 years ago
orbiter 84222e3b4f fix for auto-updater: delete old libraries before copy of new one 15 years ago
sixcooler cd6de83905 next try for for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2703 15 years ago
sixcooler bfe4693e9a fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2703 15 years ago
orbiter 93b7ddc27d fix for http://forum.yacy-websuche.de/viewtopic.php?p=19376#p19376 15 years ago
orbiter 8030ed3319 self-healing for lost crawl profile handles 15 years ago
orbiter e3e5e05ec2 fix for problem in ranking setting which was caused by the introduction of a toString() method in serverObjects 15 years ago
orbiter e3ccfb54aa fix for display problem in Firefox on MacOS X 15 years ago
orbiter 564927ce72 redesign of CrawlResult data structures because of OOM occurrences during URL deletion processes. 15 years ago
orbiter 30c8185139 fix for sid check 15 years ago
orbiter ef62d017e5 integrated session id filtering for crawler 15 years ago
orbiter d8d9984913 added framework for session id filtering (not ready yet) 15 years ago
orbiter 2bc36de336 - fix for bug in svn 6669 15 years ago
orbiter d378ca4604 better handling of concurrency in seed 15 years ago
orbiter 6538043d89 fix for http://forum.yacy-websuche.de/viewtopic.php?p=19189#p19189 15 years ago