Commit Graph

717 Commits (ac492fa2a57a64db1a7300523d7bf1eceaffac6f)

Author SHA1 Message Date
danielr d4bce6affd refactoring (initialized static fields, removed empty if/else, serialized some fields in serializable classes)
17 years ago
danielr be2c9c07ff escape some unescaped characers in URLs (fixes problems with proxy)
17 years ago
danielr ec84a52adb change for problem with NPE (seen as "PROXY Unknown Error while processing request")
17 years ago
orbiter ff755fb858 small corrections and enhancements after search timing profiling
17 years ago
danielr d1ee231866 HTTPC close more unused connections
17 years ago
danielr 181796cffb - HTTPC ConnectionInfo entfernen bei Exceptions, unnötigen Code entfernt
17 years ago
danielr 64c33e717f catched ConcurrentModificationException in ConnectionInfo.cleanUp so cleanUp is not interrupted
17 years ago
danielr d8677ba611 fixed ConcurrentModificationException in HttpConnectionInfos
17 years ago
orbiter 8dd35f74c8 fixed redirect problem (does not work for POST)
17 years ago
danielr 2617f4dcdb Connections_p.html: better formatting and remove very old entries
17 years ago
orbiter 82bf9ac1c8 - added Collage servlet from datengrab and modified it:
17 years ago
danielr 959f448e5f - disabled redirects in proxy (so client sees real path)
17 years ago
orbiter 8fe39ebd74 -fixed file transmission with POST. The only usage was in ranking transmission, therefore:
17 years ago
orbiter 202a3adb3e refactoring of HttpClient Writer processes
17 years ago
danielr 8aa9fd8f24 HTTPC with only 1 retry
17 years ago
orbiter 444dce7e81 more performance hacks
17 years ago
orbiter 2c2dcd12a2 - enhanced performance of Eco-Tables: less time-consuming size() - operations
17 years ago
orbiter e356625b22 - refacotring of stream copy handling to support time-consuming operations
17 years ago
danielr f01c50cf8d Proxy logging error (first step to resolution!?)
17 years ago
orbiter c3342e1178 - removed class with only one static method
17 years ago
orbiter f97971b63b fixed NPE problems doing a shutdown from command-line
17 years ago
danielr 7a35126e91 http timeouts von alten httpc wieder gesetzt
17 years ago
orbiter 2c1c3bb6eb - some refactoring (sorry Daniel, hab in deinem Code rumgewütet)
17 years ago
danielr d96e2badc7 - fixed POST in proxy
17 years ago
danielr 94d3d3a86f fixed Proxy (for GET, POST still does not work!)
17 years ago
danielr 8b2efb6f8c fixed garbage in HTCACHE
17 years ago
danielr fb541f9162 HTTPC: default timeout half-hour
17 years ago
danielr a94f6cdca4 HTTPC: allowed self-signed certs
17 years ago
danielr ab330cfdca Network.html: removed ; from location
17 years ago
danielr 5c3c1fdf41 replaced httpc with Apache Jakarta Commons HttpClient (includes some refactoring ;)
17 years ago
orbiter daa04f5db9 added additional check in file handler to prevent that url attacks are hidden in url path encodings
17 years ago
orbiter 7f9f639d20 - refactoring and abstraction of index reference (urls) handling: blacklisting is part of reference filtering
17 years ago
orbiter d6050b9ffb - separated the LURL data storage and Crawl result stack for process supervision.
17 years ago
orbiter 541b817502 refactoring of switchboard queueing
17 years ago
orbiter 4c584dff87 disabled soLinger to prevent that too many connections stay open (it's a TEST!)
17 years ago
orbiter 9c989fe5f7 fixed deadlock
17 years ago
orbiter fa1090113d - next try to fix the networking problem:
17 years ago
orbiter d87d295c68 one more try to fix the connection problem
17 years ago
orbiter 9eddc1506b - one try to fix the httpd problem
17 years ago
orbiter 7cc4ff05c9 some code enhancements and bugfixes
17 years ago
borg-0300 3445b1e10b *better logging
17 years ago
borg-0300 4b0339fec0 *fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=927
17 years ago
orbiter 275a226cc5 refactoring
17 years ago
orbiter 4fdf695064 - fixed a bug in remote search that prevented that any results had been generated (!)
17 years ago
orbiter 3f321ece7d added a search history to the new search page
17 years ago
orbiter 87a8747ce3 - enhanced recognition, parsing, management and double-occurrence-handling of image tags
17 years ago
orbiter 6c3cd2b4f2 - added new way to watch images from the image search:
17 years ago
orbiter bd63999801 - faster search: using different data structures that avoid multiplr calculations
17 years ago
orbiter acf771d5e1 - fixed bug with too much RAM in crawler queue
17 years ago
orbiter 0f5c4abaca more generics
17 years ago
orbiter 1a296af6ff more generics
17 years ago
orbiter 4a80902081 - added ViewProfile as rdf in foaf syntax
17 years ago
orbiter 15397298dc - refactoring of indexControlRWIs: moved statics to own class; better Dublin Core naming
17 years ago
orbiter 03e7782269 more generics
17 years ago
low012 ae6d07bdb8 *) "Did you mean:" will only be displayed if the list of suggested URLs is not empty.
17 years ago
orbiter df2a7a8ac8 more generics
17 years ago
fuchsi d517e96714 last cleanup bits to serverDate before the release. only safe refactoring (method renaming) changes outside of serverDate.
17 years ago
hermens 4748d5c1ab Some enhancements to time management:
17 years ago
fuchsi f41172f850 Merge httpDate into serverDate as suggested. Removed some unnecessary code and fixed a possible synchronization problem.
17 years ago
fuchsi 21f7e13fa1 fix stupid tiny bug introduced in rev 4276 that broke request URL parsing almost completely
17 years ago
fuchsi 5d406d0094 - fixed url "file extension" parsing when there is no extension (like http://yacy.net/ would have extracted .net/)
17 years ago
fuchsi 21b8d1b918 small cosmetic change for static fields in serverCore (special protocol ASCII entities) to improve readability
17 years ago
orbiter ca488e03f5 fixed authorization case
17 years ago
orbiter e22014dc83 some memory enhancements when generating and displaying ymage objects
17 years ago
fuchsi 39d0f10ca1 Fix parsing oof dates in HTTP headers.
17 years ago
orbiter 9b0ae4b989 added referrer to remote crawl url list
17 years ago
orbiter af10f729df fixed image search and favicon loading
17 years ago
orbiter c527969185 - enhanced monitoring of ranking parameters
17 years ago
low012 383dc815d2 *) fix for commit 4212
17 years ago
fuchsi 425e4ead66 Allow absolute paths in configuration settings.
17 years ago
orbiter a31b9097a4 preparations for mass remote crawls:
17 years ago
fuchsi 0e1738899f * Complete number localization and provide a more reasonable interface to serverObjects:
17 years ago
fuchsi f717beecb1 - Changed yFormatter handling to be more flexible and produce more readable code for server pages. There are serverObject.putNum() methods to allow adding of number type values in a formatted form, and put() methods for number types that add them without formatting. This reduces the need to transform them into Strings in server pages and removes the HTML encoding step which is unecessary for numbers.
17 years ago
orbiter 711641f167 extended client connection clean-up:
17 years ago
orbiter b19bb6e5b1 - reverted svn 4132; this did not solve the problem and removed the emergency mehtod which caused production failure for shure within some hours
17 years ago
fuchsi 1eba408d2f Make sure that sockets which couldn't be opened aren't handled as active connections, in which case they wouldn't be closed.
17 years ago
orbiter d69d386f7d added additional forced client connection closing
17 years ago
orbiter dea7bee049 - increased minimum time before an active connection is interrupted from 1 minute to 10 minutes
17 years ago
orbiter 01e0669264 re-designed some parts of DHT position calculation (effect is the same as before)
17 years ago
orbiter 842308ea97 - redesigned crawl start menu, integrated monitoring pages
17 years ago
orbiter 2f1ff048ba some fixes to socket connection time-out
17 years ago
orbiter 3c74014004 automatic deletion of dead client connections
17 years ago
orbiter 11b4f80bde - fixed non-closing client connections
17 years ago
orbiter d352853f2d fix for non-closing client sessions
17 years ago
orbiter 1488769e1f cleanup of unmaintained and outdated performance methods:
17 years ago
orbiter 3cb9cdc9be try to fix connection problem, possible cause for wrong junior status and non-passive passive peers:
17 years ago
fuchsi e192f99134 fix small bug introduced in r4089 that appeared when we tried to remove "gzip" encoding from Accept-Encodings header
17 years ago
fuchsi ae4b9308ef Fix problems with some web servers which couldn't handle the way yacy was sending requests. Thx to celle for the patch.
17 years ago
orbiter daf0f74361 joined anomic.net.URL, plasmaURL and url hash computation:
17 years ago
orbiter 4779f314fe first version of next-generation search interface:
17 years ago
orbiter 6d759ad0a7 - new bot address
17 years ago
orbiter f9e6cf6a3d more refactoring of search:
17 years ago
orbiter bb426565f0 added new yacy protocol for mass url-pull for better remote crawling distribution
17 years ago
orbiter b5346141b3 made the plasmaHTCache static (there is only one internet, so we need only one cache)
18 years ago
orbiter 61f93cbf14 some code-cleanup
18 years ago
orbiter 24e25e1141 enhanced SSI server-side support:
18 years ago
orbiter 57a5b6fa71 some generalization of remote proxy configuration and setting handling in httpc
18 years ago
orbiter 367fc28928 corrected Brausse->Brausze
18 years ago
orbiter e76fe1c078 - replaced unicode characters in copyright holder name ('Brausse')
18 years ago
orbiter 75d1437340 fix for http://forum.yacy-websuche.de/viewtopic.php?p=1123#p1123
18 years ago
orbiter 9ca46a8c69 indexing of local (intranet) urls enabled
18 years ago
orbiter 6758beae9c fix for http://forum.yacy-websuche.de/viewtopic.php?p=1092#p1092
18 years ago
orbiter 40b0547611 - documentaton changes (removed old forum links)
18 years ago
orbiter 26ddf797eb added bmp and ico image format to all parser/viewing methods
18 years ago
low012 1ea5fa2c04 *) Changed a comment to get rid of this message:
18 years ago
orbiter a9e73b6852 fixed great mess with localization paths. the problem was:
18 years ago
orbiter 36a37f758b fix for oom exception during release download
18 years ago
orbiter 84be912e90 fix for null pointer exception that occurred when missing user-agent in request header
18 years ago
orbiter e03fcf4627 SSI fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=29
18 years ago
orbiter 9bbd39b67c - removed unfinished auto-updater from roland and martin
18 years ago
orbiter 154ffd7c2c fix for wrong http connection version and SSIs
18 years ago
orbiter 1782ef57e5 - added SSI parser and include directive for <!--# include virtual="<file>" -->
18 years ago
orbiter 0e57a8062b added network definition for different YaCy networks
18 years ago
orbiter 6518bb6c08 changed release strategy:
18 years ago
allo 465145cb6f revert to insecure, but dau-proof defaults
18 years ago
allo 7ad11ceaaa security fix for peers without password. allow access only from localhost
18 years ago
orbiter e4aa8f2a08 disabled more sleep(200)
18 years ago
orbiter cb38e57622 reduced httpd final waiting time
18 years ago
orbiter b4585ad67d im Sommer 2005 wurden die ersten pings zwischen YaCy-Peers ausgetauscht.
18 years ago
karlchenofhell 669f840eab - added ViewProfile / Impressum (default on) to local peer's robots.txt
18 years ago
orbiter 66ec8b63c1 added a httpd access tracker:
18 years ago
orbiter 33ad0c8246 added a web structure computation and logging:
18 years ago
karlchenofhell 601fc7d1c5 - added source to J7Zip-modifed.jar and it's license (changelog is still to come)
18 years ago
orbiter 26f05d1fd0 avoid division by zero if search is done for no words
18 years ago
orbiter 2fa8b50e54 reverting svn 3691+3692
18 years ago
orbiter 22a0e9f117 more timeout-control
18 years ago
orbiter 24db55a541 added timeout for httpd-sockets during read
18 years ago
orbiter 111ba9e359 - fixed some width problems in new status page
18 years ago
orbiter 29fe2beac7 possibly fixed a deadlock
18 years ago
theli c2e6afbd69 *) bugfix: setting mimeType properly for dir listing with e.g. "?format=xml"
18 years ago
theli f20b596dc0 *) adding servlet to display all deployed SOAP Services
18 years ago
theli 81b4598487 *) peer profile can now be displayed as vcard
18 years ago
theli 91c2a042a7 *) bugfix for wrong proxy traffic accounting
18 years ago
orbiter 5b0a84ce09 fix for synchronization deadlock with flushMissNameCache.
18 years ago
orbiter a1fb8358b2 lets make a well-formed http link so that other crawlers don't have a problem to follow this link :-)
18 years ago
orbiter 4edb70f68b added yacybot info-page from Roland
18 years ago
orbiter d755a8026d - better OOM protection
18 years ago
karlchenofhell 88245e44d8 - improved version of robots.txt (delete your old htroot/robots.txt before updating):
18 years ago
karlchenofhell a1d68fe092 - use .class rather than Class.forName for classes in class-path
18 years ago
karlchenofhell 6fbe31425a - some code-cleanup (no more syntax-warnings here)
18 years ago
karlchenofhell c016fcb10f - added streaming-support to CrawlURLFetchStack_p servlet
18 years ago
orbiter bf69a721cb more protection against mis-use of YaCyHop interface:
18 years ago
orbiter c464157a6e replaced some toString()
18 years ago
orbiter b4aa195c27 added user-agent check for yacy-hop proxy authentication
18 years ago
orbiter d25caa07bf redesigned some parts of http authentication
18 years ago
karlchenofhell 2401e748a3 - fixed wrong replacement of POST-parameters in httpd ('<' and '>' are still replaced, don't know why): http://www.yacy-forum.de/viewtopic.php?t=3466
18 years ago
karlchenofhell e68cdeeeb3 - reverted parseArg(String) to use a byte-array to handle correct UTF-8 parsing
18 years ago
orbiter 47ab83a7c0 added flag for YaCyHop - proxy access for all paths that start with /yacy/
18 years ago
allo 25c7d4e25e fix for form (cookie) login
18 years ago
karlchenofhell 7c40197e42 - fixed error pages and <label>s for index.html
18 years ago
allo b4457763e5 fix for putSafeXML and supertemplates.
18 years ago
allo 0c81bd39d4 XSS-safe put as default.
18 years ago
orbiter 5515571950 redesign of ymage classes
18 years ago
karlchenofhell b873ad51ab - fix for http://www.yacy-forum.de/viewtopic.php?t=3369
18 years ago
karlchenofhell 340dc52a9d - ConfigProfile_p.html now transmits usable encoding for other than 7-bit ASCII charset, see TODO in httpd.parseArg(String)
18 years ago
karlchenofhell 00aa9472d6 - added decode of HTML-entities in request lines
18 years ago
orbiter 0a050bc043 enhanced ranking
18 years ago
orbiter d0c32c6aeb better protection against fraud peers
18 years ago
karlchenofhell e17591acc3 - parse HTML arguments as UTF-8 strings
18 years ago
karlchenofhell d30932c7d8 - fix for fix... sry
18 years ago
karlchenofhell 6118fb73ec - added decode of UTF-16 escapes in url-arguments (%u0123), bugfix for http://www.yacy-forum.de/viewtopic.php?t=2762
18 years ago
orbiter fb7902aa68 fix for http://www.yacy-forum.de/viewtopic.php?p=26142#26142
18 years ago
orbiter 984285bdd6 better organisation of dns hit/miss cache flush
18 years ago
orbiter 73c63578ad - activated the dns miss cache
18 years ago
orbiter e3d75f42bd final version of collection entry type definition
18 years ago
orbiter d34f10c63d some tests with reverse dns lookup
18 years ago
(no author) a51417d86b Bugfix: language of ConfigLanguage_p.html was not changed properly when a different language was choosen here
18 years ago
theli f77d624b94 *) bugfix for persistent connection support on transfer-encoded requests
18 years ago
orbiter 114a76a86e - added flag to urlhash that shows that domain is a local domain
18 years ago
(no author) e59ff8b657 Bugfix: language of ConfigBasic.html was not changed properly when a different language was choosen here. Note: there's a similair bug on ConfigLanguage_p.html
18 years ago
theli 29a1f132ec *) some strings replaced by constants
18 years ago
orbiter 215c4e65f1 code cleanup
18 years ago
theli 532c23b5c7 *) soap handler
18 years ago
theli 777e39cea0 *) new template to display the dir-listing in xml format.
18 years ago
theli 88cfdecd38 *) Bugfix: calling close must not close the wrapped input stream, otherwise
18 years ago
allo 8a5c2d0a19 fix for supertemplates, too.
18 years ago
allo c35793fb46 fix for last commit
18 years ago
allo a831c83025 create servletProperties, with the servlet specific funktions from serverObjects
18 years ago
orbiter 8b56887676 removed unused code
18 years ago
theli 68204ff729 *) Suppressing for bad client requests.
18 years ago
theli df49724f28 *) better error handling for seed upload - test download - problems
18 years ago
theli b357a13e9a *) adding synchronization block because SimpleDateFormat is not thread-safe
18 years ago
orbiter 688cbfb776 - bugfixing for flextable bug
18 years ago
allo a29b4d4fb5 extended Supertemplates for Headerincludes.
18 years ago
theli a7e11ada50 *) suppressing stacktrace for "server has closed connection"
18 years ago
orbiter c8f3a7d363 added snippet-url re-indexing
18 years ago
allo 226f2c5b2c first version, of the Serverlet Debugger
18 years ago
theli ce7ee74316 *) better errorhandling in filehandler (try catch block now starts before argument parsing)
18 years ago
theli f17ce28b6d *) plasmaHTCache:
18 years ago
orbiter 5a40ea7866 refactoring of wget string list generation
18 years ago
orbiter 310f1c41cd added option to see ranking scores in surftipps
18 years ago
theli cd5f349666 *) Better handling of large files during parsing
18 years ago
orbiter df1629b05a - code cleanup
18 years ago
theli c665f6cddb *) handling of quotes in charset string
18 years ago
theli 009a33170b *) Content-Location header added
18 years ago
theli 1aa07a52cd *) Bugfix for UnsupportedEncodingException if the media type contains multiple parameters
18 years ago
orbiter ec031eb993 first version of surftipps
18 years ago
theli 5afb0cbce8 *) setting default charset (for unkown documents) to iso-8859-1
18 years ago
theli 97d2a08ef1 *) restructuring needed to support parsing of documents using various charsets
18 years ago
theli fc594e8eda *) adding httpContentLengthInputStream.java class to allow reading of http response bodies
18 years ago
theli 2a06ce5538 *) next bugfix for UTF-8
18 years ago
theli bdc51591ae *) UTF-8 Bug solved (hopefully)
18 years ago
theli ef751b9d33 *) removing all string operations from the template engine
18 years ago
theli fded1f4a5d *) better handling of maximum file size limit in crawler
18 years ago
theli 63893003be *) Adding settings page for the crawler which allows to specify a file size limit and the timeout to use.
18 years ago
orbiter 9340dbb501 fixed all possible problems with nullpointer exception for LURLs
18 years ago
theli a5ed86105b *) bugfix for handling of ResourceInfo object in proxy
18 years ago
hydrox 59a5511dbb *) added missing static Strings as requested by theli
18 years ago
theli 6578564c9a *) Ignore more hop by hop http headers
18 years ago
theli dae763d8e3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2495 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli ffbf416e76 *) direct access to requestheader of htCache.Entry removed to make it more http independent
18 years ago
theli 3870d615e3 *) setting htCache.Entry fields to private
18 years ago
theli 393a7d10be *) setting htCache.Entry fields to private
18 years ago
theli 1c8300fcec *) Bugfix for name resolution in proxy mode
18 years ago
orbiter d78b824e85 fixed problem with default path after first start-up
19 years ago
orbiter 6ad471ef96 * applied many compiler warning recommendations
19 years ago
allo cf1186597b utf fix from theli
19 years ago
theli eee44be602 *) adding an interface for customized blacklist classes
19 years ago
theli d2e8e76218 *) now it's possible to configure the yacy blacklist separately for dht, search, proxy, crawler
19 years ago
allo a52f36787f better templatedebugging
19 years ago
allo 3480d36417 added some debug code
19 years ago
orbiter d468d665c9 some changes that may help to prevent deadlocks that cause an OutOfMemoryError
19 years ago
theli 6e676224d0 *) adding support for upnp
19 years ago
orbiter 97fa6788a1 added gettext support:
19 years ago
allo 67c486a023 some example Code, how supertemplates can be used.
19 years ago
allo 7b0e2521bb Support for a supertemplate, which can do all thing, a normal template can do.
19 years ago
allo 8795875800 dirlisting for all empty directories.
19 years ago
orbiter 3879a0ecd0 replaced java.net.URL usage by use of new class de.anomic.net.URL
19 years ago
theli b594ee9a5a *) Adding possibility to configure if the http proxy should send the
19 years ago
allo 6866bc2758 be quiet!
19 years ago
theli ed2cb040d1 *) Bugfix for http connection header validation
19 years ago
allo 0621106ef3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2214 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter 12af69dd86 cosmetics
19 years ago
allo 67a8c74be3 Fix for dynamic login with static password.
19 years ago
allo 6fe2fed87e cookieauth works with static Admin.
19 years ago
allo b23703f260 using cookieAuth.
19 years ago
allo 7f51a43cba disabled ipAuth for _p Pages (and broken Form-Login :-() for security reasons
19 years ago
allo bd22634c44 HTML-login, logout fixed.
19 years ago
hermens 3f1ebc097e Limit the size of the DNS cache to 5000 and the age of the entries to one day.
19 years ago
allo d7a3fdb18b no white pages, when clicking cancel on the password-dialog
19 years ago
rramthun 5625937d1c Language improvements
19 years ago
orbiter 26b6cddf51 synchronized the DNS cache, because the non-synchronized version resulted in deadlocks
19 years ago
orbiter 90d569d70f refactoring of index management:
19 years ago
theli b4ab183518 *) Bugfix for NullpointerException if the seeds IP could not be resolved
19 years ago
allo 9938c252dd better Errorhandling for proxyAccounts
19 years ago
orbiter 015d044c25 tried to fix some problems with latest changes to httpc
19 years ago
orbiter 55c5b41bd0 modified kelondroDyn to work better with new object caches
19 years ago
orbiter fd7c17e624 added virtual host support:
19 years ago
theli 727aac4768 *) Bugfix for Transparent-Proxy-Support <-> Port Forwarding problem
19 years ago
theli cd4aeffea2 *) Bugfix: httpdFileHandler.java did not handle filenames with encoded chars correctly
19 years ago