Commit Graph

16 Commits (2a06ce5538550878beeefec0442bf1e0fe7abab8)

Author SHA1 Message Date
theli a0ddf2ec11 *) AbstractCrawlWorker.java: delete already downloaded data on crawling error
18 years ago
theli fded1f4a5d *) better handling of maximum file size limit in crawler
18 years ago
theli 63893003be *) Adding settings page for the crawler which allows to specify a file size limit and the timeout to use.
18 years ago
theli b44514242a *) crawler/ftp/CrawlWorker.java: better errorhandling
18 years ago
theli 7d7f30139c *) crawler/ftp/CrawlWorker.java: delete old cache file
18 years ago
theli 043edfa4d8 *) ftp/ResourceInfo.java ResourceInfo object for ftp resources added
18 years ago
theli dae763d8e3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2495 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli 4825bfaaf3 *) Bugfix for PrintWriter Problem
18 years ago
theli 7930839594 *) URL.java: userinfo was not taken over when generating a new url from a base url and a rel. path
18 years ago
theli 393a7d10be *) setting htCache.Entry fields to private
18 years ago
theli ab5a9bee66 *) adding some copyright headers
18 years ago
theli fce9e7741b *) next step of restructuring for new crawlers
18 years ago
theli 4e2a950ac9 *) next step of restructuring for new crawlers
18 years ago
theli 09b106eb04 *) next step of restructuring for new crawlers
18 years ago
theli eb9b138986 *) next step of restructuring for new crawlers
18 years ago
theli 1395aae742 *) starting restructuring which is needed to add crawlers for additional protocols
18 years ago