You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

luccioman fcf6b16db4 Added new crawler attribute for finer control over Media Type detection ... New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244		6 years ago
..
data	Added new crawler attribute for finer control over Media Type detection	6 years ago
retrieval	Added new crawler attribute for finer control over Media Type detection	6 years ago
robots	…
Balancer.java	…
CrawlStacker.java	Added new crawler attribute for finer control over Media Type detection	6 years ago
CrawlStarterFromScraper.java	…
CrawlSwitchboard.java	Do not block whole server startup on persisted crawl profile load error	7 years ago
FileCrawlStarterTask.java	…
HarvestProcess.java	…
HostBalancer.java	Fixed NullPointerException case on malformed crawl queue folder name	7 years ago
HostQueue.java	Fixed crawl queue folder naming for IPv6 hosts on MS Windows filesystems	7 years ago
IllegalCrawlProfileException.java	…
LegacyBalancer.java	…
RecrawlBusyThread.java	…