You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

luccioman aa9ddf3c23 Added control over Robots.txt active threads maximum number. ... When starting a crawl from a file containing thousands of links, configuration setting "crawler.MaxActiveThreads" is effective to prevent saturating the system with too many outgoing HTTP connections threads launched by the crawler. But robots.txt was not affected by this setting and was indefinitely increasing the number of concurrently loading threads until most ot the connections timed out. To improve performance control, added a pool of threads for Robots.txt, consistently used in its ensureExist() and massCrawlCheck() methods. The Robots.txt threads pool max size can now be configured in the /PerformanceQueus_p.html page, or with the new "robots.txt.MaxActiveThreads" setting, initialized with the same default value as the crawler.		8 years ago
..
data	Customized name for Threads still using the default "Thread-n" pattern.	8 years ago
retrieval	Customized name for Threads still using the default "Thread-n" pattern.	8 years ago
robots	Added control over Robots.txt active threads maximum number.	8 years ago
Balancer.java	removed warnings	10 years ago
CrawlStacker.java	use human readable date format in CrawlStacker error message	8 years ago
CrawlStarterFromSraper.java	Advanced Crawl from local file : better processing of large files.	8 years ago
CrawlSwitchboard.java	remove wrong log line in CrawlSwitchboard	9 years ago
FileCrawlStarterTask.java	Crawl from local file : faster task end when manually terminating crawl.	8 years ago
HarvestProcess.java	fix for wrong display of error urls in HostBrowser	12 years ago
HostBalancer.java	Customized name for Threads still using the default "Thread-n" pattern.	8 years ago
HostQueue.java	to prevent crawler to concurrently access and alter same crawl queue	9 years ago
IllegalCrawlProfileException.java	Crawl from local file : faster task end when manually terminating crawl.	8 years ago
LegacyBalancer.java	use supplied url port to get robots.txt in crawlers hostqueue	9 years ago
RecrawlBusyThread.java	init Recrawl job chunk size to max crawl loader during job start, to use some system preferences	9 years ago