You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter a31b9097a4 preparations for mass remote crawls: ... two main changes must be implemented to enable mass remote crawls: - shift control of robots.txt to crawl queue (away from stacker). This is necessary since remote crawls can contain unchecked urls. Each peer must check the robots to prevent that it is misused as crawl agent for unwanted file retrieval - implement new index files that control double-check of remotely crawled urls After removal of robots.txt checking from stacker threads, the multi-threading of this process is void. Multithreading has been removed. Also the thread pools for the crawl threads had been removed, since creation of these threads is not resource-consuming, for a detailed explanation see svn 4106 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4181 6c8d7289-2bf4-0310-a012-ef5d649a1542		17 years ago
..
crawlOrder.html	preparations for bulk remote crawls	20 years ago
crawlOrder.java	preparations for mass remote crawls:	17 years ago
crawlReceipt.html	initial load with yacy 0.36	20 years ago
crawlReceipt.java	preparations for mass remote crawls:	17 years ago
hello.html	fix for peer-ping in case that remote peer does not respond with valid values	17 years ago
hello.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
list.html	initial load with yacy 0.36	20 years ago
list.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
message.html	initial load with yacy 0.36	20 years ago
message.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
profile.html	initial load with yacy 0.36	20 years ago
profile.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
query.html	initial load with yacy 0.36	20 years ago
query.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
search.html	changes towards better join-search	18 years ago
search.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
transfer.html	added missing files for last commit	19 years ago
transfer.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
transferRWI.html	added a busy-time estimation at DHT/RWI-Receive	19 years ago
transferRWI.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
transferURL.html	initial load with yacy 0.36	20 years ago
transferURL.java	* Complete number localization and provide a more reasonable interface to serverObjects:	17 years ago
urls.java	preparations for mass remote crawls:	17 years ago
urls.xml	added new yacy protocol for mass url-pull for better remote crawling distribution	17 years ago