You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

orbiter a31b9097a4 preparations for mass remote crawls: ... two main changes must be implemented to enable mass remote crawls: - shift control of robots.txt to crawl queue (away from stacker). This is necessary since remote crawls can contain unchecked urls. Each peer must check the robots to prevent that it is misused as crawl agent for unwanted file retrieval - implement new index files that control double-check of remotely crawled urls After removal of robots.txt checking from stacker threads, the multi-threading of this process is void. Multithreading has been removed. Also the thread pools for the crawl threads had been removed, since creation of these threads is not resource-consuming, for a detailed explanation see svn 4106 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4181 6c8d7289-2bf4-0310-a012-ef5d649a1542		18 years ago
..
data	preparations for mass remote crawls:	18 years ago
htmlFilter	- Changed yFormatter handling to be more flexible and produce more readable code for server pages. There are serverObject.putNum() methods to allow adding of number type values in a formatted form, and put() methods for number types that add them without formatting. This reduces the need to transform them into Strings in server pages and removes the HTML encoding step which is unecessary for numbers.	18 years ago
http	preparations for mass remote crawls:	18 years ago
icap	preparations for mass remote crawls:	18 years ago
index	distinct keepOrder strategy, more discrete implementation of enhancement introduced in SVN 4158	18 years ago
kelondro	fix for last commit	18 years ago
language	…
net	joined anomic.net.URL, plasmaURL and url hash computation:	18 years ago
plasma	preparations for mass remote crawls:	18 years ago
server	preparations for mass remote crawls:	18 years ago
tools	- Changed yFormatter handling to be more flexible and produce more readable code for server pages. There are serverObject.putNum() methods to allow adding of number type values in a formatted form, and put() methods for number types that add them without formatting. This reduces the need to transform them into Strings in server pages and removes the HTML encoding step which is unecessary for numbers.	18 years ago
urlRedirector	preparations for mass remote crawls:	18 years ago
xml	refactoring of search processes	18 years ago
yacy	preparations for mass remote crawls:	18 years ago
ymage	- Changed yFormatter handling to be more flexible and produce more readable code for server pages. There are serverObject.putNum() methods to allow adding of number type values in a formatted form, and put() methods for number types that add them without formatting. This reduces the need to transform them into Strings in server pages and removes the HTML encoding step which is unecessary for numbers.	18 years ago