- new domains now get a chance to get crawled early
- less IO operations
- new balancing method
- better dump order at shutdown time
- bugfixes regarding not found url hashes (no more superfluous cache kill)
- domain access time is now shared over all balancer stacks
- viewing the stack does no more disturbish the balancing algorithm that much
- intelligent selection of best next domain using domain access times
- extra double-check (to double-check the double-check)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3384 6c8d7289-2bf4-0310-a012-ef5d649a1542
- added an assortment importer. the old database structures can
be imported with
java -classpath classes yacy -migrateassortments
- modified wordmigration. The indexes from WORDS are now imported
to the collection database. The call is
java -classpath classes yacy -migratewords
(as it was)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3044 6c8d7289-2bf4-0310-a012-ef5d649a1542
(no more NULL values are returned, instead, an IOException is thrown)
- removed ugly damagedURLS implementation from plasmaCrawlLURL.java
(this inserted a static value into the Object which is not really a good style)
- re-coded damagedURLS collection in yacy.java by catching an exception and evaluating the exception message
to do:
- the urldbcleanup feature must be re-tested
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1200 6c8d7289-2bf4-0310-a012-ef5d649a1542
Crawler StartURLs will now also added to the errorURL-DB if an error occures on this url
*) kelondroStack.java, plasmaSwitchboardQueue.java
Adding method which returns a list of all entries in the queue. This list is used by IndexCreate_p.java
instead of an iterator to display the indexing-list.
Advantages: avoid concurrent modifications of the list while displaying it.
Speedup because now we have to access only one sync function instead of multiple ones
(one for each entry)
*) IndexCreateIndexingQueue_p.java
Using new list() function of plasmaSwitchboardQueue
*) httpdFileHandler.java
If a servelet returns the special value "LOCATION" the httpFileHandler does a Redirection of
the Browser to the URL specified by the servelet. This can e.g. be used when a http get request is
used insead of a post request, but a refresh should not be allowed.
*) IndexCreateWWWLocalQueue_p.html
Now it's possible to delete single entries of the local crawler queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@626 6c8d7289-2bf4-0310-a012-ef5d649a1542
orbiter