removed Latency update after URL selection because that causes

a completely wrong behaviour when cache fresh cases appear. Makes
re-crawling MUCH faster!
pull/1/head
Michael Peter Christen 12 years ago
parent 4491072256
commit 864abcd33d

@ -78,8 +78,6 @@ public class Latency {
h = new Host(host, DEFAULT_AVERAGE, robotsCrawlDelay);
if (map.size() > 1000 || MemoryControl.shortStatus()) map.clear();
map.put(hosthash, h);
} else {
h.update();
}
}

Loading…
Cancel
Save