You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1787 lines
92 KiB
1787 lines
92 KiB
// plasmaSwitchboard.java
|
|
// -----------------------
|
|
// part of YaCy
|
|
// (C) by Michael Peter Christen; mc@anomic.de
|
|
// first published on http://www.anomic.de
|
|
// Frankfurt, Germany, 2004, 2005
|
|
//
|
|
// $LastChangedDate$
|
|
// $LastChangedRevision$
|
|
// $LastChangedBy$
|
|
//
|
|
// This program is free software; you can redistribute it and/or modify
|
|
// it under the terms of the GNU General Public License as published by
|
|
// the Free Software Foundation; either version 2 of the License, or
|
|
// (at your option) any later version.
|
|
//
|
|
// This program is distributed in the hope that it will be useful,
|
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
// GNU General Public License for more details.
|
|
//
|
|
// You should have received a copy of the GNU General Public License
|
|
// along with this program; if not, write to the Free Software
|
|
// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
|
//
|
|
// Using this software in any meaning (reading, learning, copying, compiling,
|
|
// running) means that you agree that the Author(s) is (are) not responsible
|
|
// for cost, loss of data or any harm that may be caused directly or indirectly
|
|
// by usage of this softare or this documentation. The usage of this software
|
|
// is on your own risk. The installation and usage (starting/running) of this
|
|
// software may allow other people or application to access your computer and
|
|
// any attached devices and is highly dependent on the configuration of the
|
|
// software which must be done by the user of the software; the author(s) is
|
|
// (are) also not responsible for proper configuration and usage of the
|
|
// software, even if provoked by documentation provided together with
|
|
// the software.
|
|
//
|
|
// Any changes to this file according to the GPL as documented in the file
|
|
// gpl.txt aside this file in the shipment you received can be done to the
|
|
// lines that follows this copyright notice here, but changes must not be
|
|
// done inside the copyright notive above. A re-distribution must contain
|
|
// the intact and unchanged copyright notice.
|
|
// Contributions and changes to the program code must be marked as such.
|
|
|
|
/*
|
|
This class holds the run-time environment of the plasma
|
|
Search Engine. It's data forms a blackboard which can be used
|
|
to organize running jobs around the indexing algorithm.
|
|
The blackboard consist of the following entities:
|
|
- storage: one plasmaStore object with the url-based database
|
|
- configuration: initialized by properties once, then by external functions
|
|
- job queues: for parsing, condensing, indexing
|
|
- black/blue/whitelists: controls input and output to the index
|
|
|
|
this class is also the core of the http crawling.
|
|
There are some items that need to be respected when crawling the web:
|
|
1) respect robots.txt
|
|
2) do not access one domain too frequently, wait between accesses
|
|
3) remember crawled URL's and do not access again too early
|
|
4) priorization of specific links should be possible (hot-lists)
|
|
5) attributes for crawling (depth, filters, hot/black-lists, priority)
|
|
6) different crawling jobs with different attributes ('Orders') simultanoulsy
|
|
|
|
We implement some specific tasks and use different database to archieve these goals:
|
|
- a database 'crawlerDisallow.db' contains all url's that shall not be crawled
|
|
- a database 'crawlerDomain.db' holds all domains and access times, where we loaded the disallow tables
|
|
this table contains the following entities:
|
|
<flag: robotes exist/not exist, last access of robots.txt, last access of domain (for access scheduling)>
|
|
- four databases for scheduled access: crawlerScheduledHotText.db, crawlerScheduledColdText.db,
|
|
crawlerScheduledHotMedia.db and crawlerScheduledColdMedia.db
|
|
- two stacks for new URLS: newText.stack and newMedia.stack
|
|
- two databases for URL double-check: knownText.db and knownMedia.db
|
|
- one database with crawling orders: crawlerOrders.db
|
|
|
|
The Information flow of a single URL that is crawled is as follows:
|
|
- a html file is loaded from a specific URL within the module httpdProxyServlet as
|
|
a process of the proxy.
|
|
- the file is passed to httpdProxyCache. Here it's processing is delayed until the proxy is idle.
|
|
- The cache entry is passed on to the plasmaSwitchboard. There the URL is stored into plasmaLURL where
|
|
the URL is stored under a specific hash. The URL's from the content are stripped off, stored in plasmaLURL
|
|
with a 'wrong' date (the date of the URL's are not known at this time, only after fetching) and stacked with
|
|
plasmaCrawlerTextStack. The content is read and splitted into rated words in plasmaCondenser.
|
|
The splitted words are then integrated into the index with plasmaSearch.
|
|
- In plasmaSearch the words are indexed by reversing the relation between URL and words: one URL points
|
|
to many words, the words within the document at the URL. After reversing, one word points
|
|
to many URL's, all the URL's where the word occurrs. One single word->URL-hash relation is stored in
|
|
plasmaIndexEntry. A set of plasmaIndexEntries is a reverse word index.
|
|
This reverse word index is stored temporarly in plasmaIndexCache.
|
|
- In plasmaIndexCache the single plasmaIndexEntry'ies are collected and stored into a plasmaIndex - entry
|
|
These plasmaIndex - Objects are the true reverse words indexes.
|
|
- in plasmaIndex the plasmaIndexEntry - objects are stored in a kelondroTree; an indexed file in the file system.
|
|
|
|
The information flow of a search request is as follows:
|
|
- in httpdFileServlet the user enters a search query, which is passed to plasmaSwitchboard
|
|
- in plasmaSwitchboard, the query is passed to plasmaSearch.
|
|
- in plasmaSearch, the plasmaSearch.result object is generated by simultanous enumeration of
|
|
URL hases in the reverse word indexes plasmaIndex
|
|
- (future: the plasmaSearch.result - object is used to identify more key words for a new search)
|
|
|
|
|
|
|
|
*/
|
|
|
|
package de.anomic.plasma;
|
|
|
|
import java.io.ByteArrayInputStream;
|
|
import java.io.File;
|
|
import java.io.IOException;
|
|
import java.net.URL;
|
|
import java.text.SimpleDateFormat;
|
|
import java.util.ArrayList;
|
|
import java.util.Date;
|
|
import java.util.HashMap;
|
|
import java.util.Iterator;
|
|
import java.util.Map;
|
|
import java.util.Set;
|
|
import java.util.TreeSet;
|
|
import java.util.logging.Level;
|
|
import de.anomic.data.messageBoard;
|
|
import de.anomic.data.wikiBoard;
|
|
import de.anomic.data.userDB;
|
|
import de.anomic.htmlFilter.htmlFilterContentScraper;
|
|
import de.anomic.http.httpHeader;
|
|
import de.anomic.http.httpRemoteProxyConfig;
|
|
import de.anomic.http.httpc;
|
|
import de.anomic.kelondro.kelondroException;
|
|
import de.anomic.kelondro.kelondroMSetTools;
|
|
import de.anomic.kelondro.kelondroTables;
|
|
import de.anomic.server.serverAbstractSwitch;
|
|
import de.anomic.server.serverCodings;
|
|
import de.anomic.server.serverCore;
|
|
import de.anomic.server.serverInstantThread;
|
|
import de.anomic.server.serverObjects;
|
|
import de.anomic.server.serverSemaphore;
|
|
import de.anomic.server.serverSwitch;
|
|
import de.anomic.server.logging.serverLog;
|
|
import de.anomic.tools.bitfield;
|
|
import de.anomic.tools.crypt;
|
|
import de.anomic.yacy.yacyClient;
|
|
import de.anomic.yacy.yacyCore;
|
|
import de.anomic.yacy.yacySeed;
|
|
import de.anomic.yacy.yacyNewsPool;
|
|
|
|
public final class plasmaSwitchboard extends serverAbstractSwitch implements serverSwitch {
|
|
|
|
// load slots
|
|
public static int crawlSlots = 10;
|
|
public static int indexingSlots = 100;
|
|
public static int stackCrawlSlots = 10000;
|
|
|
|
// couloured list management
|
|
public static TreeSet blueList = null;
|
|
public static TreeSet stopwords = null;
|
|
public static plasmaURLPattern urlBlacklist;
|
|
|
|
// storage management
|
|
public File htCachePath;
|
|
private File plasmaPath;
|
|
public File listsPath;
|
|
public plasmaURLPool urlPool;
|
|
public plasmaWordIndex wordIndex;
|
|
public plasmaHTCache cacheManager;
|
|
public plasmaSnippetCache snippetCache;
|
|
public plasmaCrawlLoader cacheLoader;
|
|
public plasmaSwitchboardQueue sbQueue;
|
|
//public plasmaStackCrawlThread sbStackCrawlThread;
|
|
public plasmaCrawlStacker sbStackCrawlThread;
|
|
public messageBoard messageDB;
|
|
public wikiBoard wikiDB;
|
|
public static plasmaCrawlRobotsTxt robots;
|
|
public plasmaCrawlProfile profiles;
|
|
public plasmaCrawlProfile.entry defaultProxyProfile;
|
|
public plasmaCrawlProfile.entry defaultRemoteProfile;
|
|
public plasmaWordIndexDistribution indexDistribution;
|
|
public HashMap outgoingCookies, incomingCookies;
|
|
public kelondroTables facilityDB;
|
|
public plasmaParser parser;
|
|
public plasmaWordIndexClassicCacheMigration classicCache;
|
|
public long proxyLastAccess;
|
|
public yacyCore yc;
|
|
public HashMap indexingTasksInProcess;
|
|
public userDB userDB;
|
|
|
|
/*
|
|
* Remote Proxy configuration
|
|
*/
|
|
// public boolean remoteProxyUse;
|
|
// public boolean remoteProxyUse4Yacy;
|
|
// public String remoteProxyHost;
|
|
// public int remoteProxyPort;
|
|
// public String remoteProxyNoProxy = "";
|
|
// public String[] remoteProxyNoProxyPatterns = null;
|
|
public httpRemoteProxyConfig remoteProxyConfig = null;
|
|
|
|
|
|
/*
|
|
* Some constants
|
|
*/
|
|
private static final String STR_PROXYPROFILE = "defaultProxyProfile";
|
|
private static final String STR_REMOTEPROFILE = "defaultRemoteProfile";
|
|
private static final String STR_REMOTECRAWLTRIGGER = "REMOTECRAWLTRIGGER: REMOTE CRAWL TO PEER ";
|
|
|
|
private serverSemaphore shutdownSync = new serverSemaphore(0);
|
|
private boolean terminate = false;
|
|
|
|
private Object crawlingPausedSync = new Object();
|
|
private boolean crawlingIsPaused = false;
|
|
private static plasmaSwitchboard sb;
|
|
|
|
public plasmaSwitchboard(String rootPath, String initPath, String configPath) throws IOException {
|
|
super(rootPath, initPath, configPath);
|
|
|
|
// set loglevel and log
|
|
setLog(new serverLog("PLASMA"));
|
|
|
|
// load values from configs
|
|
this.plasmaPath = new File(rootPath, getConfig("dbPath", "PLASMADB"));
|
|
this.log.logConfig("Plasma DB Path: " + this.plasmaPath.toString());
|
|
this.listsPath = new File(rootPath, getConfig("listsPath", "LISTS"));
|
|
this.log.logConfig("Lists Path: " + this.listsPath.toString());
|
|
|
|
/* ============================================================================
|
|
* Remote Proxy configuration
|
|
* ============================================================================ */
|
|
this.remoteProxyConfig = httpRemoteProxyConfig.init(this);
|
|
this.log.logConfig("Remote proxy configuration:\n" + this.remoteProxyConfig.toString());
|
|
|
|
// // reading the proxy host name
|
|
// this.remoteProxyHost = getConfig("remoteProxyHost", "");
|
|
//
|
|
// // reading the proxy host port
|
|
// try {
|
|
// this.remoteProxyPort = Integer.parseInt(getConfig("remoteProxyPort", "3128"));
|
|
// } catch (NumberFormatException e) {
|
|
// this.remoteProxyPort = 3128;
|
|
// }
|
|
//
|
|
// // determining if remote proxy should be used for yacy -> yacy communication
|
|
// this.remoteProxyUse4Yacy = getConfig("remoteProxyUse4Yacy", "true").equalsIgnoreCase("true");
|
|
//
|
|
// // determining addresses for which the remote proxy should not be used
|
|
// this.remoteProxyNoProxy = getConfig("remoteProxyNoProxy","");
|
|
// this.remoteProxyNoProxyPatterns = this.remoteProxyNoProxy.split(",");
|
|
//
|
|
// // determining if remote Proxy should be used
|
|
// if (getConfig("remoteProxyUse", "false").equalsIgnoreCase("true")) {
|
|
// this.remoteProxyUse = true;
|
|
// this.log.logConfig("Using remote proxy:" +
|
|
// "\n\tHost: " + this.remoteProxyHost +
|
|
// "\n\tPort: " + this.remoteProxyPort +
|
|
// "\n\tUseProxy4Yacy: " + Boolean.toString(this.remoteProxyUse4Yacy)
|
|
// );
|
|
// } else {
|
|
// this.remoteProxyUse = false;
|
|
// this.remoteProxyHost = null;
|
|
// this.remoteProxyPort = 0;
|
|
// }
|
|
this.proxyLastAccess = System.currentTimeMillis() - 60000;
|
|
|
|
// configuring list path
|
|
if (!(listsPath.exists())) listsPath.mkdirs();
|
|
|
|
// load coloured lists
|
|
if (blueList == null) {
|
|
// read only once upon first instantiation of this class
|
|
String f = getConfig("plasmaBlueList", null);
|
|
File plasmaBlueListFile = new File(f);
|
|
if (f != null) blueList = kelondroMSetTools.loadList(plasmaBlueListFile); else blueList= new TreeSet();
|
|
this.log.logConfig("loaded blue-list from file " + plasmaBlueListFile.getName() + ", " +
|
|
blueList.size() + " entries, " +
|
|
ppRamString(plasmaBlueListFile.length()/1024));
|
|
}
|
|
|
|
// load the black-list / inspired by [AS]
|
|
File ulrBlackListFile = new File(getRootPath(), getConfig("listsPath", "DATA/LISTS"));
|
|
urlBlacklist = new plasmaURLPattern(ulrBlackListFile);
|
|
String f = getConfig("proxyBlackListsActive", null);
|
|
if (f != null) {
|
|
urlBlacklist.loadLists("black", f, "/");
|
|
this.log.logConfig("loaded black-list from file " + ulrBlackListFile.getName() + ", " +
|
|
urlBlacklist.size() + " entries, " +
|
|
ppRamString(ulrBlackListFile.length()/1024));
|
|
}
|
|
|
|
// load stopwords
|
|
if (stopwords == null) {
|
|
File stopwordsFile = new File(rootPath, "yacy.stopwords");
|
|
stopwords = kelondroMSetTools.loadList(stopwordsFile);
|
|
this.log.logConfig("loaded stopwords from file " + stopwordsFile.getName() + ", " +
|
|
stopwords.size() + " entries, " +
|
|
ppRamString(stopwordsFile.length()/1024));
|
|
}
|
|
|
|
// read memory amount
|
|
int ramLURL = (int) getConfigLong("ramCacheLURL", 1024) / 1024;
|
|
int ramNURL = (int) getConfigLong("ramCacheNURL", 1024) / 1024;
|
|
int ramEURL = (int) getConfigLong("ramCacheEURL", 1024) / 1024;
|
|
int ramRWI = (int) getConfigLong("ramCacheRWI", 1024) / 1024;
|
|
int ramHTTP = (int) getConfigLong("ramCacheHTTP", 1024) / 1024;
|
|
int ramMessage = (int) getConfigLong("ramCacheMessage", 1024) / 1024;
|
|
int ramWiki = (int) getConfigLong("ramCacheWiki", 1024) / 1024;
|
|
int ramRobots = (int) getConfigLong("ramCacheRobots",1024) / 1024;
|
|
int ramProfiles= (int) getConfigLong("ramCacheProfiles",1024) / 1024;
|
|
int ramPreNURL = (int) getConfigLong("ramCachePreNURL", 1024) / 1024;
|
|
this.log.logConfig("LURL Cache memory = " + ppRamString(ramLURL));
|
|
this.log.logConfig("NURL Cache memory = " + ppRamString(ramNURL));
|
|
this.log.logConfig("EURL Cache memory = " + ppRamString(ramEURL));
|
|
this.log.logConfig("RWI Cache memory = " + ppRamString(ramRWI));
|
|
this.log.logConfig("HTTP Cache memory = " + ppRamString(ramHTTP));
|
|
this.log.logConfig("Message Cache memory = " + ppRamString(ramMessage));
|
|
this.log.logConfig("Wiki Cache memory = " + ppRamString(ramWiki));
|
|
this.log.logConfig("Robots Cache memory = " + ppRamString(ramRobots));
|
|
this.log.logConfig("Profiles Cache memory = " + ppRamString(ramProfiles));
|
|
this.log.logConfig("PreNURL Cache memory = " + ppRamString(ramPreNURL));
|
|
|
|
// make crawl profiles database and default profiles
|
|
this.log.logConfig("Initializing Crawl Profiles");
|
|
File profilesFile = new File(this.plasmaPath, "crawlProfiles0.db");
|
|
this.profiles = new plasmaCrawlProfile(profilesFile, ramProfiles);
|
|
initProfiles();
|
|
log.logConfig("Loaded profiles from file " + profilesFile.getName() +
|
|
", " + this.profiles.size() + " entries" +
|
|
", " + ppRamString(profilesFile.length()/1024));
|
|
|
|
// loading the robots.txt db
|
|
this.log.logConfig("Initializing robots.txt DB");
|
|
File robotsDBFile = new File(this.plasmaPath, "crawlRobotsTxt.db");
|
|
this.robots = new plasmaCrawlRobotsTxt(robotsDBFile, ramRobots);
|
|
this.log.logConfig("Loaded robots.txt DB from file " + robotsDBFile.getName() +
|
|
", " + this.robots.size() + " entries" +
|
|
", " + ppRamString(robotsDBFile.length()/1024));
|
|
|
|
// start indexing management
|
|
log.logConfig("Starting Indexing Management");
|
|
urlPool = new plasmaURLPool(plasmaPath, ramLURL, ramNURL, ramEURL);
|
|
|
|
wordIndex = new plasmaWordIndex(plasmaPath, ramRWI, log);
|
|
int wordCacheMaxLow = (int) getConfigLong("wordCacheMaxLow", 8000);
|
|
int wordCacheMaxHigh = (int) getConfigLong("wordCacheMaxHigh", 10000);
|
|
wordIndex.setMaxWords(wordCacheMaxLow, wordCacheMaxHigh);
|
|
|
|
// start a cache manager
|
|
log.logConfig("Starting HT Cache Manager");
|
|
|
|
// create the cache directory
|
|
String cache = getConfig("proxyCache", "DATA/HTCACHE");
|
|
cache = cache.replace('\\', '/');
|
|
if (cache.endsWith("/")) { cache = cache.substring(0, cache.length() - 1); }
|
|
if (new File(cache).isAbsolute()) {
|
|
htCachePath = new File(cache); // don't use rootPath
|
|
} else {
|
|
htCachePath = new File(rootPath, cache);
|
|
}
|
|
this.log.logInfo("HTCACHE Path = " + htCachePath.getAbsolutePath());
|
|
long maxCacheSize = 1024 * 1024 * Long.parseLong(getConfig("proxyCacheSize", "2")); // this is megabyte
|
|
this.cacheManager = new plasmaHTCache(htCachePath, maxCacheSize, ramHTTP);
|
|
|
|
// make parser
|
|
log.logConfig("Starting Parser");
|
|
this.parser = new plasmaParser();
|
|
|
|
// initialize switchboard queue
|
|
sbQueue = new plasmaSwitchboardQueue(this.cacheManager, urlPool.loadedURL, new File(plasmaPath, "switchboardQueue1.stack"), 10, profiles);
|
|
indexingTasksInProcess = new HashMap();
|
|
|
|
// going through the sbQueue Entries and registering all content files as in use
|
|
int count = 0;
|
|
ArrayList sbQueueEntries = this.sbQueue.list();
|
|
for (int i=0; i<sbQueueEntries.size(); i++) {
|
|
plasmaSwitchboardQueue.Entry entry = (plasmaSwitchboardQueue.Entry) sbQueueEntries.get(i);
|
|
if ((entry != null)&&(entry.url() != null)&&(entry.cacheFile().exists())) {
|
|
plasmaHTCache.filesInUse.add(entry.cacheFile());
|
|
count++;
|
|
}
|
|
}
|
|
log.logConfig(count + " files in htcache reported to the cachemanager as in use.");
|
|
|
|
// define an extension-blacklist
|
|
log.logConfig("Parser: Initializing Extension Mappings for Media/Parser");
|
|
plasmaParser.initMediaExt(plasmaParser.extString2extList(getConfig("mediaExt","")));
|
|
plasmaParser.initSupportedRealtimeFileExt(plasmaParser.extString2extList(getConfig("parseableExt","")));
|
|
|
|
// define a realtime parsable mimetype list
|
|
log.logConfig("Parser: Initializing Mime Types");
|
|
plasmaParser.initRealtimeParsableMimeTypes(getConfig("parseableRealtimeMimeTypes","application/xhtml+xml,text/html,text/plain"));
|
|
plasmaParser.initParseableMimeTypes(getConfig("parseableMimeTypes",null));
|
|
|
|
// start a loader
|
|
log.logConfig("Starting Crawl Loader");
|
|
int remoteport;
|
|
try { remoteport = Integer.parseInt(getConfig("remoteProxyPort","3128")); }
|
|
catch (NumberFormatException e) { remoteport = 3128; }
|
|
|
|
crawlSlots = Integer.parseInt(getConfig("crawler.MaxActiveThreads", "10"));
|
|
this.crawlingIsPaused = Boolean.valueOf(getConfig("crawler.isPaused", "false")).booleanValue();
|
|
plasmaCrawlLoader.switchboard = this;
|
|
this.cacheLoader = new plasmaCrawlLoader(
|
|
this.cacheManager,
|
|
this.log);
|
|
|
|
// starting message board
|
|
this.log.logConfig("Starting Message Board");
|
|
File messageDbFile = new File(getRootPath(), "DATA/SETTINGS/message.db");
|
|
this.messageDB = new messageBoard(messageDbFile, ramMessage);
|
|
this.log.logConfig("Loaded Message Board DB from file " + messageDbFile.getName() +
|
|
", " + this.messageDB.size() + " entries" +
|
|
", " + ppRamString(messageDbFile.length()/1024));
|
|
|
|
// starting wiki
|
|
this.log.logConfig("Starting Wiki Board");
|
|
File wikiDbFile = new File(getRootPath(), "DATA/SETTINGS/wiki.db");
|
|
this.wikiDB = new wikiBoard(wikiDbFile,
|
|
new File(getRootPath(), "DATA/SETTINGS/wiki-bkp.db"), ramWiki);
|
|
this.log.logConfig("Loaded Wiki Board DB from file " + wikiDbFile.getName() +
|
|
", " + this.wikiDB.size() + " entries" +
|
|
", " + ppRamString(wikiDbFile.length()/1024));
|
|
|
|
// Init User DB
|
|
this.log.logConfig("Loading User DB");
|
|
File userDbFile = new File(getRootPath(), "DATA/SETTINGS/user.db");
|
|
this.userDB = new userDB(userDbFile, 512);
|
|
this.log.logConfig("Loaded User DB from file " + userDbFile.getName() +
|
|
", " + this.userDB.size() + " entries" +
|
|
", " + ppRamString(userDbFile.length()/1024));
|
|
|
|
// init cookie-Monitor
|
|
this.log.logConfig("Starting Cookie Monitor");
|
|
this.outgoingCookies = new HashMap();
|
|
this.incomingCookies = new HashMap();
|
|
|
|
// clean up profiles
|
|
this.log.logConfig("Cleaning Profiles");
|
|
cleanProfiles();
|
|
|
|
// init facility DB
|
|
/*
|
|
log.logSystem("Starting Facility Database");
|
|
File facilityDBpath = new File(getRootPath(), "DATA/SETTINGS/");
|
|
facilityDB = new kelondroTables(facilityDBpath);
|
|
facilityDB.declareMaps("backlinks", 250, 500, new String[] {"date"}, null);
|
|
log.logSystem("..opened backlinks");
|
|
facilityDB.declareMaps("zeitgeist", 40, 500);
|
|
log.logSystem("..opened zeitgeist");
|
|
facilityDB.declareTree("statistik", new int[]{11, 8, 8, 8, 8, 8, 8}, 0x400);
|
|
log.logSystem("..opened statistik");
|
|
facilityDB.update("statistik", (new serverDate()).toShortString(false).substring(0, 11), new long[]{1,2,3,4,5,6});
|
|
long[] testresult = facilityDB.selectLong("statistik", "yyyyMMddHHm");
|
|
testresult = facilityDB.selectLong("statistik", (new serverDate()).toShortString(false).substring(0, 11));
|
|
*/
|
|
|
|
// initializing yacyDebugMode
|
|
httpc.yacyDebugMode = getConfig("yacyDebugMode", "false").equals("true");
|
|
|
|
// generate snippets cache
|
|
log.logConfig("Initializing Snippet Cache");
|
|
snippetCache = new plasmaSnippetCache(this,cacheManager, parser,log);
|
|
|
|
// start yacy core
|
|
log.logConfig("Starting YaCy Protocol Core");
|
|
//try{Thread.currentThread().sleep(5000);} catch (InterruptedException e) {} // for profiler
|
|
this.yc = new yacyCore(this);
|
|
//log.logSystem("Started YaCy Protocol Core");
|
|
System.gc(); try{Thread.currentThread().sleep(5000);} catch (InterruptedException e) {} // for profiler
|
|
serverInstantThread.oneTimeJob(yc, "loadSeeds", yc.log, 3000);
|
|
|
|
// initializing the stackCrawlThread
|
|
this.sbStackCrawlThread = new plasmaCrawlStacker(this,this.plasmaPath,ramPreNURL);
|
|
//this.sbStackCrawlThread = new plasmaStackCrawlThread(this,this.plasmaPath,ramPreNURL);
|
|
//this.sbStackCrawlThread.start();
|
|
|
|
// deploy threads
|
|
log.logConfig("Starting Threads");
|
|
// System.gc(); // help for profiler
|
|
int indexing_cluster = Integer.parseInt(getConfig("80_indexing_cluster", "1"));
|
|
if (indexing_cluster < 1) indexing_cluster = 1;
|
|
deployThread("90_cleanup", "Cleanup", "simple cleaning process for monitoring information", null,
|
|
new serverInstantThread(this, "cleanupJob", "cleanupJobSize"), 10000); // all 5 Minutes
|
|
deployThread("82_crawlstack", "Crawl URL Stacker", "process that checks url for double-occurrences and for allowance/disallowance by robots.txt", null,
|
|
new serverInstantThread(sbStackCrawlThread, "job", "size"), 8000);
|
|
serverInstantThread indexingThread = null;
|
|
deployThread("80_indexing", "Parsing/Indexing", "thread that performes document parsing and indexing", "/IndexCreateIndexingQueue_p.html",
|
|
indexingThread = new serverInstantThread(this, "deQueue", "queueSize"), 10000);
|
|
for (int i = 1; i < indexing_cluster; i++) {
|
|
setConfig((i + 80) + "_indexing_idlesleep", getConfig("80_indexing_idlesleep", ""));
|
|
setConfig((i + 80) + "_indexing_busysleep", getConfig("80_indexing_busysleep", ""));
|
|
deployThread((i + 80) + "_indexing", "Parsing/Indexing (cluster job)", "thread that performes document parsing and indexing", null,
|
|
new serverInstantThread(this, "deQueue", "queueSize"), 10000 + (i * 1000),
|
|
Long.parseLong(getConfig("80_indexing_idlesleep" , "5000")),
|
|
Long.parseLong(getConfig("80_indexing_busysleep" , "0")),
|
|
Long.parseLong(getConfig("80_indexing_memprereq" , "1000000")));
|
|
}
|
|
deployThread("70_cachemanager", "Proxy Cache Enqueue", "job takes new proxy files from RAM stack, stores them, and hands over to the Indexing Stack", null,
|
|
new serverInstantThread(this, "htEntryStoreJob", "htEntrySize"), 10000);
|
|
deployThread("62_remotetriggeredcrawl", "Remote Crawl Job", "thread that performes a single crawl/indexing step triggered by a remote peer", null,
|
|
new serverInstantThread(this, "remoteTriggeredCrawlJob", "remoteTriggeredCrawlJobSize"), 30000);
|
|
deployThread("61_globalcrawltrigger", "Global Crawl Trigger", "thread that triggeres remote peers for crawling", "/IndexCreateWWWGlobalQueue_p.html",
|
|
new serverInstantThread(this, "limitCrawlTriggerJob", "limitCrawlTriggerJobSize"), 30000); // error here?
|
|
deployThread("50_localcrawl", "Local Crawl", "thread that performes a single crawl step from the local crawl queue", "/IndexCreateWWWLocalQueue_p.html",
|
|
new serverInstantThread(this, "coreCrawlJob", "coreCrawlJobSize"), 10000);
|
|
deployThread("40_peerseedcycle", "Seed-List Upload", "task that a principal peer performes to generate and upload a seed-list to a ftp account", null,
|
|
new serverInstantThread(yc, "publishSeedList", null), 180000);
|
|
serverInstantThread peerPing = null;
|
|
deployThread("30_peerping", "YaCy Core", "this is the p2p-control and peer-ping task", null,
|
|
peerPing = new serverInstantThread(yc, "peerPing", null), 2000);
|
|
peerPing.setSyncObject(new Object());
|
|
|
|
this.indexDistribution = new plasmaWordIndexDistribution(
|
|
this.urlPool,
|
|
this.wordIndex,
|
|
this.log,
|
|
getConfig("allowDistributeIndex", "false").equalsIgnoreCase("true"),
|
|
getConfig("allowDistributeIndexWhileCrawling","false").equalsIgnoreCase("true"),
|
|
getConfig("indexDistribution.gzipBody","false").equalsIgnoreCase("true"),
|
|
(int)getConfigLong("indexDistribution.timeout",60000),
|
|
(int)getConfigLong("indexDistribution.maxOpenFiles",800)
|
|
);
|
|
indexDistribution.setCounts(150, 1, 3, 10000);
|
|
deployThread("20_dhtdistribution", "DHT Distribution", "selection, transfer and deletion of index entries that are not searched on your peer, but on others", null,
|
|
new serverInstantThread(indexDistribution, "job", null), 600000);
|
|
|
|
// init migratiion from 0.37 -> 0.38
|
|
classicCache = new plasmaWordIndexClassicCacheMigration(plasmaPath, wordIndex);
|
|
|
|
if (classicCache.size() > 0) {
|
|
setConfig("99_indexcachemigration_idlesleep" , 10000);
|
|
setConfig("99_indexcachemigration_busysleep" , 40);
|
|
deployThread("99_indexcachemigration", "index cache migration", "migration of index cache data structures 0.37 -> 0.38", null,
|
|
new serverInstantThread(classicCache, "oneStepMigration", "size"), 30000);
|
|
}
|
|
|
|
// test routine for snippet fetch
|
|
//Set query = new HashSet();
|
|
//query.add(plasmaWordIndexEntry.word2hash("Weitergabe"));
|
|
//query.add(plasmaWordIndexEntry.word2hash("Zahl"));
|
|
//plasmaSnippetCache.result scr = snippetCache.retrieve(new URL("http://www.heise.de/mobil/newsticker/meldung/mail/54980"), query, true);
|
|
//plasmaSnippetCache.result scr = snippetCache.retrieve(new URL("http://www.heise.de/security/news/foren/go.shtml?read=1&msg_id=7301419&forum_id=72721"), query, true);
|
|
//plasmaSnippetCache.result scr = snippetCache.retrieve(new URL("http://www.heise.de/kiosk/archiv/ct/2003/4/20"), query, true, 260);
|
|
|
|
sb=this;
|
|
log.logConfig("Finished Switchboard Initialization");
|
|
}
|
|
|
|
public static plasmaSwitchboard getSwitchboard(){
|
|
return sb;
|
|
}
|
|
|
|
/**
|
|
* This method changes the HTCache size.<br>
|
|
* @param new cache size in mb
|
|
*/
|
|
public final void setCacheSize(long newCacheSize) {
|
|
this.cacheManager.setCacheSize(1048576 * newCacheSize);
|
|
}
|
|
|
|
public boolean onlineCaution() {
|
|
try {
|
|
return System.currentTimeMillis() - proxyLastAccess < Integer.parseInt(getConfig("onlineCautionDelay", "30000"));
|
|
} catch (NumberFormatException e) {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
private static String ppRamString(long bytes) {
|
|
if (bytes < 1024) return bytes + " KByte";
|
|
bytes = bytes / 1024;
|
|
if (bytes < 1024) return bytes + " MByte";
|
|
bytes = bytes / 1024;
|
|
if (bytes < 1024) return bytes + " GByte";
|
|
return (bytes / 1024) + "TByte";
|
|
}
|
|
|
|
private void initProfiles() throws IOException {
|
|
if ((profiles.size() == 0) ||
|
|
(getConfig(STR_PROXYPROFILE, "").length() == 0) ||
|
|
(profiles.getEntry(getConfig(STR_PROXYPROFILE, "")) == null)) {
|
|
// generate new default entry for proxy crawling
|
|
defaultProxyProfile = profiles.newEntry("proxy", "", ".*", ".*", Integer.parseInt(getConfig("proxyPrefetchDepth", "0")), Integer.parseInt(getConfig("proxyPrefetchDepth", "0")), false, true, true, true, false, true, true, true);
|
|
setConfig(STR_PROXYPROFILE, defaultProxyProfile.handle());
|
|
} else {
|
|
defaultProxyProfile = profiles.getEntry(getConfig(STR_PROXYPROFILE, ""));
|
|
}
|
|
if ((profiles.size() == 1) ||
|
|
(getConfig(STR_REMOTEPROFILE, "").length() == 0) ||
|
|
(profiles.getEntry(getConfig(STR_REMOTEPROFILE, "")) == null)) {
|
|
// generate new default entry for remote crawling
|
|
defaultRemoteProfile = profiles.newEntry("remote", "", ".*", ".*", 0, 0, true, false, true, true, false, true, true, false);
|
|
setConfig(STR_REMOTEPROFILE, defaultRemoteProfile.handle());
|
|
} else {
|
|
defaultRemoteProfile = profiles.getEntry(getConfig(STR_REMOTEPROFILE, ""));
|
|
}
|
|
}
|
|
private void resetProfiles() {
|
|
final File pdb = new File(plasmaPath, "crawlProfiles0.db");
|
|
if (pdb.exists()) pdb.delete();
|
|
try {
|
|
int ramProfiles = (int) getConfigLong("ramCacheProfiles",1024) / 1024;
|
|
profiles = new plasmaCrawlProfile(pdb, ramProfiles);
|
|
initProfiles();
|
|
} catch (IOException e) {}
|
|
}
|
|
public boolean cleanProfiles() {
|
|
if ((sbQueue.size() > 0) || (cacheLoader.size() > 0) || (urlPool.noticeURL.stackSize() > 0)) return false;
|
|
final Iterator iter = profiles.profiles(true);
|
|
plasmaCrawlProfile.entry entry;
|
|
boolean hasDoneSomething = false;
|
|
try {
|
|
while (iter.hasNext()) {
|
|
entry = (plasmaCrawlProfile.entry) iter.next();
|
|
if (!((entry.name().equals("proxy")) || (entry.name().equals("remote")))) {
|
|
iter.remove();
|
|
hasDoneSomething = true;
|
|
}
|
|
}
|
|
} catch (kelondroException e) {
|
|
resetProfiles();
|
|
hasDoneSomething = true;
|
|
}
|
|
return hasDoneSomething;
|
|
}
|
|
|
|
public plasmaHTCache getCacheManager() {
|
|
return cacheManager;
|
|
}
|
|
|
|
synchronized public void htEntryStoreEnqueued(plasmaHTCache.Entry entry) throws IOException {
|
|
if (cacheManager.full())
|
|
htEntryStoreProcess(entry);
|
|
else
|
|
cacheManager.push(entry);
|
|
}
|
|
|
|
synchronized public boolean htEntryStoreProcess(plasmaHTCache.Entry entry) throws IOException {
|
|
|
|
if (entry == null) return false;
|
|
|
|
// store response header
|
|
if (entry.responseHeader != null) {
|
|
this.cacheManager.storeHeader(entry.nomalizedURLHash, entry.responseHeader);
|
|
this.log.logInfo("WROTE HEADER for " + entry.cacheFile);
|
|
}
|
|
|
|
/*
|
|
* Evaluating request header:
|
|
* With the X-YACY-Index-Control header set to "no-index" a client could disallow
|
|
* yacy to index the response returned as answer to a request
|
|
*/
|
|
boolean doIndexing = true;
|
|
if (entry.requestHeader != null) {
|
|
if (
|
|
(entry.requestHeader.containsKey(httpHeader.X_YACY_INDEX_CONTROL)) &&
|
|
(((String) entry.requestHeader.get(httpHeader.X_YACY_INDEX_CONTROL)).toUpperCase().equals("NO-INDEX"))
|
|
) {
|
|
doIndexing = false;
|
|
}
|
|
}
|
|
|
|
// work off unwritten files
|
|
if (entry.cacheArray == null) {
|
|
this.log.logInfo("EXISTING FILE (" + entry.cacheFile.length() + " bytes) for " + entry.cacheFile);
|
|
} else {
|
|
String error = entry.shallStoreCacheForProxy();
|
|
if (error == null) {
|
|
this.cacheManager.writeFile(entry.url, entry.cacheArray);
|
|
this.log.logInfo("WROTE FILE (" + entry.cacheArray.length + " bytes) for " + entry.cacheFile);
|
|
} else {
|
|
this.log.logInfo("WRITE OF FILE " + entry.cacheFile + " FORBIDDEN: " + error);
|
|
}
|
|
}
|
|
|
|
if ((doIndexing) && plasmaParser.supportedContent(entry.url,entry.responseHeader.mime())){
|
|
|
|
// registering the cachefile as in use
|
|
if (entry.cacheFile.exists()) {
|
|
plasmaHTCache.filesInUse.add(entry.cacheFile);
|
|
}
|
|
|
|
// enqueue for further crawling
|
|
enQueue(this.sbQueue.newEntry(entry.url, plasmaURL.urlHash(entry.referrerURL()),
|
|
entry.requestHeader.ifModifiedSince(), entry.requestHeader.containsKey(httpHeader.COOKIE),
|
|
entry.initiator(), entry.depth, entry.profile.handle(),
|
|
entry.name()
|
|
));
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
public boolean htEntryStoreJob() {
|
|
if (cacheManager.empty()) return false;
|
|
try {
|
|
return htEntryStoreProcess(cacheManager.pop());
|
|
} catch (IOException e) {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
public int htEntrySize() {
|
|
return cacheManager.size();
|
|
}
|
|
|
|
public void close() {
|
|
log.logConfig("SWITCHBOARD SHUTDOWN STEP 1: sending termination signal to managed threads:");
|
|
terminateAllThreads(true);
|
|
log.logConfig("SWITCHBOARD SHUTDOWN STEP 2: sending termination signal to threaded indexing (stand by..)");
|
|
int waitingBoundSeconds = Integer.parseInt(getConfig("maxWaitingWordFlush", "120"));
|
|
wordIndex.close(waitingBoundSeconds);
|
|
log.logConfig("SWITCHBOARD SHUTDOWN STEP 3: sending termination signal to database manager");
|
|
try {
|
|
//sbStackCrawlThread.stopIt();
|
|
indexDistribution.close();
|
|
cacheLoader.close();
|
|
wikiDB.close();
|
|
userDB.close();
|
|
messageDB.close();
|
|
if (facilityDB != null) facilityDB.close();
|
|
sbStackCrawlThread.close();
|
|
urlPool.close();
|
|
profiles.close();
|
|
robots.close();
|
|
parser.close();
|
|
cacheManager.close();
|
|
sbQueue.close();
|
|
} catch (IOException e) {}
|
|
log.logConfig("SWITCHBOARD SHUTDOWN TERMINATED");
|
|
}
|
|
|
|
public int queueSize() {
|
|
return sbQueue.size();
|
|
//return processStack.size() + cacheLoader.size() + noticeURL.stackSize();
|
|
}
|
|
|
|
public int cacheSizeMin() {
|
|
return wordIndex.size();
|
|
}
|
|
|
|
public void enQueue(Object job) {
|
|
if (!(job instanceof plasmaSwitchboardQueue.Entry)) {
|
|
System.out.println("internal error at plasmaSwitchboard.enQueue: wrong job type");
|
|
System.exit(0);
|
|
}
|
|
try {
|
|
sbQueue.push((plasmaSwitchboardQueue.Entry) job);
|
|
} catch (IOException e) {
|
|
log.logSevere("IOError in plasmaSwitchboard.enQueue: " + e.getMessage(), e);
|
|
}
|
|
}
|
|
|
|
public boolean deQueue() {
|
|
// work off fresh entries from the proxy or from the crawler
|
|
if (onlineCaution()) {
|
|
log.logFiner("deQueue: online caution, omitting resource stack processing");
|
|
return false;
|
|
}
|
|
|
|
if (wordIndex.wordCacheRAMSize() + 1000 > (int) getConfigLong("wordCacheMaxLow", 8000)) {
|
|
log.logFine("deQueue: word index ram cache too full (" + ((int) getConfigLong("wordCacheMaxLow", 8000) - wordIndex.wordCacheRAMSize()) +
|
|
" slots left); dismissed to omit ram flush lock");
|
|
return false;
|
|
}
|
|
|
|
int stackCrawlQueueSize;
|
|
if ((stackCrawlQueueSize = sbStackCrawlThread.size()) >= stackCrawlSlots) {
|
|
log.logFine("deQueue: too many processes in stack crawl thread queue, dismissed to protect emergency case (" +
|
|
"stackCrawlQueue=" + stackCrawlQueueSize + ")");
|
|
return false;
|
|
}
|
|
|
|
plasmaSwitchboardQueue.Entry nextentry;
|
|
synchronized (sbQueue) {
|
|
if (sbQueue.size() == 0) {
|
|
//log.logDebug("DEQUEUE: queue is empty");
|
|
return false; // nothing to do
|
|
}
|
|
|
|
// do one processing step
|
|
log.logFine("DEQUEUE: sbQueueSize=" + sbQueue.size() +
|
|
", coreStackSize=" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) +
|
|
", limitStackSize=" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_LIMIT) +
|
|
", overhangStackSize=" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_OVERHANG) +
|
|
", remoteStackSize=" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE));
|
|
try {
|
|
nextentry = sbQueue.pop();
|
|
if (nextentry == null) return false;
|
|
|
|
synchronized (this.indexingTasksInProcess) {
|
|
this.indexingTasksInProcess.put(nextentry.urlHash(),nextentry);
|
|
}
|
|
|
|
} catch (IOException e) {
|
|
log.logSevere("IOError in plasmaSwitchboard.deQueue: " + e.getMessage(), e);
|
|
return false;
|
|
}
|
|
}
|
|
processResourceStack(nextentry);
|
|
return true;
|
|
}
|
|
|
|
public int cleanupJobSize() {
|
|
int c = 0;
|
|
if ((urlPool.errorURL.stackSize() > 1000)) c++;
|
|
for (int i = 1; i <= 6; i++) {
|
|
if (urlPool.loadedURL.getStackSize(i) > 1000) c++;
|
|
}
|
|
return c;
|
|
}
|
|
|
|
public boolean cleanupJob() {
|
|
|
|
boolean hasDoneSomething = false;
|
|
|
|
// clean up error stack
|
|
if ((urlPool.errorURL.stackSize() > 1000)) {
|
|
log.logFine("Cleaning Error-URLs report stack, " + urlPool.errorURL.stackSize() + " entries on stack");
|
|
urlPool.errorURL.clearStack();
|
|
hasDoneSomething = true;
|
|
}
|
|
// clean up loadedURL stack
|
|
for (int i = 1; i <= 6; i++) {
|
|
if (urlPool.loadedURL.getStackSize(i) > 1000) {
|
|
log.logFine("Cleaning Loaded-URLs report stack, " + urlPool.loadedURL.getStackSize(i) + " entries on stack " + i);
|
|
urlPool.loadedURL.clearStack(i);
|
|
hasDoneSomething = true;
|
|
}
|
|
}
|
|
// clean up profiles
|
|
if (cleanProfiles()) hasDoneSomething = true;
|
|
|
|
// clean up news
|
|
try {
|
|
log.logFine("Cleaning Incoming News, " + yacyCore.newsPool.size(yacyNewsPool.INCOMING_DB) + " entries on stack");
|
|
if (yacyCore.newsPool.automaticProcess() > 0) hasDoneSomething = true;
|
|
} catch (IOException e) {}
|
|
|
|
return hasDoneSomething;
|
|
}
|
|
|
|
/**
|
|
* With this function the crawling process can be paused
|
|
*/
|
|
public void pauseCrawling() {
|
|
synchronized(this.crawlingPausedSync) {
|
|
this.crawlingIsPaused = true;
|
|
}
|
|
setConfig("crawler.isPaused", "true");
|
|
}
|
|
|
|
/**
|
|
* Continue the previously paused crawling
|
|
*/
|
|
public void continueCrawling() {
|
|
synchronized(this.crawlingPausedSync) {
|
|
if (this.crawlingIsPaused) {
|
|
this.crawlingIsPaused = false;
|
|
this.crawlingPausedSync.notifyAll();
|
|
}
|
|
}
|
|
setConfig("crawler.isPaused", "false");
|
|
}
|
|
|
|
/**
|
|
* @return <code>true</code> if crawling was paused or <code>false</code> otherwise
|
|
*/
|
|
public boolean crawlingIsPaused() {
|
|
synchronized(this.crawlingPausedSync) {
|
|
return this.crawlingIsPaused;
|
|
}
|
|
}
|
|
|
|
public int coreCrawlJobSize() {
|
|
return urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE);
|
|
}
|
|
|
|
public boolean coreCrawlJob() {
|
|
if (urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) == 0) {
|
|
//log.logDebug("CoreCrawl: queue is empty");
|
|
return false;
|
|
}
|
|
if (sbQueue.size() >= indexingSlots) {
|
|
log.logFine("CoreCrawl: too many processes in indexing queue, dismissed (" +
|
|
"sbQueueSize=" + sbQueue.size() + ")");
|
|
return false;
|
|
}
|
|
if (cacheLoader.size() >= crawlSlots) {
|
|
log.logFine("CoreCrawl: too many processes in loader queue, dismissed (" +
|
|
"cacheLoader=" + cacheLoader.size() + ")");
|
|
return false;
|
|
}
|
|
if (onlineCaution()) {
|
|
log.logFine("CoreCrawl: online caution, omitting processing");
|
|
return false;
|
|
}
|
|
// if the server is busy, we do crawling more slowly
|
|
//if (!(cacheManager.idle())) try {Thread.currentThread().sleep(2000);} catch (InterruptedException e) {}
|
|
|
|
// if crawling was paused we have to wait until we wer notified to continue
|
|
synchronized(this.crawlingPausedSync) {
|
|
if (this.crawlingIsPaused) {
|
|
try {
|
|
this.crawlingPausedSync.wait();
|
|
}
|
|
catch (InterruptedException e){ return false;}
|
|
}
|
|
}
|
|
|
|
// do a local crawl
|
|
plasmaCrawlNURL.Entry urlEntry = urlPool.noticeURL.pop(plasmaCrawlNURL.STACK_TYPE_CORE);
|
|
String stats = "LOCALCRAWL[" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_LIMIT) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_OVERHANG) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE) + "]";
|
|
if ((urlEntry.url() == null) || (urlEntry.url().toString().length() < 10)) {
|
|
log.logInfo(stats + ": URL with hash " + ((urlEntry.hash()==null)?"Unknown":urlEntry.hash()) + " already removed from queue.");
|
|
return true;
|
|
}
|
|
String profileHandle = urlEntry.profileHandle();
|
|
//System.out.println("DEBUG plasmaSwitchboard.processCrawling: profileHandle = " + profileHandle + ", urlEntry.url = " + urlEntry.url());
|
|
if (profileHandle == null) {
|
|
log.logSevere(stats + ": NULL PROFILE HANDLE '" + urlEntry.profileHandle() + "' (must be internal error) for URL " + urlEntry.url());
|
|
return true;
|
|
}
|
|
plasmaCrawlProfile.entry profile = profiles.getEntry(profileHandle);
|
|
if (profile == null) {
|
|
log.logSevere(stats + ": LOST PROFILE HANDLE '" + urlEntry.profileHandle() + "' (must be internal error) for URL " + urlEntry.url());
|
|
return true;
|
|
}
|
|
log.logFine("LOCALCRAWL: URL=" + urlEntry.url() + ", initiator=" + urlEntry.initiator() +
|
|
", crawlOrder=" + ((profile.remoteIndexing()) ? "true" : "false") + ", depth=" + urlEntry.depth() + ", crawlDepth=" + profile.generalDepth() + ", filter=" + profile.generalFilter() +
|
|
", permission=" + ((yacyCore.seedDB == null) ? "undefined" : (((yacyCore.seedDB.mySeed.isSenior()) || (yacyCore.seedDB.mySeed.isPrincipal())) ? "true" : "false")));
|
|
|
|
processLocalCrawling(urlEntry, profile, stats);
|
|
return true;
|
|
}
|
|
|
|
public int limitCrawlTriggerJobSize() {
|
|
return urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_LIMIT);
|
|
}
|
|
|
|
public boolean limitCrawlTriggerJob() {
|
|
if (urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_LIMIT) == 0) {
|
|
//log.logDebug("LimitCrawl: queue is empty");
|
|
return false;
|
|
}
|
|
|
|
if ((coreCrawlJobSize() <= 20) && (limitCrawlTriggerJobSize() > 100)) {
|
|
// it is not efficient if the core crawl job is empty and we have too much to do
|
|
// move some tasks to the core crawl job
|
|
int toshift = limitCrawlTriggerJobSize() / 5;
|
|
if (toshift > 1000) toshift = 1000;
|
|
if (toshift > limitCrawlTriggerJobSize()) toshift = limitCrawlTriggerJobSize();
|
|
try {
|
|
for (int i = 0; i < toshift; i++) {
|
|
urlPool.noticeURL.shift(plasmaCrawlNURL.STACK_TYPE_LIMIT, plasmaCrawlNURL.STACK_TYPE_CORE);
|
|
}
|
|
log.logInfo("shifted " + toshift + " jobs from global crawl to local crawl");
|
|
} catch (IOException e) {}
|
|
}
|
|
|
|
|
|
if (sbQueue.size() >= indexingSlots) {
|
|
log.logFine("LimitCrawl: too many processes in indexing queue, dismissed to protect emergency case (" +
|
|
"sbQueueSize=" + sbQueue.size() + ")");
|
|
return false;
|
|
}
|
|
if (cacheLoader.size() >= crawlSlots) {
|
|
log.logFine("LimitCrawl: too many processes in loader queue, dismissed to protect emergency case (" +
|
|
"cacheLoader=" + cacheLoader.size() + ")");
|
|
return false;
|
|
}
|
|
|
|
|
|
// if the server is busy, we do crawling more slowly
|
|
//if (!(cacheManager.idle())) try {Thread.currentThread().sleep(2000);} catch (InterruptedException e) {}
|
|
|
|
// if crawling was paused we have to wait until we wer notified to continue
|
|
synchronized(this.crawlingPausedSync) {
|
|
if (this.crawlingIsPaused) {
|
|
try {
|
|
this.crawlingPausedSync.wait();
|
|
}
|
|
catch (InterruptedException e){ return false;}
|
|
}
|
|
}
|
|
|
|
// start a global crawl, if possible
|
|
plasmaCrawlNURL.Entry urlEntry = urlPool.noticeURL.pop(plasmaCrawlNURL.STACK_TYPE_LIMIT);
|
|
String stats = "REMOTECRAWLTRIGGER[" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_LIMIT) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_OVERHANG) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE) + "]";
|
|
if (urlEntry.url() == null) {
|
|
log.logSevere(stats + ": urlEntry.url() == null");
|
|
return true;
|
|
}
|
|
String profileHandle = urlEntry.profileHandle();
|
|
//System.out.println("DEBUG plasmaSwitchboard.processCrawling: profileHandle = " + profileHandle + ", urlEntry.url = " + urlEntry.url());
|
|
plasmaCrawlProfile.entry profile = profiles.getEntry(profileHandle);
|
|
if (profile == null) {
|
|
log.logSevere(stats + ": LOST PROFILE HANDLE '" + urlEntry.profileHandle() + "' (must be internal error) for URL " + urlEntry.url());
|
|
return true;
|
|
}
|
|
log.logFine("plasmaSwitchboard.limitCrawlTriggerJob: url=" + urlEntry.url() + ", initiator=" + urlEntry.initiator() +
|
|
", crawlOrder=" + ((profile.remoteIndexing()) ? "true" : "false") + ", depth=" + urlEntry.depth() + ", crawlDepth=" + profile.generalDepth() + ", filter=" + profile.generalFilter() +
|
|
", permission=" + ((yacyCore.seedDB == null) ? "undefined" : (((yacyCore.seedDB.mySeed.isSenior()) || (yacyCore.seedDB.mySeed.isPrincipal())) ? "true" : "false")));
|
|
|
|
boolean tryRemote =
|
|
((urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) != 0) || (sbQueue.size() != 0)) /* should do ourself */ &&
|
|
(profile.remoteIndexing()) /* granted */ &&
|
|
(urlEntry.initiator() != null) && (!(urlEntry.initiator().equals(plasmaURL.dummyHash))) /* not proxy */ &&
|
|
((yacyCore.seedDB.mySeed.isSenior()) ||
|
|
(yacyCore.seedDB.mySeed.isPrincipal())) /* qualified */;
|
|
|
|
if (tryRemote) {
|
|
boolean success = processRemoteCrawlTrigger(urlEntry);
|
|
if (success) return true;
|
|
}
|
|
|
|
processLocalCrawling(urlEntry, profile, stats);
|
|
return true;
|
|
}
|
|
|
|
public int remoteTriggeredCrawlJobSize() {
|
|
return urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE);
|
|
}
|
|
|
|
public boolean remoteTriggeredCrawlJob() {
|
|
// work off crawl requests that had been placed by other peers to our crawl stack
|
|
|
|
// do nothing if either there are private processes to be done
|
|
// or there is no global crawl on the stack
|
|
if (urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE) == 0) {
|
|
//log.logDebug("GlobalCrawl: queue is empty");
|
|
return false;
|
|
}
|
|
if (onlineCaution()) {
|
|
log.logFine("GlobalCrawl: online caution, omitting processing");
|
|
return false;
|
|
}
|
|
|
|
// if crawling was paused we have to wait until we wer notified to continue
|
|
synchronized(this.crawlingPausedSync) {
|
|
if (this.crawlingIsPaused) {
|
|
try {
|
|
this.crawlingPausedSync.wait();
|
|
}
|
|
catch (InterruptedException e){ return false; }
|
|
}
|
|
}
|
|
|
|
// we don't want to crawl a global URL globally, since WE are the global part. (from this point of view)
|
|
plasmaCrawlNURL.Entry urlEntry = urlPool.noticeURL.pop(plasmaCrawlNURL.STACK_TYPE_REMOTE);
|
|
String stats = "REMOTETRIGGEREDCRAWL[" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_LIMIT) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_OVERHANG) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE) + "]";
|
|
if (urlEntry.url() == null) {
|
|
log.logSevere(stats + ": urlEntry.url() == null");
|
|
return false;
|
|
}
|
|
String profileHandle = urlEntry.profileHandle();
|
|
//System.out.println("DEBUG plasmaSwitchboard.processCrawling: profileHandle = " + profileHandle + ", urlEntry.url = " + urlEntry.url());
|
|
plasmaCrawlProfile.entry profile = profiles.getEntry(profileHandle);
|
|
|
|
if (profile == null) {
|
|
log.logSevere(stats + ": LOST PROFILE HANDLE '" + urlEntry.profileHandle() + "' (must be internal error) for URL " + urlEntry.url());
|
|
return false;
|
|
}
|
|
log.logFine("plasmaSwitchboard.remoteTriggeredCrawlJob: url=" + urlEntry.url() + ", initiator=" + urlEntry.initiator() +
|
|
", crawlOrder=" + ((profile.remoteIndexing()) ? "true" : "false") + ", depth=" + urlEntry.depth() + ", crawlDepth=" + profile.generalDepth() + ", filter=" + profile.generalFilter() +
|
|
", permission=" + ((yacyCore.seedDB == null) ? "undefined" : (((yacyCore.seedDB.mySeed.isSenior()) || (yacyCore.seedDB.mySeed.isPrincipal())) ? "true" : "false")));
|
|
|
|
processLocalCrawling(urlEntry, profile, stats);
|
|
return true;
|
|
}
|
|
|
|
private void processResourceStack(plasmaSwitchboardQueue.Entry entry) {
|
|
// work off one stack entry with a fresh resource
|
|
try {
|
|
long stackStartTime = 0, stackEndTime = 0,
|
|
parsingStartTime = 0, parsingEndTime = 0,
|
|
indexingStartTime = 0, indexingEndTime = 0,
|
|
storageStartTime = 0, storageEndTime = 0;
|
|
|
|
// we must distinguish the following cases: resource-load was initiated by
|
|
// 1) global crawling: the index is extern, not here (not possible here)
|
|
// 2) result of search queries, some indexes are here (not possible here)
|
|
// 3) result of index transfer, some of them are here (not possible here)
|
|
// 4) proxy-load (initiator is "------------")
|
|
// 5) local prefetch/crawling (initiator is own seedHash)
|
|
// 6) local fetching for global crawling (other known or unknwon initiator)
|
|
int processCase = 0;
|
|
yacySeed initiator = null;
|
|
String initiatorHash = (entry.proxy()) ? plasmaURL.dummyHash : entry.initiator();
|
|
if (initiatorHash.equals(plasmaURL.dummyHash)) {
|
|
// proxy-load
|
|
processCase = 4;
|
|
} else if (initiatorHash.equals(yacyCore.seedDB.mySeed.hash)) {
|
|
// normal crawling
|
|
processCase = 5;
|
|
} else {
|
|
// this was done for remote peer (a global crawl)
|
|
initiator = yacyCore.seedDB.getConnected(initiatorHash);
|
|
processCase = 6;
|
|
}
|
|
|
|
log.logFine("processResourceStack processCase=" + processCase +
|
|
", depth=" + entry.depth() +
|
|
", maxDepth=" + ((entry.profile() == null) ? "null" : Integer.toString(entry.profile().generalDepth())) +
|
|
", filter=" + ((entry.profile() == null) ? "null" : entry.profile().generalFilter()) +
|
|
", initiatorHash=" + initiatorHash +
|
|
", responseHeader=" + ((entry.responseHeader() == null) ? "null" : entry.responseHeader().toString()) +
|
|
", url=" + entry.url()); // DEBUG
|
|
|
|
// parse content
|
|
parsingStartTime = System.currentTimeMillis();
|
|
plasmaParserDocument document = null;
|
|
httpHeader entryRespHeader = entry.responseHeader();
|
|
String mimeType = (entryRespHeader == null)?null:entryRespHeader.mime();
|
|
if (plasmaParser.supportedContent(entry.url(),mimeType)){
|
|
if ((entry.cacheFile().exists()) && (entry.cacheFile().length() > 0)) {
|
|
log.logFine("(Parser) '" + entry.normalizedURLString() + "' is not parsed yet, parsing now from File");
|
|
document = parser.parseSource(entry.url(), mimeType, entry.cacheFile());
|
|
} else {
|
|
log.logFine("(Parser) '" + entry.normalizedURLString() + "' cannot be parsed, no resource available");
|
|
return;
|
|
}
|
|
if (document == null) {
|
|
log.logSevere("(Parser) '" + entry.normalizedURLString() + "' parse failure");
|
|
return;
|
|
}
|
|
} else {
|
|
log.logFine("(Parser) '" + entry.normalizedURLString() + "'. Unsupported mimeType '" + ((mimeType == null) ? "null" : mimeType) + "'.");
|
|
return;
|
|
}
|
|
parsingEndTime = System.currentTimeMillis();
|
|
|
|
Date loadDate = null;
|
|
if (entry.responseHeader() != null) {
|
|
loadDate = entry.responseHeader().lastModified();
|
|
if (loadDate == null) loadDate = entry.responseHeader().date();
|
|
}
|
|
if (loadDate == null) loadDate = new Date();
|
|
|
|
// put anchors on crawl stack
|
|
stackStartTime = System.currentTimeMillis();
|
|
if (((processCase == 4) || (processCase == 5)) &&
|
|
((entry.profile() == null) || (entry.depth() < entry.profile().generalDepth()))) {
|
|
Map hl = document.getHyperlinks();
|
|
Iterator i = hl.entrySet().iterator();
|
|
String nexturlstring;
|
|
String rejectReason;
|
|
int c = 0;
|
|
Map.Entry e;
|
|
while (i.hasNext()) {
|
|
e = (Map.Entry) i.next();
|
|
nexturlstring = (String) e.getKey();
|
|
|
|
sbStackCrawlThread.enqueue(nexturlstring, entry.url().toString(), initiatorHash, (String) e.getValue(), loadDate, entry.depth() + 1, entry.profile());
|
|
|
|
// rejectReason = stackCrawl(nexturlstring, entry.normalizedURLString(), initiatorHash, (String) e.getValue(), loadDate, entry.depth() + 1, entry.profile());
|
|
// if (rejectReason == null) {
|
|
// c++;
|
|
// } else {
|
|
// urlPool.errorURL.newEntry(new URL(nexturlstring), entry.normalizedURLString(), entry.initiator(), yacyCore.seedDB.mySeed.hash,
|
|
// (String) e.getValue(), rejectReason, new bitfield(plasmaURL.urlFlagLength), false);
|
|
// }
|
|
}
|
|
log.logInfo("CRAWL: ADDED " + hl.size() + " LINKS FROM " + entry.normalizedURLString() +
|
|
", NEW CRAWL STACK SIZE IS " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE));
|
|
}
|
|
stackEndTime = System.currentTimeMillis();
|
|
|
|
// create index
|
|
String descr = document.getMainLongTitle();
|
|
URL referrerURL = entry.referrerURL();
|
|
String referrerHash = (referrerURL == null) ? plasmaURL.dummyHash : plasmaURL.urlHash(referrerURL);
|
|
String noIndexReason = "unspecified";
|
|
if (processCase == 4) {
|
|
// proxy-load
|
|
noIndexReason = entry.shallIndexCacheForProxy();
|
|
} else {
|
|
// normal crawling
|
|
noIndexReason = entry.shallIndexCacheForCrawler();
|
|
}
|
|
if (noIndexReason == null) {
|
|
// strip out words
|
|
indexingStartTime = System.currentTimeMillis();
|
|
log.logFine("Condensing for '" + entry.normalizedURLString() + "'");
|
|
plasmaCondenser condenser = new plasmaCondenser(new ByteArrayInputStream(document.getText()));
|
|
|
|
//log.logInfo("INDEXING HEADLINE:" + descr);
|
|
try {
|
|
//log.logDebug("Create LURL-Entry for '" + entry.normalizedURLString() + "', " +
|
|
// "responseHeader=" + entry.responseHeader().toString());
|
|
|
|
Date lastModified = null;
|
|
if (entry.responseHeader() == null) {
|
|
lastModified = new Date();
|
|
} else {
|
|
lastModified = entry.responseHeader().lastModified();
|
|
if (lastModified == null) lastModified = entry.responseHeader().date();
|
|
if (lastModified == null) lastModified = new Date();
|
|
}
|
|
plasmaCrawlLURL.Entry newEntry = urlPool.loadedURL.addEntry(
|
|
entry.url(), descr, lastModified, new Date(),
|
|
initiatorHash,
|
|
yacyCore.seedDB.mySeed.hash,
|
|
referrerHash,
|
|
0, true,
|
|
Integer.parseInt(condenser.getAnalysis().getProperty("INFORMATION_VALUE","0"), 16),
|
|
plasmaWordIndexEntry.language(entry.url()),
|
|
plasmaWordIndexEntry.docType(document.getMimeType()),
|
|
entry.size(),
|
|
(int) Long.parseLong(condenser.getAnalysis().getProperty("NUMB_WORDS","0"), 16),
|
|
processCase
|
|
);
|
|
|
|
String urlHash = newEntry.hash();
|
|
//log.logDebug("Remove NURL for '" + entry.normalizedURLString() + "'");
|
|
urlPool.noticeURL.remove(urlHash); // worked-off
|
|
|
|
if (((processCase == 4) || (processCase == 5) || (processCase == 6)) &&
|
|
(entry.profile().localIndexing())) {
|
|
// remove stopwords
|
|
log.logInfo("Excluded " + condenser.excludeWords(stopwords) + " words in URL " + entry.url());
|
|
indexingEndTime = System.currentTimeMillis();
|
|
|
|
// do indexing
|
|
//log.logDebug("Create Index for '" + entry.normalizedURLString() + "'");
|
|
storageStartTime = System.currentTimeMillis();
|
|
int words = 0;
|
|
String storagePeerHash;
|
|
yacySeed seed;
|
|
if (((storagePeerHash = getConfig("storagePeerHash",null))== null) ||
|
|
(storagePeerHash.trim().length() == 0) ||
|
|
((seed = yacyCore.seedDB.getConnected(storagePeerHash))==null)){
|
|
words = wordIndex.addPageIndex(entry.url(), urlHash, loadDate, condenser, plasmaWordIndexEntry.language(entry.url()), plasmaWordIndexEntry.docType(document.getMimeType()));
|
|
} else {
|
|
HashMap urlCache = new HashMap(1);
|
|
urlCache.put(newEntry.hash(),newEntry);
|
|
|
|
ArrayList tmpEntities = new ArrayList(condenser.getWords().size());
|
|
|
|
int age = plasmaWordIndex.calcVirtualAge(lastModified);
|
|
int quality = 0;
|
|
try {
|
|
quality = Integer.parseInt(condenser.getAnalysis().getProperty("INFORMATION_VALUE","0"), 16);
|
|
} catch (NumberFormatException e) {
|
|
System.out.println("INTERNAL ERROR WITH CONDENSER.INFORMATION_VALUE: " + e.toString() + ": in URL " + newEntry.url().toString());
|
|
}
|
|
|
|
String language = plasmaWordIndexEntry.language(entry.url());
|
|
char doctype = plasmaWordIndexEntry.docType(document.getMimeType());
|
|
|
|
// iterate over all words
|
|
Iterator i = condenser.getWords().iterator();
|
|
int p = 0;
|
|
while (i.hasNext()) {
|
|
String word = (String) i.next();
|
|
int count = condenser.wordCount(word);
|
|
String wordHash = plasmaWordIndexEntry.word2hash(word);
|
|
plasmaWordIndexEntity wordIdxEntity = new plasmaWordIndexEntity(wordHash);
|
|
plasmaWordIndexEntry wordIdxEntry = new plasmaWordIndexEntry(urlHash, count, p++, 0, 0,
|
|
age, quality, language, doctype, true);
|
|
wordIdxEntity.addEntry(wordIdxEntry);
|
|
tmpEntities.add(wordIdxEntity);
|
|
// wordIndex.addEntries(plasmaWordIndexEntryContainer.instantContainer(wordHash, System.currentTimeMillis(), entry));
|
|
}
|
|
//System.out.println("DEBUG: plasmaSearch.addPageIndex: added " + condenser.getWords().size() + " words, flushed " + c + " entries");
|
|
words = condenser.getWords().size();
|
|
|
|
// transfering the index to the storage peer
|
|
String error = yacyClient.transferIndex(seed,(plasmaWordIndexEntity[])tmpEntities.toArray(new plasmaWordIndexEntity[tmpEntities.size()]),urlCache,true,120000);
|
|
|
|
if (error != null) {
|
|
words = wordIndex.addPageIndex(entry.url(), urlHash, loadDate, condenser, plasmaWordIndexEntry.language(entry.url()), plasmaWordIndexEntry.docType(document.getMimeType()));
|
|
}
|
|
|
|
// cleanup
|
|
for (int j=0; j < tmpEntities.size(); j++) {
|
|
plasmaWordIndexEntity tmpEntity = (plasmaWordIndexEntity) tmpEntities.get(j);
|
|
try { tmpEntity.close(); } catch (Exception e) {}
|
|
}
|
|
}
|
|
storageEndTime = System.currentTimeMillis();
|
|
|
|
if (log.isLoggable(Level.INFO)) {
|
|
log.logInfo("*Indexed " + words + " words in URL " + entry.url() +
|
|
"\n\tDescription: " + descr +
|
|
"\n\tMimeType: " + document.getMimeType() + " | " +
|
|
"Size: " + document.text.length + " bytes | " +
|
|
"Anchors: " + ((document.anchors==null)?0:document.anchors.size()) +
|
|
"\n\tStackingTime: " + (stackEndTime-stackStartTime) + " ms | " +
|
|
"ParsingTime: " + (parsingEndTime-parsingStartTime) + " ms | " +
|
|
"IndexingTime: " + (indexingEndTime-indexingStartTime) + " ms | " +
|
|
"StorageTime: " + (storageEndTime-storageStartTime) + " ms");
|
|
}
|
|
|
|
// if this was performed for a remote crawl request, notify requester
|
|
if ((processCase == 6) && (initiator != null)) {
|
|
log.logInfo("Sending crawl receipt for '" + entry.normalizedURLString() + "' to " + initiator.getName());
|
|
yacyClient.crawlReceipt(initiator, "crawl", "fill", "indexed", newEntry, "");
|
|
}
|
|
} else {
|
|
log.logFine("Not Indexed Resource '" + entry.normalizedURLString() + "': process case=" + processCase);
|
|
}
|
|
} catch (Exception ee) {
|
|
log.logSevere("Could not index URL " + entry.url() + ": " + ee.getMessage(), ee);
|
|
if ((processCase == 6) && (initiator != null)) {
|
|
yacyClient.crawlReceipt(initiator, "crawl", "exception", ee.getMessage(), null, "");
|
|
}
|
|
}
|
|
|
|
} else {
|
|
log.logInfo("Not indexed any word in URL " + entry.url() + "; cause: " + noIndexReason);
|
|
urlPool.errorURL.newEntry(entry.url(), referrerHash,
|
|
((entry.proxy()) ? plasmaURL.dummyHash : entry.initiator()),
|
|
yacyCore.seedDB.mySeed.hash,
|
|
descr, noIndexReason, new bitfield(plasmaURL.urlFlagLength), true);
|
|
if ((processCase == 6) && (initiator != null)) {
|
|
yacyClient.crawlReceipt(initiator, "crawl", "rejected", noIndexReason, null, "");
|
|
}
|
|
}
|
|
|
|
document = null;
|
|
|
|
} catch (IOException e) {
|
|
log.logSevere("ERROR in plasmaSwitchboard.process(): " + e.toString());
|
|
} finally {
|
|
synchronized (this.indexingTasksInProcess) {
|
|
this.indexingTasksInProcess.remove(entry.urlHash());
|
|
}
|
|
|
|
// explicit delete/free resources
|
|
if ((entry != null) && (entry.profile() != null) && (!(entry.profile().storeHTCache()))) {
|
|
cacheManager.filesInUse.remove(entry.cacheFile());
|
|
cacheManager.deleteFile(entry.url());
|
|
}
|
|
entry = null;
|
|
}
|
|
}
|
|
|
|
|
|
|
|
private void processLocalCrawling(plasmaCrawlNURL.Entry urlEntry, plasmaCrawlProfile.entry profile, String stats) {
|
|
// work off one Crawl stack entry
|
|
if ((urlEntry == null) || (urlEntry.url() == null)) {
|
|
log.logInfo(stats + ": urlEntry=null");
|
|
return;
|
|
}
|
|
|
|
URL refererURL = null;
|
|
String refererHash = urlEntry.referrerHash();
|
|
if ((refererHash != null) && (!refererHash.equals(plasmaURL.dummyHash))) {
|
|
refererURL = this.urlPool.getURL(refererHash);
|
|
}
|
|
cacheLoader.loadParallel(urlEntry.url(), urlEntry.name(), (refererURL!=null)?refererURL.toString():null, urlEntry.initiator(), urlEntry.depth(), profile);
|
|
log.logInfo(stats + ": enqueued for load " + urlEntry.url() + " [" + urlEntry.hash() + "]");
|
|
return;
|
|
}
|
|
|
|
private boolean processRemoteCrawlTrigger(plasmaCrawlNURL.Entry urlEntry) {
|
|
|
|
// return true iff another peer has/will index(ed) the url
|
|
if (urlEntry == null) {
|
|
log.logInfo("REMOTECRAWLTRIGGER[" + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_CORE) + ", " + urlPool.noticeURL.stackSize(plasmaCrawlNURL.STACK_TYPE_REMOTE) + "]: urlEntry=null");
|
|
return true; // superfluous request; true correct in this context
|
|
}
|
|
|
|
// are we qualified?
|
|
if ((yacyCore.seedDB.mySeed == null) ||
|
|
(yacyCore.seedDB.mySeed.isJunior())) {
|
|
log.logFine("plasmaSwitchboard.processRemoteCrawlTrigger: no permission");
|
|
return false;
|
|
}
|
|
|
|
// check url
|
|
if (urlEntry.url() == null) {
|
|
log.logFine("ERROR: plasmaSwitchboard.processRemoteCrawlTrigger - url is null. name=" + urlEntry.name());
|
|
return true;
|
|
}
|
|
String urlhash = plasmaURL.urlHash(urlEntry.url());
|
|
|
|
// check remote crawl
|
|
yacySeed remoteSeed = yacyCore.dhtAgent.getCrawlSeed(urlhash);
|
|
|
|
if (remoteSeed == null) {
|
|
log.logFine("plasmaSwitchboard.processRemoteCrawlTrigger: no remote crawl seed available");
|
|
return false;
|
|
}
|
|
|
|
// do the request
|
|
HashMap page = yacyClient.crawlOrder(remoteSeed, urlEntry.url(), urlPool.getURL(urlEntry.referrerHash()));
|
|
|
|
|
|
// check success
|
|
/*
|
|
the result of the 'response' value can have one of the following values:
|
|
negative cases, no retry
|
|
denied - the peer does not want to crawl that
|
|
exception - an exception occurred
|
|
|
|
negative case, retry possible
|
|
rejected - the peer has rejected to process, but a re-try should be possible
|
|
|
|
positive case with crawling
|
|
stacked - the resource is processed asap
|
|
|
|
positive case without crawling
|
|
double - the resource is already in database, believed to be fresh and not reloaded
|
|
the resource is also returned in lurl
|
|
*/
|
|
if ((page == null) || (page.get("delay") == null)) {
|
|
log.logInfo("CRAWL: REMOTE CRAWL TO PEER " + remoteSeed.getName() + " FAILED. CAUSE: unknown (URL=" + urlEntry.url().toString() + ")");
|
|
if (remoteSeed != null) yacyCore.peerActions.peerDeparture(remoteSeed);
|
|
return false;
|
|
} else try {
|
|
log.logFine("plasmaSwitchboard.processRemoteCrawlTrigger: remoteSeed=" + remoteSeed.getName() + ", url=" + urlEntry.url().toString() + ", response=" + page.toString()); // DEBUG
|
|
|
|
int newdelay = Integer.parseInt((String) page.get("delay"));
|
|
yacyCore.dhtAgent.setCrawlDelay(remoteSeed.hash, newdelay);
|
|
String response = (String) page.get("response");
|
|
if (response.equals("stacked")) {
|
|
log.logInfo(STR_REMOTECRAWLTRIGGER + remoteSeed.getName() + " PLACED URL=" + urlEntry.url().toString() + "; NEW DELAY=" + newdelay);
|
|
return true;
|
|
} else if (response.equals("double")) {
|
|
String lurl = (String) page.get("lurl");
|
|
if ((lurl != null) && (lurl.length() != 0)) {
|
|
String propStr = crypt.simpleDecode(lurl, (String) page.get("key"));
|
|
plasmaCrawlLURL.Entry entry = urlPool.loadedURL.addEntry(
|
|
urlPool.loadedURL.newEntry(propStr, true),
|
|
yacyCore.seedDB.mySeed.hash, remoteSeed.hash, 1);
|
|
urlPool.noticeURL.remove(entry.hash());
|
|
log.logInfo(STR_REMOTECRAWLTRIGGER + remoteSeed.getName() + " SUPERFLUOUS. CAUSE: " + page.get("reason") + " (URL=" + urlEntry.url().toString() + "). URL IS CONSIDERED AS 'LOADED!'");
|
|
return true;
|
|
} else {
|
|
log.logInfo(STR_REMOTECRAWLTRIGGER + remoteSeed.getName() + " REJECTED. CAUSE: " + page.get("reason") + " (URL=" + urlEntry.url().toString() + ")");
|
|
return false;
|
|
}
|
|
} else {
|
|
log.logInfo(STR_REMOTECRAWLTRIGGER + remoteSeed.getName() + " DENIED. RESPONSE=" + response + ", CAUSE=" + page.get("reason") + ", URL=" + urlEntry.url().toString());
|
|
return false;
|
|
}
|
|
} catch (Exception e) {
|
|
// wrong values
|
|
log.logSevere(STR_REMOTECRAWLTRIGGER + remoteSeed.getName() + " FAILED. CLIENT RETURNED: " + page.toString(), e);
|
|
return false;
|
|
}
|
|
}
|
|
|
|
private static SimpleDateFormat DateFormatter = new SimpleDateFormat("EEE, dd MMM yyyy");
|
|
public static String dateString(Date date) {
|
|
if (date == null) return ""; else return DateFormatter.format(date);
|
|
}
|
|
|
|
/*
|
|
public class presearch extends Thread {
|
|
Set queryhashes;
|
|
char[] order;
|
|
String urlmask;
|
|
long time;
|
|
int searchcount, fetchcount;
|
|
public presearch(Set queryhashes, char[] order, long time, String urlmask, int searchcount, int fetchcount) {
|
|
this.queryhashes = queryhashes;
|
|
this.order = order;
|
|
this.urlmask = urlmask;
|
|
this.time = time;
|
|
this.searchcount = searchcount;
|
|
this.fetchcount = fetchcount;
|
|
}
|
|
public void run() {
|
|
plasmaWordIndexEntity idx = null;
|
|
try {
|
|
// search the database locally
|
|
log.logFine("presearch: started job");
|
|
idx = searchManager.searchHashes(queryhashes, time);
|
|
log.logFine("presearch: found " + idx.size() + " results");
|
|
plasmaSearchResult acc = searchManager.order(idx, queryhashes, stopwords, order, time, searchcount);
|
|
if (acc == null) return;
|
|
log.logFine("presearch: ordered results, now " + acc.sizeOrdered() + " URLs ready for fetch");
|
|
|
|
// take some elements and fetch the snippets
|
|
snippetCache.fetch(acc, queryhashes, urlmask, fetchcount);
|
|
} catch (IOException e) {
|
|
log.logSevere("presearch: failed", e);
|
|
} finally {
|
|
if (idx != null) try { idx.close(); } catch (Exception e){}
|
|
}
|
|
log.logFine("presearch: job terminated");
|
|
}
|
|
}
|
|
|
|
*/
|
|
|
|
//public serverObjects searchFromLocal(Set querywords, String order1, String order2, int count, boolean global, long time /*milliseconds*/, String urlmask) {
|
|
public serverObjects searchFromLocal(plasmaSearchQuery query) {
|
|
|
|
// tell all threads to do nothing for a specific time
|
|
wordIndex.intermission(2 * query.maximumTime);
|
|
intermissionAllThreads(2 * query.maximumTime);
|
|
|
|
serverObjects prop = new serverObjects();
|
|
try {
|
|
// filter out words that appear in bluelist
|
|
query.filterOut(blueList);
|
|
|
|
// log
|
|
log.logInfo("INIT WORD SEARCH: " + query.queryWords + ":" + query.queryHashes + " - " + query.wantedResults + " links, " + (query.maximumTime / 1000) + " seconds");
|
|
long timestamp = System.currentTimeMillis();
|
|
|
|
// start a presearch, which makes only sense if we idle afterwards.
|
|
// this is especially the case if we start a global search and idle until search
|
|
//if (query.domType == plasmaSearchQuery.SEARCHDOM_GLOBALDHT) {
|
|
// Thread preselect = new presearch(query.queryHashes, order, query.maximumTime / 10, query.urlMask, 10, 3);
|
|
// preselect.start();
|
|
//}
|
|
|
|
// create a new search event
|
|
plasmaSearchEvent theSearch = new plasmaSearchEvent(query, log, wordIndex, urlPool.loadedURL, snippetCache);
|
|
plasmaSearchResult acc = theSearch.search();
|
|
|
|
// fetch snippets
|
|
if (query.domType != plasmaSearchQuery.SEARCHDOM_GLOBALDHT)
|
|
snippetCache.fetch(acc.cloneSmart(), query.queryHashes, query.urlMask, 10);
|
|
log.logFine("SEARCH TIME AFTER ORDERING OF SEARCH RESULT: " + ((System.currentTimeMillis() - timestamp) / 1000) + " seconds");
|
|
|
|
// result is a List of urlEntry elements: prepare answer
|
|
if (acc == null) {
|
|
prop.put("totalcount", "0");
|
|
prop.put("orderedcount", "0");
|
|
prop.put("linkcount", "0");
|
|
} else {
|
|
prop.put("globalresults", acc.globalContributions);
|
|
prop.put("totalcount", acc.globalContributions + acc.localContributions);
|
|
prop.put("orderedcount", Integer.toString(acc.sizeOrdered()));
|
|
int i = 0;
|
|
int p;
|
|
URL url;
|
|
plasmaCrawlLURL.Entry urlentry;
|
|
String urlstring, urlname, filename, urlhash;
|
|
String host, hash, address, descr = "";
|
|
yacySeed seed;
|
|
plasmaSnippetCache.result snippet;
|
|
//kelondroMScoreCluster ref = new kelondroMScoreCluster();
|
|
while ((acc.hasMoreElements()) && (i < query.wantedResults)) {
|
|
urlentry = acc.nextElement();
|
|
url = urlentry.url();
|
|
urlhash = urlentry.hash();
|
|
host = url.getHost();
|
|
if (host.endsWith(".yacyh")) {
|
|
// translate host into current IP
|
|
p = host.indexOf(".");
|
|
hash = yacySeed.hexHash2b64Hash(host.substring(p + 1, host.length() - 6));
|
|
seed = yacyCore.seedDB.getConnected(hash);
|
|
filename = url.getFile();
|
|
if ((seed == null) || ((address = seed.getAddress()) == null)) {
|
|
// seed is not known from here
|
|
removeReferences(urlentry.hash(), plasmaCondenser.getWords(("yacyshare " + filename.replace('?', ' ') + " " + urlentry.descr()).getBytes()));
|
|
urlPool.loadedURL.remove(urlentry.hash()); // clean up
|
|
continue; // next result
|
|
}
|
|
url = new URL("http://" + address + "/" + host.substring(0, p) + filename);
|
|
urlname = "http://share." + seed.getName() + ".yacy" + filename;
|
|
if ((p = urlname.indexOf("?")) > 0) urlname = urlname.substring(0, p);
|
|
urlstring = htmlFilterContentScraper.urlNormalform(url);
|
|
} else {
|
|
urlstring = htmlFilterContentScraper.urlNormalform(url);
|
|
urlname = urlstring;
|
|
}
|
|
descr = urlentry.descr();
|
|
|
|
// check bluelist again: filter out all links where any bluelisted word
|
|
// appear either in url, url's description or search word
|
|
// the search word was sorted out earlier
|
|
/*
|
|
String s = descr.toLowerCase() + url.toString().toLowerCase();
|
|
for (int c = 0; c < blueList.length; c++) {
|
|
if (s.indexOf(blueList[c]) >= 0) return;
|
|
}
|
|
*/
|
|
//addScoreForked(ref, gs, descr.split(" "));
|
|
//addScoreForked(ref, gs, urlstring.split("/"));
|
|
if (urlstring.matches(query.urlMask)) { //.* is default
|
|
snippet = snippetCache.retrieve(url, query.queryHashes, false, 260);
|
|
if (snippet.source == plasmaSnippetCache.ERROR_NO_MATCH) {
|
|
// suppress line: there is no match in that resource
|
|
} else {
|
|
prop.put("results_" + i + "_description", descr);
|
|
prop.put("results_" + i + "_url", urlstring);
|
|
prop.put("results_" + i + "_urlhash", urlhash);
|
|
prop.put("results_" + i + "_urlname", urlname);
|
|
prop.put("results_" + i + "_date", dateString(urlentry.moddate()));
|
|
prop.put("results_" + i + "_size", Long.toString(urlentry.size()));
|
|
if (snippet.line == null) {
|
|
prop.put("results_" + i + "_snippet", 0);
|
|
prop.put("results_" + i + "_snippet_text", "");
|
|
} else {
|
|
prop.put("results_" + i + "_snippet", 1);
|
|
prop.put("results_" + i + "_snippet_text", snippet.line);
|
|
}
|
|
i++;
|
|
}
|
|
}
|
|
}
|
|
log.logFine("SEARCH TIME AFTER RESULT PREPARATION: " + ((System.currentTimeMillis() - timestamp) / 1000) + " seconds");
|
|
|
|
// calc some more cross-reference
|
|
long remainingTime = query.maximumTime - (System.currentTimeMillis() - timestamp);
|
|
if (remainingTime < 0) remainingTime = 1000;
|
|
/*
|
|
while ((acc.hasMoreElements()) && (((time + timestamp) < System.currentTimeMillis()))) {
|
|
urlentry = acc.nextElement();
|
|
urlstring = htmlFilterContentScraper.urlNormalform(urlentry.url());
|
|
descr = urlentry.descr();
|
|
|
|
addScoreForked(ref, gs, descr.split(" "));
|
|
addScoreForked(ref, gs, urlstring.split("/"));
|
|
}
|
|
**/
|
|
//Object[] ws = ref.getScores(16, false, 2, Integer.MAX_VALUE);
|
|
Object[] ws = acc.getReferences(16);
|
|
log.logFine("SEARCH TIME AFTER XREF PREPARATION: " + ((System.currentTimeMillis() - timestamp) / 1000) + " seconds");
|
|
|
|
/*
|
|
System.out.print("DEBUG WORD-SCORE: ");
|
|
for (int ii = 0; ii < ws.length; ii++) System.out.print(ws[ii] + ", ");
|
|
System.out.println(" all words = " + ref.getElementCount() + ", total count = " + ref.getTotalCount());
|
|
*/
|
|
prop.put("references", ws);
|
|
prop.put("linkcount", Integer.toString(i));
|
|
prop.put("results", Integer.toString(i));
|
|
}
|
|
|
|
// log
|
|
log.logInfo("EXIT WORD SEARCH: " + query.queryWords + " - " +
|
|
prop.get("totalcount", "0") + " links found, " +
|
|
prop.get("orderedcount", "0") + " links ordered, " +
|
|
prop.get("linkcount", "?") + " links selected, " +
|
|
((System.currentTimeMillis() - timestamp) / 1000) + " seconds");
|
|
return prop;
|
|
} catch (IOException e) {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
public serverObjects searchFromRemote(plasmaSearchQuery query) {
|
|
|
|
// tell all threads to do nothing for a specific time
|
|
wordIndex.intermission(2 * query.maximumTime);
|
|
intermissionAllThreads(2 * query.maximumTime);
|
|
|
|
serverObjects prop = new serverObjects();
|
|
try {
|
|
log.logInfo("INIT HASH SEARCH: " + query.queryHashes + " - " + query.wantedResults + " links");
|
|
long timestamp = System.currentTimeMillis();
|
|
plasmaSearchEvent theSearch = new plasmaSearchEvent(query, log, wordIndex, urlPool.loadedURL, snippetCache);
|
|
int idxc = theSearch.localSearch(query.maximumTime * 8 / 10);
|
|
long remainingTime = query.maximumTime - (System.currentTimeMillis() - timestamp);
|
|
if (remainingTime < 500) remainingTime = 500;
|
|
plasmaSearchResult acc = theSearch.order(remainingTime, 10);
|
|
|
|
// result is a List of urlEntry elements
|
|
if (acc == null) {
|
|
prop.put("totalcount", "0");
|
|
prop.put("linkcount", "0");
|
|
prop.put("references", "");
|
|
} else {
|
|
prop.put("totalcount", Integer.toString(acc.sizeOrdered()));
|
|
int i = 0;
|
|
StringBuffer links = new StringBuffer();
|
|
String resource = "";
|
|
//plasmaIndexEntry pie;
|
|
plasmaCrawlLURL.Entry urlentry;
|
|
plasmaSnippetCache.result snippet;
|
|
while ((acc.hasMoreElements()) && (i < query.wantedResults)) {
|
|
urlentry = acc.nextElement();
|
|
snippet = snippetCache.retrieve(urlentry.url(), query.queryHashes, false, 260);
|
|
if (snippet.source == plasmaSnippetCache.ERROR_NO_MATCH) {
|
|
// suppress line: there is no match in that resource
|
|
} else {
|
|
if (snippet.line == null) {
|
|
resource = urlentry.toString();
|
|
} else {
|
|
resource = urlentry.toString(snippet.line);
|
|
}
|
|
if (resource != null) {
|
|
links.append("resource").append(i).append("=").append(resource).append(serverCore.crlfString);
|
|
i++;
|
|
}
|
|
}
|
|
}
|
|
prop.put("links", links.toString());
|
|
prop.put("linkcount", Integer.toString(i));
|
|
|
|
// prepare reference hints
|
|
Object[] ws = acc.getReferences(16);
|
|
StringBuffer refstr = new StringBuffer();
|
|
for (int j = 0; j < ws.length; j++) refstr.append(",").append((String) ws[j]);
|
|
prop.put("references", (refstr.length() > 0)?refstr.substring(1):refstr.toString());
|
|
}
|
|
|
|
// add information about forward peers
|
|
prop.put("fwhop", ""); // hops (depth) of forwards that had been performed to construct this result
|
|
prop.put("fwsrc", ""); // peers that helped to construct this result
|
|
prop.put("fwrec", ""); // peers that would have helped to construct this result (recommendations)
|
|
|
|
// log
|
|
log.logInfo("EXIT HASH SEARCH: " + query.queryHashes + " - " + idxc + " links found, " +
|
|
prop.get("linkcount", "?") + " links selected, " +
|
|
((System.currentTimeMillis() - timestamp) / 1000) + " seconds");
|
|
return prop;
|
|
} catch (IOException e) {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
|
|
public serverObjects action(String actionName, serverObjects actionInput) {
|
|
// perform an action. (not used)
|
|
|
|
return null;
|
|
}
|
|
|
|
|
|
public String toString() {
|
|
// it is possible to use this method in the cgi pages.
|
|
// actually it is used there for testing purpose
|
|
return "PROPS: " + super.toString() + "; QUEUE: " + sbQueue.toString();
|
|
}
|
|
|
|
// method for index deletion
|
|
public int removeAllUrlReferences(URL url, boolean fetchOnline) {
|
|
return removeAllUrlReferences(plasmaURL.urlHash(url), fetchOnline);
|
|
}
|
|
|
|
public int removeAllUrlReferences(String urlhash, boolean fetchOnline) {
|
|
// find all the words in a specific resource and remove the url reference from every word index
|
|
// finally, delete the url entry
|
|
|
|
// determine the url string
|
|
plasmaCrawlLURL.Entry entry = urlPool.loadedURL.getEntry(urlhash);
|
|
URL url = entry.url();
|
|
if (url == null) return 0;
|
|
// get set of words
|
|
//Set words = plasmaCondenser.getWords(getText(getResource(url, fetchOnline)));
|
|
Set words = plasmaCondenser.getWords(snippetCache.parseDocument(url, snippetCache.getResource(url, fetchOnline)).getText());
|
|
// delete all word references
|
|
int count = removeReferences(urlhash, words);
|
|
// finally delete the url entry itself
|
|
urlPool.loadedURL.remove(urlhash);
|
|
return count;
|
|
}
|
|
|
|
public int removeReferences(URL url, Set words) {
|
|
return removeReferences(plasmaURL.urlHash(url), words);
|
|
}
|
|
|
|
public int removeReferences(final String urlhash, final Set words) {
|
|
// sequentially delete all word references
|
|
// returns number of deletions
|
|
Iterator iter = words.iterator();
|
|
String word;
|
|
final String[] urlEntries = new String[] {urlhash};
|
|
int count = 0;
|
|
while (iter.hasNext()) {
|
|
word = (String) iter.next();
|
|
// delete the URL reference in this word index
|
|
count += wordIndex.removeEntries(plasmaWordIndexEntry.word2hash(word), urlEntries, true);
|
|
}
|
|
return count;
|
|
}
|
|
|
|
public int adminAuthenticated(httpHeader header) {
|
|
String adminAccountBase64MD5 = getConfig("adminAccountBase64MD5", "");
|
|
if (adminAccountBase64MD5.length() == 0) return 2; // not necessary
|
|
String authorization = ((String) header.get(httpHeader.AUTHORIZATION, "xxxxxx")).trim().substring(6);
|
|
if (authorization.length() == 0) return 1; // no authentication information given
|
|
if ((((String) header.get("CLIENTIP", "")).equals("localhost")) && (adminAccountBase64MD5.equals(authorization))) return 3; // soft-authenticated for localhost
|
|
if (adminAccountBase64MD5.equals(serverCodings.standardCoder.encodeMD5Hex(authorization))) return 4; // hard-authenticated, all ok
|
|
return 0; // wrong password
|
|
}
|
|
|
|
public void terminate() {
|
|
this.terminate = true;
|
|
this.shutdownSync.V();
|
|
}
|
|
|
|
public boolean isTerminated() {
|
|
return this.terminate;
|
|
}
|
|
|
|
public boolean waitForShutdown() throws InterruptedException {
|
|
this.shutdownSync.P();
|
|
return this.terminate;
|
|
}
|
|
}
|