You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
yacy_search_server/source/net/yacy/htroot/News.java

211 lines
10 KiB

// News.java
// -----------------------
// part of YaCy
// (C) by Michael Peter Christen; mc@yacy.net
// first published on http://www.anomic.de
// Frankfurt, Germany, 2005
// last major change: 29.07.2005
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program; if not, write to the Free Software
// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
// You must compile this file with
// javac -classpath .:../classes Network.java
// if the shell's current path is HTROOT
package net.yacy.htroot;
import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import net.yacy.cora.date.GenericFormatter;
import net.yacy.cora.protocol.HeaderFramework;
import net.yacy.cora.protocol.RequestHeader;
import net.yacy.cora.util.ConcurrentLog;
import net.yacy.http.servlets.YaCyDefaultServlet;
import net.yacy.peers.NewsDB;
import net.yacy.peers.NewsPool;
import net.yacy.peers.Seed;
import net.yacy.search.Switchboard;
12 years ago
import net.yacy.server.serverObjects;
import net.yacy.server.serverSwitch;
public class News {
public static serverObjects respond(final RequestHeader header, final serverObjects post, final serverSwitch env) {
final Switchboard sb = (Switchboard) env;
final serverObjects prop = new serverObjects();
final boolean overview = (post == null) || "0".equals(post.get("page", "0"));
final int tableID = (overview) ? -1 : (post == null ? 0 : post.getInt("page", 0)) - 1;
// execute commands
if (post != null) {
if ((post.containsKey("deletespecific")) && (tableID >= 0)) {
major step forward to network switching (target is easy switch to intranet or other networks .. and back) This change is inspired by the need to see a network connected to the index it creates in a indexing team. It is not possible to divide the network and the index. Therefore all control files for the network was moved to the network within the INDEX/<network-name> subfolder. The remaining YACYDB is superfluous and can be deleted. The yacyDB and yacyNews data structures are now part of plasmaWordIndex. Therefore all methods, using static access to yacySeedDB had to be rewritten. A special problem had been all the port forwarding methods which had been tightly mixed with seed construction. It was not possible to move the port forwarding functions to the place, meaning and usage of plasmaWordIndex. Therefore the port forwarding had been deleted (I guess nobody used it and it can be simulated by methods outside of YaCy). The mySeed.txt is automatically moved to the current network position. A new effect causes that every network will create a different local seed file, which is ok, since the seed identifies the peer only against the network (it is the purpose of the seed hash to give a peer a location within the DHT). No other functional change has been made. The next steps to enable network switcing are: - shift of crawler tables from PLASMADB into the network (crawls are also network-specific) - possibly shift of plasmaWordIndex code into yacy package (index management is network-specific) - servlet to switch networks git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4765 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
if (sb.adminAuthenticated(header) < 2) {
prop.authenticationRequired();
return prop; // this button needs authentication, force log-in
}
final Iterator<String> e = post.keySet().iterator();
String check;
String id;
while (e.hasNext()) {
check = e.next();
if ((check.startsWith("del_")) && "on".equals(post.get(check, "off"))) {
id = check.substring(4);
try {
sb.peers.newsPool.moveOff(tableID, id);
} catch (final Exception ee) {ConcurrentLog.logException(ee);}
}
}
}
if ((post.containsKey("deleteall")) && (tableID >= 0)) {
major step forward to network switching (target is easy switch to intranet or other networks .. and back) This change is inspired by the need to see a network connected to the index it creates in a indexing team. It is not possible to divide the network and the index. Therefore all control files for the network was moved to the network within the INDEX/<network-name> subfolder. The remaining YACYDB is superfluous and can be deleted. The yacyDB and yacyNews data structures are now part of plasmaWordIndex. Therefore all methods, using static access to yacySeedDB had to be rewritten. A special problem had been all the port forwarding methods which had been tightly mixed with seed construction. It was not possible to move the port forwarding functions to the place, meaning and usage of plasmaWordIndex. Therefore the port forwarding had been deleted (I guess nobody used it and it can be simulated by methods outside of YaCy). The mySeed.txt is automatically moved to the current network position. A new effect causes that every network will create a different local seed file, which is ok, since the seed identifies the peer only against the network (it is the purpose of the seed hash to give a peer a location within the DHT). No other functional change has been made. The next steps to enable network switcing are: - shift of crawler tables from PLASMADB into the network (crawls are also network-specific) - possibly shift of plasmaWordIndex code into yacy package (index management is network-specific) - servlet to switch networks git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4765 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
if (sb.adminAuthenticated(header) < 2) {
prop.authenticationRequired();
return prop; // this button needs authentication, force log-in
}
try {
if ((tableID == NewsPool.PROCESSED_DB) || (tableID == NewsPool.PUBLISHED_DB)) {
sb.peers.newsPool.clear(tableID);
} else {
sb.peers.newsPool.moveOffAll(tableID);
}
} catch (final Exception e) {
ConcurrentLog.logException(e);
}
}
}
// generate properties for output
if (overview) {
// show overview
* Complete number localization and provide a more reasonable interface to serverObjects: - put(key, value) methods are now used if a value added to the map should be kept as it is. Numbers are transformed (but not formatted) to an equivalent String representation. - putASIS(...) have been removed, now done with simple put(...) (see above). - puNum(...) can be used for number values which should be stored in a formatted way, either depending on the current locale setting for yacy (default) or in a "none" locale (see javadocs and setLocalize()). - putHTML(...) escapes special characters into corresponding HTML enities ('<' => '&lt;') which was done with put(...) before and so was called too often, becauses it is necessary only for very few cases. Additionally there is a "forXML" mode which only replaces < > & ". In short: Use put(...) for almost everything, use putXY(...) if you need some special transformation of the value. A few bugs have been fixed as well, and there should be a small performance improvement for complex pages with a lot of values. * added additional Sum/Avg rows to access tracker pages, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=456 * removed duplicate code (mostly related to the big changes above). TODO: - make sure, number formats work as expected _everywhere_, report overseen stuff http://forum.yacy-websuche.de/viewtopic.php?f=5&t=437 - probably a good idea to add special putDate() methods as they are used in many pages and create duplicated formatting code + maybe some centralized handling for memory value formatting. - further improve the speed of page creation for the WatchCrawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4178 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
prop.put("table", "0");
prop.put("page", "0");
prop.putNum("table_insize", sb.peers.newsPool.size(NewsPool.INCOMING_DB));
prop.putNum("table_prsize", sb.peers.newsPool.size(NewsPool.PROCESSED_DB));
prop.putNum("table_ousize", sb.peers.newsPool.size(NewsPool.OUTGOING_DB));
prop.putNum("table_pusize", sb.peers.newsPool.size(NewsPool.PUBLISHED_DB));
} else {
// generate table
* Complete number localization and provide a more reasonable interface to serverObjects: - put(key, value) methods are now used if a value added to the map should be kept as it is. Numbers are transformed (but not formatted) to an equivalent String representation. - putASIS(...) have been removed, now done with simple put(...) (see above). - puNum(...) can be used for number values which should be stored in a formatted way, either depending on the current locale setting for yacy (default) or in a "none" locale (see javadocs and setLocalize()). - putHTML(...) escapes special characters into corresponding HTML enities ('<' => '&lt;') which was done with put(...) before and so was called too often, becauses it is necessary only for very few cases. Additionally there is a "forXML" mode which only replaces < > & ". In short: Use put(...) for almost everything, use putXY(...) if you need some special transformation of the value. A few bugs have been fixed as well, and there should be a small performance improvement for complex pages with a lot of values. * added additional Sum/Avg rows to access tracker pages, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=456 * removed duplicate code (mostly related to the big changes above). TODO: - make sure, number formats work as expected _everywhere_, report overseen stuff http://forum.yacy-websuche.de/viewtopic.php?f=5&t=437 - probably a good idea to add special putDate() methods as they are used in many pages and create duplicated formatting code + maybe some centralized handling for memory value formatting. - further improve the speed of page creation for the WatchCrawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4178 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
prop.put("table", "1");
prop.put("page", tableID + 1);
prop.put("table_page", tableID + 1);
if (sb.peers != null) {
final int maxCount = Math.min(1000, sb.peers.newsPool.size(tableID));
final Iterator<NewsDB.Record> recordIterator = sb.peers.newsPool.recordIterator(tableID);
NewsDB.Record record;
Seed seed;
int i = 0;
while ((recordIterator.hasNext()) && (i < maxCount)) {
record = recordIterator.next();
if (record == null) continue;
seed = sb.peers.getConnected(record.originator());
if (seed == null) seed = sb.peers.getDisconnected(record.originator());
final String category = record.category();
prop.put("table_list_" + i + "_id", record.id());
* Complete number localization and provide a more reasonable interface to serverObjects: - put(key, value) methods are now used if a value added to the map should be kept as it is. Numbers are transformed (but not formatted) to an equivalent String representation. - putASIS(...) have been removed, now done with simple put(...) (see above). - puNum(...) can be used for number values which should be stored in a formatted way, either depending on the current locale setting for yacy (default) or in a "none" locale (see javadocs and setLocalize()). - putHTML(...) escapes special characters into corresponding HTML enities ('<' => '&lt;') which was done with put(...) before and so was called too often, becauses it is necessary only for very few cases. Additionally there is a "forXML" mode which only replaces < > & ". In short: Use put(...) for almost everything, use putXY(...) if you need some special transformation of the value. A few bugs have been fixed as well, and there should be a small performance improvement for complex pages with a lot of values. * added additional Sum/Avg rows to access tracker pages, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=456 * removed duplicate code (mostly related to the big changes above). TODO: - make sure, number formats work as expected _everywhere_, report overseen stuff http://forum.yacy-websuche.de/viewtopic.php?f=5&t=437 - probably a good idea to add special putDate() methods as they are used in many pages and create duplicated formatting code + maybe some centralized handling for memory value formatting. - further improve the speed of page creation for the WatchCrawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4178 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
prop.putHTML("table_list_" + i + "_ori", (seed == null) ? record.originator() : seed.getName());
prop.put("table_list_" + i + "_cre", GenericFormatter.SHORT_SECOND_FORMATTER.format(record.created()));
prop.put("table_list_" + i + "_crerfcdate", HeaderFramework.formatRFC1123(record.created()));
prop.putHTML("table_list_" + i + "_cat", category);
prop.put("table_list_" + i + "_rec", (record.received() == null) ? "-" : GenericFormatter.SHORT_SECOND_FORMATTER.format(record.received()));
prop.put("table_list_" + i + "_dis", record.distributed());
final Map<String, String> attributeMap = record.attributes();
* Complete number localization and provide a more reasonable interface to serverObjects: - put(key, value) methods are now used if a value added to the map should be kept as it is. Numbers are transformed (but not formatted) to an equivalent String representation. - putASIS(...) have been removed, now done with simple put(...) (see above). - puNum(...) can be used for number values which should be stored in a formatted way, either depending on the current locale setting for yacy (default) or in a "none" locale (see javadocs and setLocalize()). - putHTML(...) escapes special characters into corresponding HTML enities ('<' => '&lt;') which was done with put(...) before and so was called too often, becauses it is necessary only for very few cases. Additionally there is a "forXML" mode which only replaces < > & ". In short: Use put(...) for almost everything, use putXY(...) if you need some special transformation of the value. A few bugs have been fixed as well, and there should be a small performance improvement for complex pages with a lot of values. * added additional Sum/Avg rows to access tracker pages, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=456 * removed duplicate code (mostly related to the big changes above). TODO: - make sure, number formats work as expected _everywhere_, report overseen stuff http://forum.yacy-websuche.de/viewtopic.php?f=5&t=437 - probably a good idea to add special putDate() methods as they are used in many pages and create duplicated formatting code + maybe some centralized handling for memory value formatting. - further improve the speed of page creation for the WatchCrawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4178 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
prop.putHTML("table_list_" + i + "_att", attributeMap.toString());
int j = 0;
if (!attributeMap.isEmpty()) {
for (final Entry<String, String> attribute: attributeMap.entrySet()) {
prop.put("table_list_" + i + "_attributes_" + j + "_name", attribute.getKey());
prop.putHTML("table_list_" + i + "_attributes_" + j + "_value", attribute.getValue());
j++;
}
}
* Complete number localization and provide a more reasonable interface to serverObjects: - put(key, value) methods are now used if a value added to the map should be kept as it is. Numbers are transformed (but not formatted) to an equivalent String representation. - putASIS(...) have been removed, now done with simple put(...) (see above). - puNum(...) can be used for number values which should be stored in a formatted way, either depending on the current locale setting for yacy (default) or in a "none" locale (see javadocs and setLocalize()). - putHTML(...) escapes special characters into corresponding HTML enities ('<' => '&lt;') which was done with put(...) before and so was called too often, becauses it is necessary only for very few cases. Additionally there is a "forXML" mode which only replaces < > & ". In short: Use put(...) for almost everything, use putXY(...) if you need some special transformation of the value. A few bugs have been fixed as well, and there should be a small performance improvement for complex pages with a lot of values. * added additional Sum/Avg rows to access tracker pages, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=456 * removed duplicate code (mostly related to the big changes above). TODO: - make sure, number formats work as expected _everywhere_, report overseen stuff http://forum.yacy-websuche.de/viewtopic.php?f=5&t=437 - probably a good idea to add special putDate() methods as they are used in many pages and create duplicated formatting code + maybe some centralized handling for memory value formatting. - further improve the speed of page creation for the WatchCrawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4178 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
prop.put("table_list_" + i + "_attributes", j);
// generating link / title / description (taken over from Surftips.java)
String link, title, description;
if (category.equals(NewsPool.CATEGORY_CRAWL_START)) {
link = record.attribute("startURL", "");
title = (record.attribute("intention", "").isEmpty()) ? link : record.attribute("intention", "");
description = "Crawl Start Point";
} else if (category.equals(NewsPool.CATEGORY_PROFILE_UPDATE)) {
link = record.attribute("homepage", "");
title = "Home Page of " + record.attribute("nickname", "");
description = "Profile Update";
} else if (category.equals(NewsPool.CATEGORY_BOOKMARK_ADD)) {
link = record.attribute("url", "");
title = record.attribute("title", "");
description = "Bookmark: " + record.attribute("description", "");
} else if (category.equals(NewsPool.CATEGORY_SURFTIPP_ADD)) {
link = record.attribute("url", "");
title = record.attribute("title", "");
description = "Surf Tipp: " + record.attribute("description", "");
} else if (category.equals(NewsPool.CATEGORY_SURFTIPP_VOTE_ADD)) {
link = record.attribute("url", "");
title = record.attribute("title", "");
description = record.attribute("url", "");
} else if (category.equals(NewsPool.CATEGORY_WIKI_UPDATE)) {
if(seed == null) {
link = "";
} else {
final Set<String> ips = seed.getIPs();
if(!ips.isEmpty()) {
link = "http://" + seed.getPublicAddress(ips.iterator().next()) + "/Wiki.html?page=" + record.attribute("page", "");
} else {
link = "";
}
}
title = record.attribute("author", "Anonymous") + ": " + record.attribute("page", "");
description = "Wiki Update: " + record.attribute("description", "");
} else if (category.equals(NewsPool.CATEGORY_BLOG_ADD)) {
if(seed == null) {
link = "";
} else {
final Set<String> ips = seed.getIPs();
if(!ips.isEmpty()) {
link = "http://" + seed.getPublicAddress(ips.iterator().next()) + "/Blog.html?page=" + record.attribute("page", "");
} else {
link = "";
}
}
title = record.attribute("author", "Anonymous") + ": " + record.attribute("page", "");
description = "Blog Entry: " + record.attribute("subject", "");
} else {
link = "";
title = "";
description = "";
}
prop.putHTML("table_list_" + i + "_link", link);
* Complete number localization and provide a more reasonable interface to serverObjects: - put(key, value) methods are now used if a value added to the map should be kept as it is. Numbers are transformed (but not formatted) to an equivalent String representation. - putASIS(...) have been removed, now done with simple put(...) (see above). - puNum(...) can be used for number values which should be stored in a formatted way, either depending on the current locale setting for yacy (default) or in a "none" locale (see javadocs and setLocalize()). - putHTML(...) escapes special characters into corresponding HTML enities ('<' => '&lt;') which was done with put(...) before and so was called too often, becauses it is necessary only for very few cases. Additionally there is a "forXML" mode which only replaces < > & ". In short: Use put(...) for almost everything, use putXY(...) if you need some special transformation of the value. A few bugs have been fixed as well, and there should be a small performance improvement for complex pages with a lot of values. * added additional Sum/Avg rows to access tracker pages, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=456 * removed duplicate code (mostly related to the big changes above). TODO: - make sure, number formats work as expected _everywhere_, report overseen stuff http://forum.yacy-websuche.de/viewtopic.php?f=5&t=437 - probably a good idea to add special putDate() methods as they are used in many pages and create duplicated formatting code + maybe some centralized handling for memory value formatting. - further improve the speed of page creation for the WatchCrawler. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4178 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
prop.putHTML("table_list_" + i + "_title", title);
prop.putHTML("table_list_" + i + "_description", description);
i++;
}
prop.put("table_list", i);
}
}
// add the peer web context (used by News.rss)
prop.put("context", YaCyDefaultServlet.getContext(header, sb));
// return rewrite properties
return prop;
}
}