f1ori
dca9e16f51
* don't index pages, which redirect, twice
...
* there fore auto-redirection of HTTPClient for crawling is disabled and the old code is reactivated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7332 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
eb79b952ef
*) cleaner code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7331 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
38fdf43587
*) renamed classes according to standard Java coding conventions
...
*) String.isEmpty() was introduced in Java 1.6, but we still use Java 1.5
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7330 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
025e3f4790
*) renamed classes according to standard Java coding conventions
...
*) removed unsused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7328 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
3b9aa0504e
*) removed unsused code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7327 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
db3db0fdb9
*) trying to make this class less confusing (probably failing)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7326 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
54e63b556e
intermediate step for a YMark auto-tagging function based on word frequencies.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7325 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
403ee9c014
added a drill-down for metadata and word count to /api/ymarks/test_treeview.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7324 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
a025b1da89
* fix bug when browsing local filesystem (e. g. repository) with yacy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7323 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
11ae5b108e
enabled rebuildIndex for /Table_YMark_p.html (rebuilds the tags and folders index)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7320 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
b87bf88ac8
using less memory on merging and rewriting blobs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7317 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
94a9be18a4
added a ymark table administration: /Table_YMark_p.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7316 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
25339f93c7
more updates to ymarks
...
- working xbel import/export
- exported xbel includes yacy specific metadata but still validates against PUBLIC DTD
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7315 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
d62e449a11
* fix FilterEngine, forgot comparision-operator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7314 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
cdd65aca71
update to ymarks
...
- get_xbel.xml is almost working
- startet ymark api documentation info.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7313 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
808edffaf6
ymarks
...
- some refactoring
- working xbel and html import (/api/ymarks/test_import.html)
- working treeview (/api/ymarks/test_treeview.html)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7312 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
2c539b514a
* add domaincheck (local/global/domainlist) to urlcleaner
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7311 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
117fc86b3d
fix for http://forum.yacy-websuche.de/viewtopic.php?p=21199#p21199
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7308 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
441fbc26e2
security patch for WeakPriorityBlockingQueue (produced a deadlock)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7307 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5dcb838293
- removed thread overhead when calling dns services
...
- fixed localsearch (changed it by accident)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7306 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4c50d3428e
smaller file size for array stacks to support smaller deletion sizes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7305 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
09badc697b
- low-memory patch for crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7304 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
becc463d8a
enhanced did-you-mean
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7300 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
43586a2ace
a update to ymarks (please test if you wish):
...
- import HTML (e.g. FF export) via /api/ymarks/import.html
- view your import via /api/ymarks/test.html
- get a xml list via /api/ymarks/get_ymark_list.xml?tags=&folders=
- delete bookmark tables via standard interface /Tables_p.html
it is still very experimental!!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7299 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
93c535d111
fixed http://forum.yacy-websuche.de/viewtopic.php?p=21113#p21113
...
fixed a concurrent modification exception during search and a time-out problem
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7298 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
04932dc268
added rdf data structure for rss feeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7297 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
84f2953cd8
fix for rss loader / rss type recognition
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7296 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4c72885cba
added a sitemap entry parser and loader for sitemaps
...
(a recursion if a sitemap refers to another sitemap)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7295 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
790e0b1894
- enhanced index deletion in IndexControlRWIs_p: delete also robots.txt database and cache if demanded
...
- added option for details of deletion
- added deletion to new ConfigHTCache_p servlet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7294 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
f5324b27f2
more updates to the new bookmarks (ymarks)....
...
- split YMarkTables and YMarkIndex in two different classes
- HTML import is working properly
- XBEL import is still broken
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7292 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
445619f3ec
added a submenu ConfigHTCache_p.html to set the size of the HTCache separately from the proxy configuration.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7291 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
85c65475fa
smal but important correction of last commit @ HTTPClient
...
(if there is a response it really should be taken to its end)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7290 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
acd93b1b31
* add failsafe mechanisme to domainlist retrieval
...
domainlist is saved locally, if none of the given urls in network.unit.domainlist
could be retrieved, the file from the last boot is used instead
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7289 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
70c95608d4
Added CORS Access header for yacysearch.rss output
...
used some of the recommendations from Copro:
http://forum.yacy-websuche.de/viewtopic.php?p=21015#p21015
Original Request:
http://forum.yacy-websuche.de/viewtopic.php?p=20829#p20829
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7288 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
18729351e7
upnp: hint for wrongly detected local ip address
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7286 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
def4253555
* add option to network definition to provide a domainlist (syntax like in blacklists)
...
* crawler and search allow only urls matching one in domainlist (if list is provided)
* this may be useful to prevent dedicated networks from being "polluted"
* FilterEngine is improved Backlist-object, Blacklist may inherit from FilterEngine in the future
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7285 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ac6b503adf
untar files without gzip decompression even if the file has gz extension. this is done when the decompression fails.
...
decompressed gzip files with gz extension may appear if the server sets a gzip compression header
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7282 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
efe0667fdd
more new bookmark (ymarks) code with experimental html and xbel import
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7281 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
mikeworks
caabebf9be
Fixed spelling mistake omiting -> omitting in debug messages in ConfigUpdate_p.java and Switchboard.java
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7280 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
fb92f9ae8e
added mime type image/jpeg (image/jpg is wrong but it is left here because it does not harm and this error also exists in configuration of web servers)
...
see also:
http://forum.yacy-websuche.de/viewtopic.php?p=21129#p21129
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7279 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
155d556568
- better memory protection
...
- more logging
- little bit of refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7278 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
7d8de34778
* add a bit documentation to DigestURI, use DigestURI(string) instead of DigestURI(string, null)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7276 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
25a8e55bc9
more logging about bad seeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7275 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
959b8c6fa0
- allow greater seed size
...
- more logging for bad seeds
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7274 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e103419a56
- removed <3 peers barrier for peer ping feedback
...
- more logging
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7273 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
d0e6c03b51
some updates to the new bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7272 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
facfd204e9
added a parent configuration option.
...
see /ConfigPortal.html
requested here:
http://forum.yacy-websuche.de/viewtopic.php?p=21099#p21099
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7271 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e3964f2c31
better catch of network definition load error; continue with secondary network load definition location
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7270 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
65a0381f76
*) cleaning up code (still not done)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7267 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e3e3b49d52
- enhanced main release recognition
...
- yacybot user agent now includes the yacy network name (not the peer name!)
- refactoring and clean-up (mostly turned tab into spaces)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7266 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
9c94ebdee4
small changes to new bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7265 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
244b56e9d3
an update to the new bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7264 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
dc40f51b8d
*) added headlines as proposed by Vega
...
*) <pre> will be displayed monospaced in wiki and blog again
*) bugfix for <pre> spanning multiple lines
*) replaced deprecated <s> tag with <span> equivalent
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7262 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
f035f257da
added some more bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7261 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
22ed9c380c
*) fixed bug which was introduced in r7226 (shame on me) which made wiki unusable (all entries were stored with empty subject as key -> edits were lost)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7260 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
60fd2e549d
* log failures when writing config file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7259 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
58e74282af
added a word counter statistic in condenser which is used by the did-you-mean to calculate best matches for given search words.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7258 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
863065abc4
added user agent logging to access tracker
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7256 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
a79728b97d
some updates to experimental bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7254 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
ef782cd026
and even more experimental bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7253 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ed4371dcf3
enhanced navigation implementation and enhanced tag cloud computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7252 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ca738ac924
- added a tag cloud to search results (using the topics)
...
- some refactoring of score classes
- added default package for new classes add_ymark and delete_ymark
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7251 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
7aca763ca8
Some more experimental bookmark code...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7250 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
4270ed696c
Experimental code (I need to transfer the code to my macbook, sorry) for the new bookmarks API based on the Tables concept (same as for crawl starts). Currently you can add a bookmark by api/ymarks/add_ymark.xml?url= http://www.yacy.net&title=YaCy and watch the result via the standard view Tables_p.html.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7249 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e4d561971e
added more score cluster options and made score cluster usage more transparent
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7248 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e8f90201a5
fix for scheduling of rss feeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7247 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7cd9d9d22a
- enhanced DidYouMean computation using a faster count on index entries; this causes that results can be ranked better
...
- added limitations on DidYouMean result sets according to input and output string length
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7246 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
de722090b5
enhancements in did-you-mean guessing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7243 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a59c885ee0
autocomplete and did-you-mean can now understand _all_ languages and can generate suggestions in all languages and character types
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7242 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b7acd92ce4
Auto-Suggestions for YaCy Search:
...
- added a suggest servlet according to opensearch and firefox standard
- integrated the suggest servlet into opensearch description file
- integrated a autocomplete plugin for jquery
- added a autocomplete addition to the yacy search windows showing autosuggest queries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7241 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
24f1cba7b2
performance hacks:
...
- faster generation of index abstract compression during remote search
- less synchronization in IO record reading
- request index abstract generation only if necessary and faster time-out in remote search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7239 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6a166c2040
patches for bad proxy behaviour
...
- accept ipv6 localhost clients
- index media files (url only)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7238 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
d607b30b6a
performance enhancements for search and code review for database functions
...
- removed read cache from Records data structure because the read cache had no cache hit during search operation
- copied old read-cache class to CachedRecords and the old, now new Records class does not have the cache any more and a code review checked that data structures and synchronization is clean
- removed unnecessary synchronization from Table class during get()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7237 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
45b1ab3d07
custom + generic skins:
...
- added a generic skin which is filled with actual color assignment using a servlet
- enabled css servlets
- added a generic color scheme in configuration file
- added configuration input in Customization/Appearance servlet
- added a jquery color picker widget
- placed color picked widget to input field of generic colour definition input fields
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7235 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
fcd40cd30f
- disabled domZones (buggy, must think about better solution)
...
- increased time-out for dns resolver and isLocal property
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7233 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ec38eca278
fix for new URI equal method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7232 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0d363a94d7
more performance hacks
...
this makes YaCy search results VERY fast for all verify=false search cases
and it enhances the search speed also for all other snippet-fetch cases.
With this change my peer performed 100 Queries Per Second (!!!) while doing 10 queries simultanously (!!!)
in an intranet index of 20000 URLs on my 16-core Mac
Check this yourself by doing:
cd bin
./searchtestmulti.sh
after finishing the run, divide 1000 by the given time per query (which is the qps for one thread)
and then multiply again by 10 (because 10 search threads has been started)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7231 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b8aee6d402
performance hacks for better search performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7230 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
091dd3f6ec
- enhanced intranet search speed
...
- enhanced intranet portscan speed (better time-out)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7227 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
b9f405d1e8
*) added comments
...
*) more beautyful and easier to understand code (IMO)
*) added display= parameter to a lot of links in Wiki.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7226 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6e6994e328
latest bugfixes to search and indexing function after test of demo presentation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7223 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
aacf572a26
- enhancements for search speed
...
- bug fixes in many classes including basic data structure classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7217 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
61c82f3105
gzip-compresson @ transferRWI & transferURL back again
...
This reduce upload-volume to suit limited bandwidth of home-users like me :-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7215 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
2c549ae341
fixed a number of small bugs:
...
- better crawl star for files paths and smb paths
- added time-out wrapper for dns resolving and reverse resolving to prevent blockings
- fixed intranet scanner result list check boxes
- prevented htcache usage in case of file and smb crawling (not necessary, documents are locally available)
- fixed rss feed loader
- fixes sitemap loader which had not been restricted to single files (crawl-depth must be zero)
- clearing of crawl result lists when a network switch was done
- higher maximum file size for crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7214 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f6eebb6f99
replaced auto-dom filter with easy-to-understand Site Link-List crawler option
...
- nobody understand the auto-dom filter without a lenghtly introduction about the function of a crawler
- nobody ever used the auto-dom filter other than with a crawl depth of 1
- the auto-dom filter was buggy since the filter did not survive a restart and then a search index contained waste
- the function of the auto-dom filter was in fact to just load a link list from the given start url and then start separate crawls for all these urls restricted by their domain
- the new Site Link-List option shows the target urls in real-time during input of the start url (like the robots check) and gives a transparent feed-back what it does before it can be used
- the new option also fits into the easy site-crawl start menu
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7213 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3057a0b939
- intranet scanner now produces urls with host names, not ips if possible
...
- CrawStartIntranet servlet shows IPs and host names
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7210 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c60aed4435
no caching in browser of dynamic web pages sent by YaCy http
...
this may prevent unnecessary IO caused by cache storage of the browser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7207 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e63896f2a8
added an intranet scanner and a servlet which shows all intranet addresses and an option to start a site-crawl for all these addresses at once.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7203 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e54cb7fb0c
more bugfixes (also for latest commit)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7202 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
be6b48311c
misc bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7201 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
d2fd93135c
- moved yacybot user agent string definition to MultiProtocolURI since there are basic access mechanisms where the bot string is needed
...
- migrated the 'yacy' user agent to 'yacybot' in many client methods since the 'yacy' user agent is only used for the proxy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7199 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
afa708d552
*) added <s>...</s> tag to WikiCode -> works just as the HTML equivalent
...
*) code changes (PMD) without functional changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7193 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a83186ac7d
fix for bug in cytrails
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7192 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
48c0d508ac
fixes for crawling of smb links (file length not always available)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7190 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0bc6284e27
- added bugfix for access tracker in case of concurrency conflicts
...
- added missing entry for new icu4j path in Mac App
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7188 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
e670e1ef8e
add charset auto-detection for htmlParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7186 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
ddcd5ae78c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2989
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7185 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
8fe1102452
fix http://forum.yacy-websuche.de/viewtopic.php?p=20889#p18426
...
reuse code from htmlParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7184 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
10a9cb1971
simplified snippet computation process and separated the algorithm into two classes
...
also enhances selection criteria for best snippet line computation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7182 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
4450c240b7
npe fix http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2982
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7181 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago