f1ori
bd0318ba81
* YaCy only supports gzip-encoding, so remove any other encoding from request
...
* fixes http://www.yacy-forum.org/viewtopic.php?f=2&t=163
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5132 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bb5c898441
enhancements to localsearch behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
42e2d195ac
added hint from http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1294
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5130 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
39964e88fa
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1329#p9121
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5129 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3f3673b6e5
extended balancer:
...
- added automatic time delay in case that a large number of urls come from the same domain
- added additional time delay in case that an url is a dynamic (CGI) url. This shall cause less IO on targets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5128 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3c6e8d2015
set default ppm when network is switched
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3288c19c1a
reduce remote crawl PPM for fresh peers in freeworld to 6 PPM
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5124 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
5ce9a100bb
fix(2) for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5122 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
cf29ca19d4
possible fix for POST character encoding http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5121 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
a2eeb6138c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5120 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d09ddabd09
corrected a design mistake (5-byte hashes not necessary)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c97d0fcee7
modified the domain list export function:
...
- used the new superfast domain list generation from the domain statistics
- better interactive behavior
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
77ee0765a4
- added domain statistic generation to IndexControlURLs_p.html servlet
...
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
80a7bc93d6
- added statistical evaluation about domains that appear during crawling
...
- added tables that show this statistics in CrawlResults web pages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5113 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4fbee21cea
- added fetch-ahead again (had been removed in last commit)
...
- reverted default query mode to verify=false
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5111 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
423a89ebe8
* fix if yacy was installed to a path with whitespace
...
* show nice dots when waiting for restart/update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5110 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc03b0437a
fixed a error case where a second search after a first search with a different search word failed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5109 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
eca171ba2e
fix for case where javascript was not filtered by the html parser
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9667#p9667
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5108 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
e645bae29f
display table in log
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5106 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ead39064c5
fixed problem with wrong result number calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5105 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
hermens
2437beb96c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1360&p=9321#p9321
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5104 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7b12e77a63
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1393&hilit=&p=9655#p9655
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5103 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
05dbba4bab
added logging conditions to all fine and finest log line calls
...
this will prevent an overhead for the generation of the log lines in case that they then are not printed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5102 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d3d41e2ee4
- fixed problem with searching with quotes (still not complete, but not as bad as before)
...
- fixed parsing of crawl-delay statements when seconds were given with float numbers
- enhanced performance of profiling (not too many loggings; not more than one per second)
- removed some debug output
- fixed wrong return type in logging
- added a logging condition in httpd to prevent that logging statements are generated when they are not written (should be added everywhere!)
- fixed wrong word distance computation in RWI management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5101 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
3fbfd5a78b
* fix for non-changing offset on new search term
...
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
219b93df6a
- fixed internal error after receiving chunked POST
...
- removed debug output
- added info for "501 Unknown" messages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5098 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
c245c7a45e
delete index.dhtin/out.heap if restore fails
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9613#p9613
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5097 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
cd19d0aee6
- added warnings for failed transferRWI (dht-in)
...
- fixed parseMultipart (uncompress gzipped body) (dht-in)
- fixed parseMultipart (using content-length only if uncompressed)
- better gzipped POST (chunked instead of content-length) (dht-out)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5096 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
df4ff423c4
added additional properties to query id's to distinguish search events better
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5093 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
d6d9b0f14a
fixed transferRWI.html 'Read timed out'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5092 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
e503158527
Proxy: fix for never ending loading after POST
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5091 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
1a1d57e449
Proxy: added binary passthrough for POST
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5089 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
aa6ae77e5e
- autoReCrawl: fix for filter settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5088 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
8ae29bad57
- fix to previous change of Crawl Profile Names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5087 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
434104e4a0
- change Crawl profile name for autoreCrawl
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5085 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
9ff4fc11da
partial fix (images,audio,video) for proxy and content-type problem http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5084 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
0df2e47012
changed auto recrawl to comply with new date format
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5083 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
d9d9c522a1
addendum to last commit
...
moved recrawl times for standard profiles to constants
calculate new specific dates in cleanup job
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5082 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
480497f7c9
changed recrawl
...
use a specific date to define old documents
this solves an unwanted recrawl-loop during a running crawl
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5081 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
da1b0b2fc6
added two new classes that will be used for the new htcache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5080 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
536e77e8b7
modifications towards a single database operation to read/write http header and cached file at once:
...
- removed distinction between header file types for http and ftp; ftp is simulated by using http properties
- removed all old resourceInfo classes that handled this distinction
- introduced a new distinction between http request and http response objects
- unified new response objects with two other object types that had been introduced elsewhere
- changed all servlet call methods to use the new http request header object type
- divided static object keys for http header properties into request and response types
- refactoring here and there (a large number of type changes and many methods merged/moved)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5079 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
08cdf6db8a
fix for wrong "VegaYacyB" peers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5077 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
4d937f6b21
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1396
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5073 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
bd931a82f7
- added dynamic filters to autoReCrawl.conf
...
- Restrict to sub-path: sub
- Restrict to start-domain: dom
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5070 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
b3fc5e96a3
- removed unused import from bookmarksDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5067 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
bc048db7b6
- bugfix for bookmarksDB's rebuildDates()
...
- dates are now saved as String.valueOf(TimeStamp)
- it might be a good idea to delete (backup) bookmarkDates.db and restart YaCy to rebuild it
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5066 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
3c68905540
remove redundant null checks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5065 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
753a1ae430
- changed default browser from netscape to firefox
...
- fixed "Inefficient use of keySet iterator instead of entrySet iterator" [WMI_WRONG_MAP_ITERATOR, FindBugs]
- fixed some possible null pointer accesses
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5063 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7989335ed6
Preparations to replace the HTCache with a new storage data structure:
...
- refactoring of the HTCache (separation of cache entry)
- added new storage class for BLOBs. (not used yet, this is half-way to a new structure)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5062 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
be28af50f5
- fixed "yacy2yacy no proxy"-problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5058 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago