orbiter
ce4715e305
removed indexing of anchor links and tagging such words as part of urls (that was wrong)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5219 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ce57de6cb3
- fixed re-setting of DHT Send/Receive settings
...
- small change to network grafics: smaller circles / more URLs necessary for full radius; more PPM necessary for full crawling circles
- fixed exclusion search ('-' did not work any more)
- fixed NPE bug when FTP loader wrote to the error-db
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5218 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
31c31e54e4
new tray icon image for different icon sizes (e.g. linux)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5216 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
9589dfe080
* removed trayicon popupmenu title
...
* added some menu items to trayicon
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5213 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
5a637f004d
localized tray
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5212 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
9d4f0325e1
- removed shutdown from search page (we have it in tray now!)
...
- fixed doubleclick action for tray
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
214277dad6
- revert r5202
...
- cleanup
- installer checks for JRE 1.6 only
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5210 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
7afa084207
* add nativ java trayicon, using reflections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5209 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
b97ff24b43
bookmarksDB / xbel.xml:
...
- added support for folder=/foldername
- it crashes if foldername ends with /
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5207 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
6e7d113eac
fix for wrong index initialization after network switch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5203 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
0a0cc3bf67
added missing classes to build target "run"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5201 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7b35d54c6c
fixed some problems with network switching (was not completely 'clean')
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5200 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f0b42e5a98
fixed NPE
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5199 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
8e0de7f180
update to language statistic evaluation:
...
- the condenser does not abandon too small words any more before feeding the statistics
- for text indexing no more urls are used to feed the index (this was wrong, but in contrast the indexing of urls for media search is necessary)
- urls are not used any more to feed the statistics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5197 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1198eeecc7
added language selection to search query:
...
- the language can be selected using a LANGUAGE:<language> element in the query line, i.e.:
java LANGUAGE:en
- the language can be selected with a post element in google-style syntax with the 'rl' element:
?lr=lang_en&query=java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5193 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
00c1535f84
added ranking and evaluation of language type in a search
...
the wanted language is taken from the browser user-agent string
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5192 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
a81cb78211
finally some putHTML on htroot/xml/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5188 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
bfcf9b7aa3
- added language detection using metadata from documents: html and odt documents provide this information
...
- metadata and results from statistical analysis are compared and result is printed out as debug lines
- added ranking profile for wanted language
- added class with ISO 639 table, a list of all valid country codes that will be used for the language identification
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5187 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
5e8bd0f29c
small fixes to getpageinfo_p.xml and htmlFilterContentScraper.java with respect to keyword extraction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5185 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
5b2a57bfd0
- /xml/util/getpageinfo_p.xml added <desc> and <lang> tags
...
- changed htmlFilterContentScraper.getKeywords() to split either space or comma charater not both
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5183 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
e1f67262f7
- added and removed some debugging output
...
- fixed a bug with merge method
- patched wrong output of language identification (not fixed, only patched!)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5181 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ce2a7ed116
integrated language detection classes into condenser environment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5180 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
2b13705839
fixed a mistake in indexing queue processing: documents had been parsed before it was checked if they should be indexed or not. parsing was not necessary for this check, so the check was moved in the queue in front of the document parsing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5179 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
21dbb39afa
switched two balancer cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5177 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1bbf362cef
update to the crawl balancer: better organization and better crawl delay prediction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5176 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ddcf285499
- fixed a bug in performance setting (did not work with german translation)
...
- reduced maximum number of error url references to save some memory (this was actually a small memory leak)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5174 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
0cd0fee546
fixed bug with wrong proxy result enqueueing. See:
...
http://forum.yacy-websuche.de/viewtopic.php?p=8130#p8130
- removed the online status property. This influenced the proxy behavior and created some complexity that was not needed because the online status was never used as it was ceated for (offline browsing)
- checked all proxy identification procedures during crawling and enhanced transparency and error checking
- fixed a proxy identification routine that caused the wrong selection of the proxy result queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5173 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
670244849d
fix for http://forum.yacy-websuche.de/viewtopic.php?p=9835#p9835
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5164 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
fd9233244e
configurable free disk space via disk.free
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5163 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
25a62cdc3f
small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5161 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
73f233bb11
* set resource observer to 1000MB
...
* transparent favicon
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5160 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
5fbccfd75e
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1366&p=9348#p9348
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5155 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
a28faabfd2
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1351&p=9242#p9242
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5154 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
7b63c66a08
- bugfix in bookmarksDB.Tag.hasPublicItems()
...
- this anoying little bug prevented display of public items without admin login for /xml/bookmarks/...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5151 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1fb1665e71
increased dht interval to avoid peer selection failure
...
(maybe too less peers available to fill the big gaps)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5143 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1eb813bd43
shifted index deletion-on-exit rule to the class where the errors are produced
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5141 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
ba76995d2c
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1415
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5140 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
bea6c13139
* with r5137 robotParser didn't work at all -> fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5139 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
3ded1efe84
kelondroExceptionCounter didn't work
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5138 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
ae677e1738
* fix problem in robotparser, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1421&p=9742
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5137 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
383d89481e
count errors before deleting collection.index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5136 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
0bb4fbc403
delete corrupted collecion.index on exit for rebuild on next start
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9725#p9725
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5135 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
b68d06a6e8
performance settings based on network's remote crawl speed
...
removed some _pro values from config
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5134 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d60b2b198d
proxy fixed 'not modified' http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1419
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5133 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
bd0318ba81
* YaCy only supports gzip-encoding, so remove any other encoding from request
...
* fixes http://www.yacy-forum.org/viewtopic.php?f=2&t=163
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5132 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
bb5c898441
enhancements to localsearch behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
42e2d195ac
added hint from http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1294
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5130 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
39964e88fa
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1329#p9121
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5129 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3f3673b6e5
extended balancer:
...
- added automatic time delay in case that a large number of urls come from the same domain
- added additional time delay in case that an url is a dynamic (CGI) url. This shall cause less IO on targets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5128 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3c6e8d2015
set default ppm when network is switched
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3288c19c1a
reduce remote crawl PPM for fresh peers in freeworld to 6 PPM
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5124 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
5ce9a100bb
fix(2) for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5122 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
cf29ca19d4
possible fix for POST character encoding http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5121 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
a2eeb6138c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5120 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d09ddabd09
corrected a design mistake (5-byte hashes not necessary)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
c97d0fcee7
modified the domain list export function:
...
- used the new superfast domain list generation from the domain statistics
- better interactive behavior
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
77ee0765a4
- added domain statistic generation to IndexControlURLs_p.html servlet
...
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
80a7bc93d6
- added statistical evaluation about domains that appear during crawling
...
- added tables that show this statistics in CrawlResults web pages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5113 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
4fbee21cea
- added fetch-ahead again (had been removed in last commit)
...
- reverted default query mode to verify=false
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5111 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
423a89ebe8
* fix if yacy was installed to a path with whitespace
...
* show nice dots when waiting for restart/update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5110 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
fc03b0437a
fixed a error case where a second search after a first search with a different search word failed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5109 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
eca171ba2e
fix for case where javascript was not filtered by the html parser
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9667#p9667
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5108 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
e645bae29f
display table in log
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5106 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ead39064c5
fixed problem with wrong result number calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5105 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
hermens
2437beb96c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1360&p=9321#p9321
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5104 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7b12e77a63
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1393&hilit=&p=9655#p9655
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5103 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
05dbba4bab
added logging conditions to all fine and finest log line calls
...
this will prevent an overhead for the generation of the log lines in case that they then are not printed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5102 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d3d41e2ee4
- fixed problem with searching with quotes (still not complete, but not as bad as before)
...
- fixed parsing of crawl-delay statements when seconds were given with float numbers
- enhanced performance of profiling (not too many loggings; not more than one per second)
- removed some debug output
- fixed wrong return type in logging
- added a logging condition in httpd to prevent that logging statements are generated when they are not written (should be added everywhere!)
- fixed wrong word distance computation in RWI management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5101 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
3fbfd5a78b
* fix for non-changing offset on new search term
...
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
219b93df6a
- fixed internal error after receiving chunked POST
...
- removed debug output
- added info for "501 Unknown" messages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5098 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
c245c7a45e
delete index.dhtin/out.heap if restore fails
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9613#p9613
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5097 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
cd19d0aee6
- added warnings for failed transferRWI (dht-in)
...
- fixed parseMultipart (uncompress gzipped body) (dht-in)
- fixed parseMultipart (using content-length only if uncompressed)
- better gzipped POST (chunked instead of content-length) (dht-out)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5096 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
df4ff423c4
added additional properties to query id's to distinguish search events better
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5093 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d6d9b0f14a
fixed transferRWI.html 'Read timed out'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5092 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
e503158527
Proxy: fix for never ending loading after POST
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5091 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
1a1d57e449
Proxy: added binary passthrough for POST
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5089 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
aa6ae77e5e
- autoReCrawl: fix for filter settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5088 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
8ae29bad57
- fix to previous change of Crawl Profile Names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5087 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
434104e4a0
- change Crawl profile name for autoreCrawl
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5085 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
9ff4fc11da
partial fix (images,audio,video) for proxy and content-type problem http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5084 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
0df2e47012
changed auto recrawl to comply with new date format
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5083 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
d9d9c522a1
addendum to last commit
...
moved recrawl times for standard profiles to constants
calculate new specific dates in cleanup job
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5082 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
480497f7c9
changed recrawl
...
use a specific date to define old documents
this solves an unwanted recrawl-loop during a running crawl
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5081 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
da1b0b2fc6
added two new classes that will be used for the new htcache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5080 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
536e77e8b7
modifications towards a single database operation to read/write http header and cached file at once:
...
- removed distinction between header file types for http and ftp; ftp is simulated by using http properties
- removed all old resourceInfo classes that handled this distinction
- introduced a new distinction between http request and http response objects
- unified new response objects with two other object types that had been introduced elsewhere
- changed all servlet call methods to use the new http request header object type
- divided static object keys for http header properties into request and response types
- refactoring here and there (a large number of type changes and many methods merged/moved)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5079 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
borg-0300
08cdf6db8a
fix for wrong "VegaYacyB" peers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5077 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
4d937f6b21
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1396
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5073 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
bd931a82f7
- added dynamic filters to autoReCrawl.conf
...
- Restrict to sub-path: sub
- Restrict to start-domain: dom
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5070 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
b3fc5e96a3
- removed unused import from bookmarksDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5067 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
bc048db7b6
- bugfix for bookmarksDB's rebuildDates()
...
- dates are now saved as String.valueOf(TimeStamp)
- it might be a good idea to delete (backup) bookmarkDates.db and restart YaCy to rebuild it
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5066 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
3c68905540
remove redundant null checks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5065 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
753a1ae430
- changed default browser from netscape to firefox
...
- fixed "Inefficient use of keySet iterator instead of entrySet iterator" [WMI_WRONG_MAP_ITERATOR, FindBugs]
- fixed some possible null pointer accesses
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5063 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7989335ed6
Preparations to replace the HTCache with a new storage data structure:
...
- refactoring of the HTCache (separation of cache entry)
- added new storage class for BLOBs. (not used yet, this is half-way to a new structure)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5062 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
be28af50f5
- fixed "yacy2yacy no proxy"-problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5058 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
f99c307eff
* correct debian build dependencies
...
* add huge mem page detection in general initscript
* disable logging completely in jmimemagic-library
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5056 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
bdae051d9a
- extended new performance graph (better timing)
...
- added paths for new libraries in classpath for eclipse
- refactoring to remove compiler warnings (static access to finals variables)
- removed some unused import
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5055 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d9cea5ff23
removed annotations which broke the build with java 1.5
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5054 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
a087090bbb
fixed starting crawl results in "No parser available to parse mimetype 'application/octet-stream'"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5047 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7e7e6a099a
undo 5044
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5046 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
f2d0bd7790
fix for NPE in JakartaHttpClient.setProxy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5045 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago