danielr
0f9c0bd0d5
fix for ConcurrentModificationException at de.anomic.index.indexContainerHeap$heapCacheIterator.next(indexContainerHeap.java:324)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5300 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
103ad2a437
some javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5299 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b098522977
some very small advances to index utf-8 (not working yet), inserted also debugging code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5298 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2f49666908
integrated the character decoding into the parser, removed old code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5297 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
49293c1358
fix for deadlock in new encoder :-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5296 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0edec2b760
FULL redesign of algorithms in htmlTools to encode/decode strings from/to unicode and html.
...
The old process used a not really efficient way to detect html encoding strings in texts.
All calling methods had been adoped to call the new class in an enhanced way with less parameters.
Many classes in interfaces used a XML encoding only (instead of full html conversion from unicode to html); this behavior was not changed with this commit but should be controlled again since it points out possible XSS leaks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5295 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
958ec20cd0
removed specialized umlaute-handling in html parser. This has to be replaced by something that is able to transfer all possible html encodings into utf-8. Please see SVN 5293 for test cases.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5294 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2e53cbc66a
should compile now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5292 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
f3bf2e379e
should compile again
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5291 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
dd8441f102
fix bug: data from plasmaParser is allready converted to UTF-8
...
After removing the restrictions in the code, YaCy should be able to index Unicode-charaters!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5290 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6941bf42b1
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5288 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9b0c4b1063
redesign of parts of the new BLOB buffer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5287 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1778fb420d
- added some performance tweaks to the new BLOB buffer
...
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9663e61449
added another class to handle BLOB writings to the new HTCACHE data storage:
...
- entries are buffered and written as stream with many entries at once (saves many IO accesses)
- entries are compressed with gzip: increases capacity of cache
- concurrency for stream-writing and compression: all writings to the cache are non-blocking
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5284 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
382226da94
fix for bug introduced in SVN 5281: parameters were switched
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5282 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
f2fd043797
refactoring (moved duplicate code into methods)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5281 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
c612046e5e
r5278 java 1.5 compatible
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5280 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
af71ec93bf
ops, forgot to import something
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5279 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
9e65e9141c
* always use UTF-8 for encoding hashes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5278 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
826ca79735
refactoring and new architecture to store the files of the web cache:
...
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods
this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
f095137238
- respecting httpdMaxBusySessions (refusing new connections if limit is hit)
...
- comments in serverBusyThread converted to JavaDoc
- better debug output for npe-case in diskUsage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5274 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8ba33f104e
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5269 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
998861acfd
- some refactoring in BLOBHeap to enable more gap processing functions
...
- better gap merging in BLOBHeap
- shrinking of heap file if gap is at end of file when file is closed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5268 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
9d50bfd0b3
fix for npe: http://forum.yacy-websuche.de/viewtopic.php?p=10562
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5267 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
766cad6e93
enhancement in memory management of BLOB Heap files / merging of deleted entries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5266 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7860d5d632
fix for bug in seed list management (cause was bad class overloading, only visual effects!)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ffed5fc415
fixed problem with lost peers in database
...
migrated seedDB from BLOBTree to BLOBHeap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6fb865fbdc
- fix of bug in iterator in kelondroBLOBHeap which caused bug in crawl profile listing
...
- some refactoring of classes that use kelondroMap (Map instead of HashMap)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5262 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2d65887723
- fix for bug in new profile handling
...
- added a new feature in ymageChart (cannot be seen yet, just wait... will be used in profiling chart)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5261 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ff68f394dd
fix for problem with balancer and lost crawl profiles:
...
if crawl profile ist lost, no robots.txt is loaded any more
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fb8d9850ea
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1462
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5248 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
0d1a2f6183
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1461
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5247 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9ac16f565b
- fixed several bugs in database management functions
...
- fixed a display bug for the performance graph
- fixed deadlock when initialization of awt happens simultanously
- removed some debugging output
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5245 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
820a03f9d6
- removed some warnings
...
- used fix in SVN 5233 for ysearch.java and search.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fe2792e9ce
use accept-language header instead of user agent for language detection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5235 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c8bdd965ec
- larger update time for status page
...
- balancer writes cause of robots.txt in log file for crawl delay
- removed log output for forced GC
- smaller RAM flush for RWI cache, should cause more usage of cache and faster crawling
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5228 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
dda771db9d
- search result layout
...
- tray only for windows
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ce4715e305
removed indexing of anchor links and tagging such words as part of urls (that was wrong)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5219 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ce57de6cb3
- fixed re-setting of DHT Send/Receive settings
...
- small change to network grafics: smaller circles / more URLs necessary for full radius; more PPM necessary for full crawling circles
- fixed exclusion search ('-' did not work any more)
- fixed NPE bug when FTP loader wrote to the error-db
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5218 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
31c31e54e4
new tray icon image for different icon sizes (e.g. linux)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5216 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
9589dfe080
* removed trayicon popupmenu title
...
* added some menu items to trayicon
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5213 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
5a637f004d
localized tray
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5212 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
9d4f0325e1
- removed shutdown from search page (we have it in tray now!)
...
- fixed doubleclick action for tray
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
214277dad6
- revert r5202
...
- cleanup
- installer checks for JRE 1.6 only
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5210 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
7afa084207
* add nativ java trayicon, using reflections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5209 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
b97ff24b43
bookmarksDB / xbel.xml:
...
- added support for folder=/foldername
- it crashes if foldername ends with /
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5207 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6e7d113eac
fix for wrong index initialization after network switch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5203 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
0a0cc3bf67
added missing classes to build target "run"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5201 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7b35d54c6c
fixed some problems with network switching (was not completely 'clean')
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5200 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f0b42e5a98
fixed NPE
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5199 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8e0de7f180
update to language statistic evaluation:
...
- the condenser does not abandon too small words any more before feeding the statistics
- for text indexing no more urls are used to feed the index (this was wrong, but in contrast the indexing of urls for media search is necessary)
- urls are not used any more to feed the statistics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5197 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1198eeecc7
added language selection to search query:
...
- the language can be selected using a LANGUAGE:<language> element in the query line, i.e.:
java LANGUAGE:en
- the language can be selected with a post element in google-style syntax with the 'rl' element:
?lr=lang_en&query=java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5193 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
00c1535f84
added ranking and evaluation of language type in a search
...
the wanted language is taken from the browser user-agent string
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5192 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
a81cb78211
finally some putHTML on htroot/xml/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5188 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bfcf9b7aa3
- added language detection using metadata from documents: html and odt documents provide this information
...
- metadata and results from statistical analysis are compared and result is printed out as debug lines
- added ranking profile for wanted language
- added class with ISO 639 table, a list of all valid country codes that will be used for the language identification
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5187 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
5e8bd0f29c
small fixes to getpageinfo_p.xml and htmlFilterContentScraper.java with respect to keyword extraction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5185 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
5b2a57bfd0
- /xml/util/getpageinfo_p.xml added <desc> and <lang> tags
...
- changed htmlFilterContentScraper.getKeywords() to split either space or comma charater not both
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5183 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e1f67262f7
- added and removed some debugging output
...
- fixed a bug with merge method
- patched wrong output of language identification (not fixed, only patched!)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5181 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ce2a7ed116
integrated language detection classes into condenser environment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5180 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2b13705839
fixed a mistake in indexing queue processing: documents had been parsed before it was checked if they should be indexed or not. parsing was not necessary for this check, so the check was moved in the queue in front of the document parsing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5179 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
21dbb39afa
switched two balancer cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5177 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1bbf362cef
update to the crawl balancer: better organization and better crawl delay prediction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5176 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ddcf285499
- fixed a bug in performance setting (did not work with german translation)
...
- reduced maximum number of error url references to save some memory (this was actually a small memory leak)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5174 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0cd0fee546
fixed bug with wrong proxy result enqueueing. See:
...
http://forum.yacy-websuche.de/viewtopic.php?p=8130#p8130
- removed the online status property. This influenced the proxy behavior and created some complexity that was not needed because the online status was never used as it was ceated for (offline browsing)
- checked all proxy identification procedures during crawling and enhanced transparency and error checking
- fixed a proxy identification routine that caused the wrong selection of the proxy result queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5173 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
670244849d
fix for http://forum.yacy-websuche.de/viewtopic.php?p=9835#p9835
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5164 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
fd9233244e
configurable free disk space via disk.free
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5163 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
25a62cdc3f
small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5161 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
73f233bb11
* set resource observer to 1000MB
...
* transparent favicon
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5160 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5fbccfd75e
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1366&p=9348#p9348
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5155 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
a28faabfd2
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1351&p=9242#p9242
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5154 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
7b63c66a08
- bugfix in bookmarksDB.Tag.hasPublicItems()
...
- this anoying little bug prevented display of public items without admin login for /xml/bookmarks/...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5151 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1fb1665e71
increased dht interval to avoid peer selection failure
...
(maybe too less peers available to fill the big gaps)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5143 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1eb813bd43
shifted index deletion-on-exit rule to the class where the errors are produced
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5141 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
ba76995d2c
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1415
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5140 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
bea6c13139
* with r5137 robotParser didn't work at all -> fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5139 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
3ded1efe84
kelondroExceptionCounter didn't work
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5138 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
ae677e1738
* fix problem in robotparser, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1421&p=9742
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5137 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
383d89481e
count errors before deleting collection.index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5136 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
0bb4fbc403
delete corrupted collecion.index on exit for rebuild on next start
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9725#p9725
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5135 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
b68d06a6e8
performance settings based on network's remote crawl speed
...
removed some _pro values from config
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5134 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
d60b2b198d
proxy fixed 'not modified' http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1419
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5133 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
bd0318ba81
* YaCy only supports gzip-encoding, so remove any other encoding from request
...
* fixes http://www.yacy-forum.org/viewtopic.php?f=2&t=163
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5132 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bb5c898441
enhancements to localsearch behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
42e2d195ac
added hint from http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1294
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5130 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
39964e88fa
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1329#p9121
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5129 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3f3673b6e5
extended balancer:
...
- added automatic time delay in case that a large number of urls come from the same domain
- added additional time delay in case that an url is a dynamic (CGI) url. This shall cause less IO on targets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5128 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3c6e8d2015
set default ppm when network is switched
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3288c19c1a
reduce remote crawl PPM for fresh peers in freeworld to 6 PPM
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5124 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
5ce9a100bb
fix(2) for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5122 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
cf29ca19d4
possible fix for POST character encoding http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5121 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
a2eeb6138c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5120 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d09ddabd09
corrected a design mistake (5-byte hashes not necessary)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c97d0fcee7
modified the domain list export function:
...
- used the new superfast domain list generation from the domain statistics
- better interactive behavior
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
77ee0765a4
- added domain statistic generation to IndexControlURLs_p.html servlet
...
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
80a7bc93d6
- added statistical evaluation about domains that appear during crawling
...
- added tables that show this statistics in CrawlResults web pages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5113 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4fbee21cea
- added fetch-ahead again (had been removed in last commit)
...
- reverted default query mode to verify=false
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5111 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
423a89ebe8
* fix if yacy was installed to a path with whitespace
...
* show nice dots when waiting for restart/update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5110 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc03b0437a
fixed a error case where a second search after a first search with a different search word failed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5109 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
eca171ba2e
fix for case where javascript was not filtered by the html parser
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9667#p9667
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5108 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
e645bae29f
display table in log
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5106 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ead39064c5
fixed problem with wrong result number calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5105 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
hermens
2437beb96c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1360&p=9321#p9321
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5104 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7b12e77a63
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1393&hilit=&p=9655#p9655
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5103 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
05dbba4bab
added logging conditions to all fine and finest log line calls
...
this will prevent an overhead for the generation of the log lines in case that they then are not printed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5102 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d3d41e2ee4
- fixed problem with searching with quotes (still not complete, but not as bad as before)
...
- fixed parsing of crawl-delay statements when seconds were given with float numbers
- enhanced performance of profiling (not too many loggings; not more than one per second)
- removed some debug output
- fixed wrong return type in logging
- added a logging condition in httpd to prevent that logging statements are generated when they are not written (should be added everywhere!)
- fixed wrong word distance computation in RWI management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5101 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
3fbfd5a78b
* fix for non-changing offset on new search term
...
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
219b93df6a
- fixed internal error after receiving chunked POST
...
- removed debug output
- added info for "501 Unknown" messages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5098 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
c245c7a45e
delete index.dhtin/out.heap if restore fails
...
see http://forum.yacy-websuche.de/viewtopic.php?p=9613#p9613
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5097 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
cd19d0aee6
- added warnings for failed transferRWI (dht-in)
...
- fixed parseMultipart (uncompress gzipped body) (dht-in)
- fixed parseMultipart (using content-length only if uncompressed)
- better gzipped POST (chunked instead of content-length) (dht-out)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5096 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
df4ff423c4
added additional properties to query id's to distinguish search events better
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5093 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
d6d9b0f14a
fixed transferRWI.html 'Read timed out'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5092 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
e503158527
Proxy: fix for never ending loading after POST
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5091 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
1a1d57e449
Proxy: added binary passthrough for POST
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5089 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
aa6ae77e5e
- autoReCrawl: fix for filter settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5088 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
8ae29bad57
- fix to previous change of Crawl Profile Names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5087 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
434104e4a0
- change Crawl profile name for autoreCrawl
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5085 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
9ff4fc11da
partial fix (images,audio,video) for proxy and content-type problem http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5084 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
0df2e47012
changed auto recrawl to comply with new date format
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5083 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
d9d9c522a1
addendum to last commit
...
moved recrawl times for standard profiles to constants
calculate new specific dates in cleanup job
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5082 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
480497f7c9
changed recrawl
...
use a specific date to define old documents
this solves an unwanted recrawl-loop during a running crawl
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5081 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
da1b0b2fc6
added two new classes that will be used for the new htcache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5080 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
536e77e8b7
modifications towards a single database operation to read/write http header and cached file at once:
...
- removed distinction between header file types for http and ftp; ftp is simulated by using http properties
- removed all old resourceInfo classes that handled this distinction
- introduced a new distinction between http request and http response objects
- unified new response objects with two other object types that had been introduced elsewhere
- changed all servlet call methods to use the new http request header object type
- divided static object keys for http header properties into request and response types
- refactoring here and there (a large number of type changes and many methods merged/moved)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5079 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
08cdf6db8a
fix for wrong "VegaYacyB" peers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5077 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
4d937f6b21
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1396
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5073 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
bd931a82f7
- added dynamic filters to autoReCrawl.conf
...
- Restrict to sub-path: sub
- Restrict to start-domain: dom
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5070 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
b3fc5e96a3
- removed unused import from bookmarksDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5067 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
bc048db7b6
- bugfix for bookmarksDB's rebuildDates()
...
- dates are now saved as String.valueOf(TimeStamp)
- it might be a good idea to delete (backup) bookmarkDates.db and restart YaCy to rebuild it
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5066 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
3c68905540
remove redundant null checks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5065 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
753a1ae430
- changed default browser from netscape to firefox
...
- fixed "Inefficient use of keySet iterator instead of entrySet iterator" [WMI_WRONG_MAP_ITERATOR, FindBugs]
- fixed some possible null pointer accesses
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5063 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7989335ed6
Preparations to replace the HTCache with a new storage data structure:
...
- refactoring of the HTCache (separation of cache entry)
- added new storage class for BLOBs. (not used yet, this is half-way to a new structure)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5062 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
be28af50f5
- fixed "yacy2yacy no proxy"-problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5058 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
f99c307eff
* correct debian build dependencies
...
* add huge mem page detection in general initscript
* disable logging completely in jmimemagic-library
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5056 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bdae051d9a
- extended new performance graph (better timing)
...
- added paths for new libraries in classpath for eclipse
- refactoring to remove compiler warnings (static access to finals variables)
- removed some unused import
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5055 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
d9cea5ff23
removed annotations which broke the build with java 1.5
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5054 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
a087090bbb
fixed starting crawl results in "No parser available to parse mimetype 'application/octet-stream'"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5047 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7e7e6a099a
undo 5044
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5046 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
f2d0bd7790
fix for NPE in JakartaHttpClient.setProxy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5045 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
bb6a6fc233
fixed 'FileUploadException Stream ended unexpectedly'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5044 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
8422ee5ec4
- fixed UnsupportedEncoding (in proxy) using defaultCharset if no characterEncoding can be determined
...
- serverFileUtils.copy* use now Charset instead of String
- added some warnings for ignored exceptions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5043 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
hermens
3ac1988059
Add some sanity checks for invalid seeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5042 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
hermens
cff4393f0c
Fix HTCache so oldest Files get deleted first
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5041 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
31d97f2b9f
replaced httpd.parseMultipart() by a 'right' implementation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5040 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
621b473b18
* removed some warnings of findbugs ( http://findbugs.sf.net )
...
- removed unnecessary code (unused variables, String.toString)
- corrected some calculations (cast int to double or long ;)
- improved little performance (using Integer.valueOf() instead of new Integer)
- log if some File-actions fail (mkdir(), delete(), ...) and some ignored exceptions
- finalized some (more) fields
- finally close some streams
- made inner classes static if not using environment
- generalized some equals (from specificClass to Object)
- fixed some potential nullpointer accesses
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5039 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
0500b1179e
added a 2 min start up delay to serverBusyThread autoReCrawl to avoid a Null Pointer Exception...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5035 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
e1574fe02e
- added autoReCrawl folders to bookmarks (DATA/SETTINGS/autoReCrawl.conf)
...
- the serverBusyThread checks folders every 60 min. (==> autoReCrawl_idlesleep in yacy.conf)
- added option to create bookmarks from CrawlStart URL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5033 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ebb40d324b
enhanced memory chart: shows now also the size of the word cache as third vector.
...
The PPM is now shown without a scale, but with a new anotation at the chart entry.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5032 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
17b7845eb5
* refactoring
...
- moved constants from plasmaSwitchboard to own class (all 232 ;)
- moved remoteProxy-Methods to httpRemoteProxyConfig, better names
- removed some unnecessary code (else-statements)
* formatting (correct indentation)
* minor bugfixes (due to findbugs.sf.net)
* hopefully fixed "missing quote" (announcing StringParts as UTF-8)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5031 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
3bb870bfcd
added final where possible
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5030 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
7e92484400
fix for open browser on windows 2000
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5029 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
b0724e5ec0
* add config option to disable cookie monitoring (disabled by default)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5028 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
0b2f67577e
Index Transfer:
...
- fix for chunk size calculation
- fix: if chunk size was 1, an infinite selection loop ran because no entries were found. if chunk size fails <=3 it will be set back to 500
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5023 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
694084c570
fix for NPE on shutdown
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5021 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
5f77f55ed7
possible fix for negative speed values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5019 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
50ef5c406f
- refactoring of robots parser (removed opaque Objects[] result vector)
...
- added Allow-component to robots result object
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5016 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7913bdb75b
Flextable: filename in errormessage if inconsistent
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5015 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
d42eae25f8
yacyTray:
...
fix for unproper shutdown
some messages
installer:
start shortcuts minimized
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5014 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
c3d461d191
- removed superfluous copyright statement
...
- updated my email address
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5011 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3ca98fee42
removed superfluous copyright statement
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5010 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
c049d80fbd
fixed login problem with yacy as proxy (POST and Cookies)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5009 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
62afea0c9f
some improvements for yacyTray
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5008 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7c110e07f0
removed debug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5006 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
eadc204130
gzip POST wiederholbar gemacht (macht transferURL stabiler)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5004 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
28c39e2aa4
fix for new starter files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5002 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
fa695c2d9f
tray is now only shown on Windows and doesn't block on linux
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4997 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
d77ed28e2f
temporary disabled tray because of flaws on only-shell-linux
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4996 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
f8a1e3175e
new yacyTray
...
this will make a YaCy icon in the tray area on supported platforms
enabled by default
the search page will open on double click
used JDIC 0.9.4 from https://jdic.dev.java.net/
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4992 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
05c26d58d9
fixed missing remove operation in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4990 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
606b323a2d
fixed bug that appeared when a new crawl ist started
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4989 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
38eb5bd1ee
fixed a bug in kelondroBLOBHeap. The following files are probably inconsistent and should be deleted:
...
DATA/HTCACHE/responseHeader.heap
DATA/PLASMADB/crawlRobotsTxt.heap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4988 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
28d5703f8a
- fixed a bug in Robots.txt loader which could have caused that robots.txt files had been loaded from the same domain more than once
...
- patch in BLOBHeap to prevent OOM during startup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4987 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7b1c9e6aee
discovered and removed a (possibly large) memory leak:
...
many classes used the kelondroMapDataMining (was: kelondroMapObjects) which adds statistical
functions to the kelondroMap (was: kelondroObjects), but these functions were not used by these
classes. Especially the HTCACHE and robots.txt database allocate a very large number of objects
for statistical use, but never used them. By replacing the kelondroMapDataMining with the
kelondroMap object for these classes now less memory is allocated.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4986 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
0f5fe8cc53
refactoring of method calling for objects from kelondroMapDataMining
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4985 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
01d1ae6676
patch for negative time in case that the time of the computer is changed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4984 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
4acf0a61cd
refactoring of kelondroObjects (mainly renaming to kelondroMap)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4982 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
441e9c861e
fix for npe in HTCache cleaning process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4981 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f7aaeb3fad
created new main menu entry 'Customization and Integration'
...
- moved some already existing servlets to this menu
- renamed the skin servlet to appearance
- added a set-to-default-button to the search page appearance setting
- removed the peer profile servlet which is now replaced by a field in the new appearance servlet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4980 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
5488543b8f
disabled disk usage logpoints
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4979 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1e6d12f146
Major update to BLOB data structures:
...
- introduced a new BLOB file format: kelondroBLOBHeap. This is a flat file with an index in RAM.
very similar to the eco-tables, but with flexible value sizes. It will replace the kelondroBLOBTree,
which is based on a kelondroTree, a file-AVL-based index data structure.
- the HTCACHE header file was replaced by the new blob heap file structure
- the robots.txt file was replaced by the new blob heap file structure
- the robots parser was enhanced (bugfixing for double-loading of the same robots.txt)
- other BLOB-dependent data structures were prepared to use also the new BLOB heap
- fixed a bug in the snippet fetch process: the file header was not written to the header index
There should now be less IO during snippet fetch and during crawling
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4978 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
81f75f5056
- removed unnecessary classes (these objects are much easier to handle using generics)
...
- generalized BLOB referencing. This is the preparation to use another BLOB class, the kelondroHeap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4977 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
b38f467e3c
better SRU compliance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4976 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7052f2f61f
- added copyright header of ResourceObserver
...
- commented/removed some code to eliminate code warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4974 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1400cdc91e
- refactoring of resourceObserver (moved it to crawler)
...
- partly redesign of diskUsage: little bit more functional behavior, less side effects, better error case handling
- the resourceObserver can now show a error message if the diskUsage is 'out of order'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4973 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
b6301a54fa
* added class ListDirs to provoid generic listing of directories in systemdirectories and jar-files
...
* yacy runs, when classes are in a jar-file (->build-jar ant-target)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4971 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
f2e2d09916
- fix for index transfer
...
- imported a random startpoint function from plasmaDHTChunk
in case there was already a gap at the beginning of the index, the transfer process was endless selecting from first startpoint
tested & working on my index
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4970 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
a6719dfd2b
- refactoring of robots parser
...
- no more keep-order parameter in remove (it was not possible to make this strict, and not useful)
- some small enhancements in balancer
- robots parser without references in switchboard
- changes synchronization in robots
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4969 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
e81be7d4f2
added many missing user-agent declarations for yacy http client connections.
...
the most important fix was the addition of the yacybot user-agent for robots.txt loading,
because web masters look for that access to see if the crawler behaves correctly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4968 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
474e29ce4a
added options to configure the 'corporate identity'-icons, the home page link and the greeting line from
...
the skin menue. Additionally an example is given there how to integrate a search page with an iframe.
Please see the skin menu.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4967 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
474659a71f
- modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order
...
- added option to set minimum crawl delta for domains in balancer
- added default values to crawl deltas in yacy.init
- added configuration for these deltas in performance queues
- enhanced performance setting computation (more time for indexing queue for a faster flush
- remote crawling is now enabled during local crawling if indexer has space and time for more links
- added database stub for new distributed file system
- refactoring of time computation to get an abstraction level that will be used by a TTL rule in new distributed file system
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4966 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
080cda97ef
added another peer selection rule:
...
- select also non-robinson (dht-) peers if their peer tags match with search words
- the peer tag '*' can now act as catch-all rule: shall be selected always
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4963 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d37fd064f9
changed peer selection for search targets:
...
- less dht targets are selected
- more other peers are selected: all robinson peers with more than one million urls
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4962 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
69aac0d74c
modified the diskUsage class regarding the following two aspects:
...
1. The usage and dependency of the plasmaSwitchboad was used many times in the past but this was
a bad mistake. The classes should be independent from the switchboard to support a better abstraction. Therefore the object was removed. The parameters from the switchboard are computed outside and then handed over.
2. the class is considered as a tightly connected to hardware resources. Classes which handle data that cannot be replicated because it would need to replicate hadware should not support dynamic object allocation, but should be coded as collection of private static methods. Therefore all class objects had been transformed into static private objects.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4961 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
da917cf4b1
undo reduced menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4960 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
0c1dc703e4
- set staticIP at startUp
...
- added setting for reduced menu (simpleMenu)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4959 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
f7f9ceb967
diskUsage: replaced blocking sleep with semaphore
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4957 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
4a53649ee7
fixed dht-urls and ranking distribution log statistics
...
* NOTE: please have in mind that there can be whitespaces in pathnames
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4956 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
8d83185cb4
fixed dht-chunks/protocol log statistics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4955 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
63eadfdf84
fixed unlimited FileSizeLimit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4954 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
2dc7c00c1c
fixed indexing log statistics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4953 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
dba7ba079e
fixed NPE seen with queues_p.xml (serverClassLoader finds already loaded classes)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4952 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
273fb01142
revert last fix; was wrong
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4950 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
b6f50851fa
fix memory requirement calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4949 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
ac85c52bae
better readability for MIN_FREE_DISK_SPACE
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4948 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
54a73b58cf
fixed restart on Windows when directory had spaces in it's name
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4947 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
609aaf0df3
rework of the windows part
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4943 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
1a4f26ba30
exclude HTDOCS from recursiv scan
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4942 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
6c07e894d9
add needed sleep
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4941 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
hermens
d742cc080c
Fix for RAMCache not flushing
...
see: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1255
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4940 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
6b7e873962
resourceObserver refactoring and some synchronisation for console output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4939 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
6bdd99e065
- more asserts to solve the ooB-problem
...
- better caching (?), lets see how it behaves
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4937 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
b928ae492a
some code-cleanup and possible speed enhancements in different core methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4935 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
6a9cc29cdd
workaround for IndexOutOfBoundsException in ResultURLs.getExecutorHash() seen @ CrawlResults.html?process=4
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4934 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
c998dc6556
- added security functions to flush url and search caches in case that memory is full
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4933 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f4ae8082c3
- better error analysis for ooRange Exception in kelondroBase64Ordering
...
- quadcore support for kelondroRowSet array ordering
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4932 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
84cbe75005
more asserts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4930 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
e269c12710
small changes in partition routine
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4929 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
31efb8fbee
- fix for LOG path generation when the DATA/LOG does not exists (fix for bug introduced in SVN 4923)
...
- some more/better asserts
- slight performance enhancements in remove method in index management. Works for all who do not run using asserts (the majority)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4928 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
877299cc74
better installer on Windows Vista
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4927 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
21c87c36e3
added a log line
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4925 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
68c38c2d34
- WatchCrawler shows status without JavaScript
...
- Performance can be scaled + DHT-profile
- names for pool-threads
- some small refactorings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4923 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
fc79f013c4
better solution to update shortcut
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4920 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
c0dfe49743
also exclude collection.0028.commons and RANKING at startup check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4919 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
11656741f1
exclude LOCALE and RELEASE at startup check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4917 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
48edbef5c7
* fix: display proper port on 1st startup
...
* new message on portchange
* first implementation of external link-update for search page (still inactive)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4915 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
0727bb1e63
rework of console message handling; add of debugging output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4914 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
43c47218ef
fox for open browser on Windows
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4912 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f5ef7f222e
- fixed a bug in parser (directory paths had not been recognized)
...
- no access check when a search is made only local without snippet fetch
- added comment and status message in resourceObserver (this takes very long at startup time!)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4911 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
ed24eab737
small fix for windows in resource observer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4909 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
hermens
75b4a5ced4
reinstate old timeout values for transferRWI and transferURL
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4903 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3330181aa0
refactoring:
...
find a better way to store BLOBs; generalize current BLOG data structure (kelondroDyn)
and prepare it to replace it with something better. The best candidate is the kelondroHeap,
which will become the kelondroBLOBHeap;
removed also some never-used classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4902 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
9a9737a54e
fix for "no more elements available" exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4901 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
260553c3a5
better messages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4897 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
6afeb535cd
another bugfix for the windows drive check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4896 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
b416af7568
bugfix for the windows drive check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4895 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
4b71912e76
fixed wrong class name
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4894 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7feae906aa
- organize imports
...
- removed potential null pointer accesses
- removed unnecessary casts
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4893 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
det
f597185026
Initial import of the resource observer framework
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4892 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d3037c2950
Accept all SSL-certificates (not only valid and self-signed), but put a warning into log file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4888 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
e0e7f86f82
some bugfixes for the peer-ping process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4885 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
cbe722c480
small code cleanUp
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4884 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
b21598bdd0
- enhanced handling of own IP address inside seed
...
- prevention of false information of own IP address
- enabled searching before an own IP address is assigned (before first ping happened)
- removed warning about limited search function
- added better time-out settings for peer-ping process (10 seconds complete, 5 seconds for back-ping)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4883 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
40d7f485f3
- fixed several NPE bugs
...
- fixed loosing of own seed hash (hopefully)
- fixed a bug with crawl start s beginning with (bookmark) files
- added better IP recognition during hello process
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4882 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
2f381b8d7a
- fixed at least two causes for a NPE after a use case switch.
...
A large refactoring was neccessary
- added another crawl start option: automatic restriction to sub-path
- removed crawlStartSimple and renamed crawl start expert
to crawl start (without expert)
- some changes to texts in crawl start
- added some more deletions when an web index is deleted:
delete also queues and robots cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4881 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
2a604b7402
added superfast search result computation which can be obtained for local search when snippet fetching is disabled. An example search for the rss interface would be:
...
http://localhost:8080/yacysearch.rss?query=yacy&Enter=Search&contentdom=text&count=10&resource=local&verify=false
(just add "&verify=false")
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4878 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
9bef20b537
- added cleanup for unused server loggings: they are removed after the client had not been seen since one hour
...
- removed configBasic popup trigger when no password is set
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4875 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
5f35ba6f05
added a list for content of the brute-force registration hashtable in access tracker
...
the purpose of this list is not only to view attempts from read attacks, but also to show if the
yacy-yacy protocol is working with respect to the bug that was fixed with SVN 4869
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4873 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
74e3a547db
more logging for mySeed loading error cases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4871 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
6f1a3fce05
BF Bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4869 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
b09af53643
proper links to directories in repository dirlisting
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4867 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
ba4091c5b2
proxy sends status code now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4860 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1a1841392c
small fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4859 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
698293ef32
fix for lonely peers in networks with only one peer, especially intranet peers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4858 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
8a0e401320
- fix for bad code in peer actions
...
- fix for bad images in basic config
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4854 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
0b52ef3e4b
- update of grafics
...
- check for startsWith("127.") instead of equals("127.0.0.0")
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4853 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
18ad12eceb
added another fix for localhost addresses in seeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4852 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
11e00a0849
- refactoring of seedURL handling
...
- additional check for seedURL pointing to localhost: deny such peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4851 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
e91bf4c8cc
- fix for bad reset of index / bad index location after deletion
...
- some modification of rssTerminal window location and size
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4850 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
f284386b63
update deploy improvements for windows - ready for release now :-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4849 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
fd8bd5d0d1
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1176 (encoding issue)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4848 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
25192e0d36
added a deletion button to indexControlRWIs that deletes the complete web index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4847 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
01b3e9431a
- fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1140&p=7626#p7626
...
- less dots for ppm bar in watchcrawler (one dot for each 10 ppm)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4846 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
8be462986e
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1174&p=7841#p7841
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4845 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f8b015949c
fix for bug in html scraper that appears if opening and closing tag are not both in same case
...
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1173&p=7836#p7836
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4844 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d8277e6af1
- added parsing of numeric html entities for crawler
...
- fixed a bug in search response
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4843 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
low012
f06c046f6c
*) refactoring (several instances of Indetificator can be created and used parallel now)
...
*) fixed a potential NPE
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4841 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
0c173821fd
more access security regarding database access and snippet retrieval: restrict number of results for not-authorized searchers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4838 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
eac62a6882
* ported restart on Windows to unix-style, works on _noconsole now
...
* removed Win9x scripts from build for more tidiness and less decisions for newbies
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4835 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
03438ee977
added missing implementation of network-path reference
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4834 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
2ba7914f0b
fix for NPE exception while fetching remote crawl jobs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4833 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
4a48717017
* automatic update for windows
...
pleas disable before release because 2nd update fails at the moment
and commandline handling has to be improved for windows
* update via new unTar class
please review stream- and exceptionhandling because I'm fairly new to Java
maybe it can be done concurrent
* updated windows startscripts to values from yacy.init
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4832 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
faed00d75d
added use cases to basic configuration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4831 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
4229cd275c
fixed several details about network switching, default password, random password and localhost authentification
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4830 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d9d1c8de70
more protection against remote shutdown attacks: prevent loading using the crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4829 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
c1d721dd2d
fix for attacks on localhost-authorized peers from web pages with links to localhost addresses:
...
checking of referer in access
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4828 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
6aa9bf43f9
fixed server port change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4826 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
low012
a7dadf7f2f
*) first version of a way to determine the language a text is written in (not perfect, but it works)
...
*) statistical data of languages can be found in the *.lng files in the new directory called "langfiles"
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4824 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1127d62b64
some enhancements to the access tracker (less synchronization)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4812 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
56a300f92a
bugfix / local-search predicate
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4811 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
2f29ab8779
more target server access security
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4809 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3bd1db776a
implemented switch for admin authorization from localhost:
...
- access is granted for localhost users to administration pages by default
- the default setting can be changed in the BasicConfig.html page
- if the BasicConfig page was accessed with post and no password was submitted, a random password is generated
- a headless installation MUST give a password upon first call of the configuration page, otherwise they will not be able to access it again
- if no password is given within 10 minutes after start-up, a random password is generated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4804 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
cfe6790498
- added option to switch between yacy networks, especially between the two default networks (freeworld and intranet),
...
from the ConfigNetwork online interface
- to make this possible, a large refactoring and reorganisation of data structures was necessary
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4803 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
78087da287
- changed seed file storage to clear text
...
- fixed kill script
- fixed saving of seed file (had been corrupted by latest changes)
- some refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4799 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
5fde679acb
- fixed problem in performance configuration
...
- extended rss fetch size for rssTerminal
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4798 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
db032fb6de
- added RWI transmissions to the event terminal
...
- fixed bug in Collage
- added 'embedded mode' to collage
- integrated Collage to terminal_p as iframe in embedded mode (Pictures now visible in terminal_p)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4797 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
239cc4428d
- better domain graph, faster when more links exist, looks better
...
- new authorization rule: localhost is always authorized for administration. This solves many problems with ajax, and also fixed a problem in rssTerminal
- fix bug in RSSFeed which prevented that entries had been recognized as individual, new entries
- added reloading/updating of status image on status page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4796 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
415b92bb07
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1148&hilit=&p=7711#p7711
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4795 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
2113672bf2
small fix on tag comporator functions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4794 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
dd75b3cabc
- patch for bad profiles
...
- time-out when deleting profiles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4793 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
2080ff72b7
ftpc fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4789 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
e021278bf0
unescape link display in search results
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4788 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
3f1721b827
informational comment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4787 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
04a51b775a
changed .org/.net back to America
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4786 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
74b1a60043
fixed "java.lang.NoClassDefFoundError: org/a"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4784 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
f44d5d302b
updated TLDs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4782 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
hermens
5bfc02ccfb
Repair publishThread
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4781 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f42c8cf69c
updated terminal and dynamic webstructure applet: can now change when crawl is running
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4780 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
906c144799
- design update to new terminal and rssTerminal
...
- added terminal to main menu
- removed transfer size limitation in server
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4779 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7ec01d444a
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4778 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ad0f905124
fix for npe in crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4777 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
ae03a54d23
pdfParser: updated lib, fixed ClassNotFoundException: CMSError
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4776 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
0d3808bd9e
minor refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4775 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
719f5defb1
updated some grafics at new terminal_p
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4774 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lotus
9bc56a9edc
xss protection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4772 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
b32736762c
enhanced rssTerminal
...
- 3 lines possible
- distinguishing of private and public data, if not authorized only public data is shown
- shows now more events, including local searches in clear text if user is logged in
- simplyfied peer events
- better recognition of 'real' new peers
- presentation of peer pings from other peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4771 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
fbb712c669
refactoring:
...
moved importer classes to crawler and plasma package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4770 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1689030ee8
refactoring: moved all crawler classes into their own package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4768 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d2ba1fd2ab
major step forward to network switching (target is easy switch to intranet or other networks .. and back)
...
This change is inspired by the need to see a network connected to the index it creates in a indexing team.
It is not possible to divide the network and the index. Therefore all control files for the network was moved to the network within the INDEX/<network-name> subfolder.
The remaining YACYDB is superfluous and can be deleted.
The yacyDB and yacyNews data structures are now part of plasmaWordIndex. Therefore all methods, using static access to yacySeedDB had to be rewritten. A special problem had been all the port forwarding methods which had been tightly mixed with seed construction. It was not possible to move the port forwarding functions to the place, meaning and usage of plasmaWordIndex. Therefore the port forwarding had been deleted (I guess nobody used it and it can be simulated by methods outside of YaCy).
The mySeed.txt is automatically moved to the current network position. A new effect causes that every network will create a different local seed file, which is ok, since the seed identifies the peer only against the network (it is the purpose of the seed hash to give a peer a location within the DHT).
No other functional change has been made. The next steps to enable network switcing are:
- shift of crawler tables from PLASMADB into the network (crawls are also network-specific)
- possibly shift of plasmaWordIndex code into yacy package (index management is network-specific)
- servlet to switch networks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4765 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d70a472460
added file for previous commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4764 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
8c5f062e0b
corrected YaCy version in HTTP User-Agent
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4762 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d7b21bc90c
re-added gzip POST for transferRWI/URL (HTTP/1.1 compliant)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4761 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d4bce6affd
refactoring (initialized static fields, removed empty if/else, serialized some fields in serializable classes)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4755 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
be2c9c07ff
escape some unescaped characers in URLs (fixes problems with proxy)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4753 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d0678f7ab9
refactoring as result of
...
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=959&p=7560#p7560
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4752 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
483e9a2066
- shifted tld recognition methods from yacyURL to serverDomains
...
- changed isLocal Property in such a way that it is possible to see if a domain is in the internet (and not intranet)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4751 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
a3df23659c
re-implementation of charset checking
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4750 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
75a1702133
- fix for ConcurrentModificationException during shutdown
...
- fix for Ranking distribution problem (suma-lab peer does not exist any more)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4749 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
32b5b057b9
- modified, simplified old kelondroHTCache object; I believe it should be replaced by something completely new
...
- removed tree data type in kelondroHTCache
- added new class kelondroHeap; may be the core for a storage object that will once replace the many-files strategy of kelondroHTCache
- removed compatibility mode in indexRAMRI
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4747 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
ec84a52adb
change for problem with NPE (seen as "PROXY Unknown Error while processing request")
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4745 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
5813cc149f
fix for bad rssTerminal behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4744 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
88216c1f1f
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1103&hilit=&p=7362#p7362
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4743 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d0b893523e
- protection against RAM overflow caused by new peer rss news
...
- more XSS protection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4742 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
685794e7e7
fix for parser/encoding Exception
...
see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111&hilit=&sid=55a320b54e1e3bda9410e7c50b5147f1&p=7431#p7431
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4741 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
cf042e6957
reverted change by mistake in yacyVersion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4740 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
9935e83c86
added new news window into the status page. At this moment it is just a test.
...
The news inside the window are about peer arrivals and departures, remote search accesses and crawls
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4739 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
bac38cfa18
added very rudimentary peer news as rss feed. An example can be retrieved with
...
http://localhost:8080/xml/feed.rss?channel=PEERNEWS
to be extended and integrated in interface ...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4738 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
6495227ad6
the class rssReader is replaced by RSSReader, RSSFeed and RSSMessage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4737 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
724bbdf9b2
refactoring of RSS reader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4736 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
b9a2a2d287
more search performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4735 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
ff755fb858
small corrections and enhancements after search timing profiling
...
search should be a little bit faster now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4734 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
e024e3b9cf
added new default profiles to distinguish snippet fetch for local and global search
...
the difference is, that a local search will no not cause a re-indexing of loaded pages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4731 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
9b03310f8a
bin jetzt wach :/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4729 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7bd8601f04
delete old releases compatible with java 1.5 ;)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4728 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
da386a1924
fixed deleteOldDownloads if there are no downloads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4726 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
21418a22a3
removed DEBUG output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4725 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
79a3edeeef
deleting downloaded releases after x days (default 30)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4724 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
763f9d4f5d
serverCore: setting timeout for new connection before SSLDetect
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4723 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1995faef8d
- refactoring of Colage back-end: move to plasma package
...
- renamed also the plasmaCrawlResults to have a consistent naming for url and image queues
- added a double-check for the images
- added additional queues for the images: all worse-quality images go there, so the queue can be used also if no sizes are given; no image is lost
- added a cleanup for the stacks so they cannot flood the memory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4722 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d7e89c2aca
fixed near-deadlock situation when deleting crawl profiles
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4721 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
5e3ce46339
- better logging when rejecting a url because it is not in declared domain
...
- more XSS attack protection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4720 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
48ffd61e6a
changed "patched wrong" to warning, so it goes to the logfile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4716 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
2f629d20a7
- tried to fix the '4217666-problem'
...
- removed more unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4715 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
512f48e7d6
- removed unused methods
...
- fixed xss attack on peer list in CrawlStartSimple
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4714 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3c76342619
- added servlet to configure the search page greeting line
...
- added information output about the current network definition in the network servlet
- better description and usage of profile entries in User Profile servlet regarding FOAF format
- reformatting of menues at status page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4710 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
d1ee231866
HTTPC close more unused connections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4702 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
181796cffb
- HTTPC ConnectionInfo entfernen bei Exceptions, unnötigen Code entfernt
...
- FTPC (GET-)connections bei Fehlern auf jeden Fall schliessen
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4701 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
04c1226c80
added/fixed missing integrity-test else-case during deploy in case that we update with a tar file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4700 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
45ae3da7e7
another patch to prevent NPE in EcoTable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4698 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
96e39b297a
reduced StackTraces (by connect timed out)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4696 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
93376acdca
fixed a bad chunkcache limit check which could have caused ArrayIndexOutOfBoundsExceptions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4695 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
1cab240198
patch for possible NPE in EcoTable iterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4694 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
9a32a4c328
fixed concurrentModificationException during hello-process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4693 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago