orbiter
3f113f38a8
removed unused imports
...
removed unused libs from eclipse class path
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6201 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
8931c8d6b4
improvments to debianpackage:
...
* autoupdate completely disabled, display hint
* restart-button in interface works!
* moved all build-Variables to yacyBuildProperties
* fixed some warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6195 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
caedd72400
- enhanced logging and exception details for parsers
...
- removed inconsistencies in mime type declaration (one mime type should only appear once in all parsers)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6192 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
57a88d435b
redesign of parser mime type detection and parser steering
...
There is now a mime-blacklist instead of a mime-whitelist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6190 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
e15d27bc63
avoiding double/wrong parser errors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6189 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
21b8704fb4
refactoring of the ParserDispatcher and ParserConfig: resulted into Idiom, Parser and Classification classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6188 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8ca1f5d400
- some work to integrate the html parser the same way as the other parsers are integrated (not finished)
...
- added migration of code of settings pages (hmm.. does not work correctly yet, sorry)
- more refactoring
- removed more unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6187 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0e8647d62f
refactoring of search classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6184 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dafffd0153
refactoring of parsers and document processing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6182 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
8041e91f56
*) Ooops!
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6181 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
69551ff3d9
*) added several MIME types (derived from http://filext.com/ ), some of them might be rather uncommon
...
*) added an annotation forgotten in last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6180 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
11dfb2d54f
minor changes:
...
*) added annotations
*) set svn properties and added keywords to comments of parser classes
*) made a variable final to prevent (theoretical case of) change of object instance in synchronized block
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6179 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
77d2a3782c
removed strange debugging strings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6177 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
4320f69574
universal handling for crashed parsers
...
reverting r6090/1
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6176 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
024744245c
small refactoring to prepare for new queues
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6173 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
8544cfd5a6
* remove seperate build-files for parsers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6170 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
995da28c73
all stack/heap files that had been stored in DATA/PLASMA are now stored in the network-specific QUEUES path
...
There is no migration. All crawls must be restarted.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6167 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
48d78166ed
* fix double copy of libraries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6164 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
409538e17a
code cleanup and code simplifcation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6161 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
302a02cec8
moved all libraries from libx to lib
...
removed libx directory
all libraries are now in lib, instead the test libraries in libt which are not part of releases
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6157 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
154bbc3364
code cleanup: call of static methods directly to the class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6155 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
222850414e
simplification of the code: removed unused classes, methods and variables
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6154 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
93dfb51fd4
problems with code style
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6153 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9a674d8047
- After the removal of the Tree class some code simplifications are possible. This affects mostly the Records class, which can be refactored and the result of the refactoring results in a reduced number of classes.
...
- The EcoTable was renamed to Table.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6151 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c5122d6836
completed migration of BLOBTree to BLOBHeaps:
...
- removed migration code
- removed BLOBTree
after the removal of the BLOBTree, a lot of dead code appeared:
- removed dead code that was needed for BLOBTree
Some more classes may have not much use any more after the removal of BLOBTree, but still have some component that are needed elsewhere. Additional Refactoring steps are needed to clean up dependencies and then more code may appear that is unused and can be removed as well.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6150 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
ebe6c823ac
*) changed svn properties agains (hopefully doing it right this time)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6147 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
a80ac3a415
*) fixed wrong parser descriptions
...
*) changed svn properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6146 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
457b6c0d6d
*) updated Apache POI library to be able to parse Visio files
...
*) updated PPT and XLS parsers to use new Apache POI library
*) added new Visio (VSD) parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6145 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
79875782af
be a bit more lazy when removing domain navigation entries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6120 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
57af311627
fix for wrong urls in navigator when a tenant is used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6119 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
91785d895c
*) minor changes in comments
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6109 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
db70badcf0
possibility to set remote host on upnp device
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6097 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
15180fc95e
- patch for future computation in SplitTable
...
- added same concurrent process for has() from SPlitTable in ArrayStack
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6093 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
bf6b92343c
try to avoid stuck pdf parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6091 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
c695c7f512
try to remove hung swf parser from queue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6090 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc69a76197
update to web structure picture:
...
- allow bigger size
- better instructions for api usage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6089 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ae015e8e98
refactoring of blob package classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6088 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8b8877c233
moved image collector
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6087 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
be1c7ddc64
refactoring of search classes -- moved Ranking Profile to search package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6086 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
31f60a3b3e
when doing searches, also apply a online caution to DHT transmission and stop transmissions while heavy load caused by searching. This omits the many requests to the URL database that are needed for DHT transfer and it avoids collisions with URL retrieval needed for search results.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6080 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ce1adf9955
serialized all logging using concurrency:
...
high-performance search query situations as seen in yacy-metager integration showed deadlock situation caused by synchronization effects inside of sun.java code. It appears that the logger is not completely safe against deadlock situations in concurrent calls of the logger. One possible solution would be a outside-synchronization with 'synchronized' statements, but that would further apply blocking on all high-efficient methods that call the logger. It is much better to do a non-blocking hand-over of logging lines and work off log entries with a concurrent log writer. This also disconnects IO operations from logging, which can also cause IO operation when a log is written to a file. This commit not only moves the logger from kelondro to yacy.logging, it also inserts the concurrency methods to realize non-blocking logging.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6078 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
aec3e7995a
autoconfig.pac can be used to browse .yacy-domains only
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6077 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bc6dd8194b
refactoring: moved search query class to new search package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6075 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b8e738a7be
a collection of
...
- small bug fixes
- better/more comments
- more asserts
- fixed synchronization
- test case enhancements
- code cleanup
- performance hacks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6073 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
945777aa80
replaced rwi term counting method by one that computes the maximum of the blobs that contibute to the RWI. An addition of the blob sizes is wrong/incorrect and does not reflect the real size. Truncation the size operation to the maximum of all blobs is also incorrect, but not as wrong as the sum of all blob sizes wich double-counts many rwi entries.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6064 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
28b86385cd
patch for bad behaving swf parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6050 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
733385cdd7
enahnced database access times by removal of unnecessary synchronization.
...
added also more hacks that resulted from high-volum query testing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6047 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2c5554c912
small enhancements in search result computation speed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6039 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e0b3984805
added navigation keys for site and author facets to remote search interface
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6038 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
27fa6a66ad
- completed the author navigation
...
- removed some unused variables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6037 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago