f1ori
67da20647f
* add new odf parser based on sax-xml-parser
...
* remove odf_utils-jar
* test metadata in ParserTest
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6231 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
de4f0a006f
removed superfluous windows target
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6230 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
06557485f5
* added parser unittest!
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6229 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
69dfd03985
reactivate unittests
...
* fix old tests
* add buildtarget "ant test"
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6228 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
6d0e6d591b
* ops, fix compiler error :(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6227 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
3e5beb1654
* fix for empty version in seedlist
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6226 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5bb8074150
removed the indexing queue. This queue was superfluous since the introduction of the blocking queues last year, where documents are parsed, analysed and stored in the index with concurrency.
...
- The indexing queue was a historic data structure that was introduced at the very beginning at the project as a part of the switchboard organisation object structure. Without the indexing queue the switchboard queue becomes also superfluous. It has been removed as well.
- Removing the switchboard queue requires that all servlets are called without a opaque generic ('<?>'). That caused that all serlets had to be modified.
- Many servlets displayed the indexing queue or the size of that queue. In the past months the indexer was so fast that mostly the indexing queue appeared empty, so there was no use of it any more. Because the queue has been removed, the display in the servlets had also to be removed.
- The surrogate work task had been a part of the indexing queue control structure. Without the indexing queue the surrogates needed its own task management. That has been integrated here.
- Because the indexing queue had a special queue entry object and properties attached to this object, the propterties had to be moved to the queue entry object which is part of the new indexing queue withing the blocking queue, the Response Object. That object has now also the new properties of the removed indexing queue entry object.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6225 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
597393db3b
changed default visibility of classes/objects in upnp lib
...
(eclipse tells me that this would improve performance,
however, this removes compiler warnings)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6224 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
eea4c17ef2
removed rpm parser
...
- no-one used that thing
- loading huge rpm files bay be causes for crashes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6223 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b332dfad67
- inserted request object into response object which carries this now instead generating new objects
...
- fixed a problem with the crawler introduced in SVN 6216
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6222 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ca72ed7526
-removed superfluous crawl cache
...
-refactoring of crawler classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6221 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8103ccec4c
removed compiler warnings in imported classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6220 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
52e371b8f7
suppress warnings for upnplib code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6219 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
477807e0e6
* updated jxpath to latest v1.3
...
* added upnplib as source
without packages:
jmx
remote
samples
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6218 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
049fb23a8d
removed unused/unsupported ant targets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6217 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
13c63f4082
a set of small fixes to crawling behaviour
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6216 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
a564df3984
update to mime types in parsers and httpd.mime
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6215 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
43c8defd79
enhanced parser with more extension + mime attributes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6214 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
aee35bff6f
replaced StringBuffer with StringBuilder in tar lib
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6213 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
49bbb9bd45
replaced tar library with integrated apache ant tar lib
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6212 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f987fc6b4a
added tar classes from apache ant tools
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6211 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f2d4b6d7fa
added tar classes from apache ant tools
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6210 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b2263bc720
enhanced document type recognition
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6209 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
aa38eb5a20
* maxfilesize -1 for infinite filesize
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6208 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7d493cf8cc
moved parser configuration in separate servelet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6207 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
9cfe89c8fc
* process content-length as soon as it is received
...
* corrected indentation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6206 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5240d22773
removed unused library jsmooth
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6205 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3d26161dd1
removed unused libraries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6204 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
50cf80056f
removed jmimemagic library
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6203 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e3c7f61145
removed unused libraries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6202 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3f113f38a8
removed unused imports
...
removed unused libs from eclipse class path
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6201 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
9f083bb6b2
check filetype before loading (no more mp4 loading)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6200 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
b118bdd994
*) Deleted obsolete license file.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6199 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
076ae02c44
* added pl and py to extensions excepted by htmlParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6198 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d5e51cfd09
* workaround for non-working build property replacements
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6197 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
f814e0fa81
enable warnings and fix most of it
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6196 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
8931c8d6b4
improvments to debianpackage:
...
* autoupdate completely disabled, display hint
* restart-button in interface works!
* moved all build-Variables to yacyBuildProperties
* fixed some warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6195 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
fc1dc38b55
*) added spaces to make sure that no words are concatinated by accident
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6194 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f242e7d7bc
*) using Apache POI library to parse Word documents now
...
*) removed tm-extractors library (can be found at http://www.textmining.org/ if necessary again)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6193 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
caedd72400
- enhanced logging and exception details for parsers
...
- removed inconsistencies in mime type declaration (one mime type should only appear once in all parsers)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6192 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4b74ad0a46
fixed setting of parser configuration servlets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6191 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
57a88d435b
redesign of parser mime type detection and parser steering
...
There is now a mime-blacklist instead of a mime-whitelist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6190 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
e15d27bc63
avoiding double/wrong parser errors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6189 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
21b8704fb4
refactoring of the ParserDispatcher and ParserConfig: resulted into Idiom, Parser and Classification classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6188 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8ca1f5d400
- some work to integrate the html parser the same way as the other parsers are integrated (not finished)
...
- added migration of code of settings pages (hmm.. does not work correctly yet, sorry)
- more refactoring
- removed more unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6187 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
1ee109761f
*) added changes which were lost
...
*) additional annotations
*) additional svn properties
*) _no_ functional changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6186 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
499723891d
removed all non-http daemons; they had not been used and may be a potential security risk.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6185 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0e8647d62f
refactoring of search classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6184 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
92be8f39bf
removed unused packages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6183 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dafffd0153
refactoring of parsers and document processing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6182 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago