ef8cd80593
fix for npe
9 years ago
0347bfa71f
Apply collection query constraint/modifiert to rwi result stack.
...
Collection is not available in pure rwi entries (but in local solr metadata)
But if user wishes to filter by query constraint also rwi shall adhere to this
(even if only rwi entries with parsed or solr received metadata may fit)
9 years ago
2a67d2ba6f
Corrected error management for unsupported image formats, parsing
...
errors, and unavailable resources : avoid logging to much Exceptions as
these errors easily occur when searching images.
9 years ago
d6e9834040
Merge branch 'master' of
...
https://github.com/Scarfmonster/yacy_search_server
# Conflicts:
# .classpath
# build.xml
9 years ago
d82d311995
Merge branch 'master' of https://github.com/luccioman/yacy_search_server
...
# Conflicts:
# .classpath
9 years ago
b5371ea8c1
read/init crawl queue in a thread
...
to speed-up YaCy start on large existing crawler queues
9 years ago
1160b13172
remove unused md5 from ViewFile servlet params
9 years ago
e163ea88f6
fix vsdParser (Visio) parser return statement
...
(final block un-necessary throw)
9 years ago
b2c8bc0ae6
remove md5_s from default index fields
...
it is not assigned a value / not used
Due to above also excluded from transfer protocol.
10 years ago
e40ae0943b
- No max dimensions specified : render raw image data when source and
...
target image format are the same.
- Corrected scaling condition.
10 years ago
90686a75a2
fix flux factor (additional crawl delay by access count) calculation
10 years ago
4af27289e5
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
297fdb60d3
throw exception if crawler hostqueue can't create hostpath directory.
...
In rare cases hostname may not be a valid filesystem directory name,
which can't be created (e.g. containing '*' char). To prevent crawl queue
looping on this invalid entry by throwing a malformedurlexception.
10 years ago
755efac17d
Use same max file size when loading all resource bytes or opening stream
...
content
10 years ago
bc6c79fc12
Corrected scaling function for non RGB images.
10 years ago
1565559df8
Refactoring : extracted write InputStream method.
10 years ago
f0478bb14d
BMP and ICO image formats support : integrated /haraldk/TwelveMonkeys
...
imageio-bmp-3.2 library.
- better BMP format flavours support
- handle PNG encoded icons
- handle transparency
Added some javadoc url references to .classpath
10 years ago
07437986e7
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
97cc03ef6a
start using a template for urlproxy header
...
It is included as iframe /proxmsg/urlproxyheader.html
to allow full servlet functionallity and flexibility to display some
index/meta data in future.
10 years ago
f01d49c37a
Process large or local file images dealing directly with content
...
InputStream.
10 years ago
3c4c77099d
If available, check content length before downloading. Check also
...
content length is not over Integer.MAX_VALUE.
10 years ago
5bbb2e1730
Ensure resource is closed when reading a full file InputStream
10 years ago
6291a57300
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
0d3c5b223e
have psParser cleanup temp file
10 years ago
7d0d19cb8e
avoid File.deleteOnExit() on temp files
...
JVM registers each file in a list regardless of already deleted and never
cleans up the list during runtime.
This accumulates to a considerable amount of mem during large crawls and/or
long uptime.
To tackle this, all temp files are now created in a subdir of java.io.tmpdir
and the jvm tmpdir property is set to this subdir, which is deleted by
code on shutdown.
Additionally let pdfParser use this tmp subdir too.
10 years ago
bfe51001e3
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
02e4489a23
set tmpfile.deleteOnExit by default,
...
to make sure files are removed on shutdown.
10 years ago
2985baaa01
Exclude repetitive protocol part in tokenized url
...
used as description if none is avail. from parser.
10 years ago
ca3d26a401
harmonize wordsintitle & CollectionSchema.title_words_val calculation,
...
remove obsolete partial init of wordreference from urimetadata
10 years ago
52a9040ae6
Sort out double keywords (dc_subject) early in parsed documents
...
- by direct using Set vs. List
- remove not neede String[] getter
10 years ago
49331dc523
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
47d70732f6
improve locale translator
...
- skip empty line
- robustness file section detection (space independant)
10 years ago
646afe9183
do not store subfield *_coordinate + make all num-fields being docvalues
10 years ago
194df613de
not using 'location' as defaultfacetfield - since we removed it being
...
default.
10 years ago
d3b9349b6f
simplification / speedup of GenerationMemoryStrategy
10 years ago
4a905ec134
fix to not let the AccessTracker-Log grow to much, but have enough data
...
to monitor.
(+gitignore-correction)
10 years ago
20e18d79f8
harmonize document title for archive parsers
10 years ago
f11b5e8309
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
112ae013f4
update bzip and bzip parser process,
...
to return one document for the file with combined parser results of the
containing file and registers it with supplied url and mime of the archive.
10 years ago
e76a90837b
update zip and tar parser process,
...
to return one document for the file with combined parser results of the
containing files.
10 years ago
4e673ffc9a
Ensure closing of InputStream even when an exception occurs.
10 years ago
10696b53f7
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
8532565c7d
optimize order of parsers to try
...
- start with a parser matching the remote supplied mime
10 years ago
681889ae64
use current tar library for untar files
...
- remove old source copy
10 years ago
5d71fc70e3
fix tarParser early exit on looping content
...
- adjust check of data available according to doc
- return null on no recognized content (to not exit TextParser next parser try)
- use commons.compress directly
10 years ago
bcc2e7cb5b
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
2fcf6f104c
fix bzipParser recognition
...
- Bzip2Inputstream checks magic byte itself to identify bz2 (leave it in input)
- try to suppy fitting mime for parsing bz2 content
10 years ago
745e97a575
Merge branch 'master' of https://github.com/yacy/yacy_search_server
10 years ago
a60b1fb6c2
differentiate api call getLocalPort() from getConfigInt()
10 years ago
11f3666660
increase use of pre.defined CATCHALL_QUERY string
10 years ago
Michael Peter Christen