You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

luccioman fcf6b16db4 Added new crawler attribute for finer control over Media Type detection ... New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244		6 years ago
..
content	Small perf improvement : initialize threads names early when possible	7 years ago
importer	…
language	…
parser	Updated pdf cache clear steps consistently with current pdfbox version	7 years ago
AbstractParser.java	…
Condenser.java	…
DateDetection.java	Removed remaining unsafe accesses to SimpleDateFormat instances.	7 years ago
Document.java	Added a crawl filtering possibility on documents Media Type (MIME)	7 years ago
ImageParser.java	…
LargeNumberCache.java	…
LibraryProvider.java	Upgraded the OpenGeoDB dump URL	7 years ago
Parser.java	…
Phrase.java	…
ProbabilisticClassifier.java	…
SentenceReader.java	Reduced memory footprint of text snippet extraction	7 years ago
SnippetExtractor.java	Reduced memory footprint of text snippet extraction	7 years ago
TextParser.java	Added new crawler attribute for finer control over Media Type detection	6 years ago
Tokenizer.java	…
VocabularyScraper.java	…
WordTokenizer.java	Reduced text snippet extraction processing time.	7 years ago