You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

luccioman fcf6b16db4 Added new crawler attribute for finer control over Media Type detection ... New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244		6 years ago
..
content	Small perf improvement : initialize threads names early when possible	7 years ago
importer	added a crawl filter based on <div> tag class names	7 years ago
language	Fixed language detector initialization and NullPointerException cases.	8 years ago
parser	Updated pdf cache clear steps consistently with current pdfbox version	7 years ago
AbstractParser.java	added a crawl filter based on <div> tag class names	7 years ago
Condenser.java	Added basic support for autotagging microdata annotated item types.	7 years ago
DateDetection.java	Removed remaining unsafe accesses to SimpleDateFormat instances.	7 years ago
Document.java	Added a crawl filtering possibility on documents Media Type (MIME)	7 years ago
ImageParser.java	BMP and ICO image formats support : integrated /haraldk/TwelveMonkeys	9 years ago
LargeNumberCache.java	Cleaned up some Javadoc warnings.	8 years ago
LibraryProvider.java	Upgraded the OpenGeoDB dump URL	7 years ago
Parser.java	added a crawl filter based on <div> tag class names	7 years ago
Phrase.java	more performance hacks	15 years ago
ProbabilisticClassifier.java	Fixed a NullPointerException case.	8 years ago
SentenceReader.java	Reduced memory footprint of text snippet extraction	7 years ago
SnippetExtractor.java	Reduced memory footprint of text snippet extraction	7 years ago
TextParser.java	Added new crawler attribute for finer control over Media Type detection	6 years ago
Tokenizer.java	Refactoring : documented and extracted autotagging processing functions.	7 years ago
VocabularyScraper.java	added enrichment of synonyms and vocabularies for imported documents	10 years ago
WordTokenizer.java	Reduced text snippet extraction processing time.	7 years ago