fb3032c530
Added a crawl filtering possibility on documents Media Type (MIME)
7 years ago
e45afedee4
Added support for enclosures (media links) to the RSS loader
7 years ago
aaefd5219c
Reduce log verbosity of RSS loader on feed items with no link
7 years ago
cf62b571bd
Added RSS reader support for `enclosure` feed item sub element.
...
Enclosure element (see
http://www.rssboard.org/rss-specification#ltenclosuregtSubelementOfLtitemgt
) can be seen for example in podcasts feeds.
7 years ago
e5f5de0fc7
Added some JavaDoc to the RSSMessage class.
7 years ago
0d7625ecfb
Handle Solr fields restrict and alias in YaCy html and exml writers
...
Thus allowing for example to read more easily the local Solr index full
metadata in HTML by restricting if desired to some fields of interest.
See Solr documentation about the 'fl' (Field List) parameter at
https://lucene.apache.org/solr/guide/6_6/common-query-parameters.html#CommonQueryParameters-Thefl_FieldList_Parameter
7 years ago
3da2739bbd
Parse and index more common audio metadata text tag fields.
7 years ago
846aba00fa
Added parsing of URLs eventually present in audio metadata tags
7 years ago
187075b878
added nav filter
7 years ago
bcbd0ae1a4
Enabled partial parsing of audio resources.
7 years ago
fda0189613
Updated audio file extensions with ones recently added to audioTagParser
7 years ago
978e2be95b
Let a chance for other parsers on audioTagParser error
...
As done in all other parsers, eventually falling back in the end to the
genericParser which creates a minimal index entry.
7 years ago
9e5846a26e
Small fix on svg parser error message
7 years ago
11611dbdcf
Reuse existing File copy function to handle audio parser tmp files
7 years ago
f77f8f40f9
Factored audio parser tag processing
7 years ago
9a7a353d0e
Removed some unnecessary intermediate list creation on array copy.
7 years ago
fb6457f5bc
Fixed NPE case when on audio resource parsed with null tag
7 years ago
c3ff50c17a
Updated the list of audio file formats supported by the audioTagParser
...
Follows upgrade to Jaudiotagger dependency to version 2.2.5.
7 years ago
1b90479a76
Added missing vocabulary navigator increment on results from RWI
7 years ago
46c9da6428
Allow creation of vocabularies from remote CSV file URLs.
7 years ago
17c7a85f18
Make StreamResponse usable in Java try-with-resources statements
7 years ago
b67742336e
Provide user interface messages on vocabulary creation read/write errors
7 years ago
3e8dd90211
Use https rather than http in links and queries to openstreetmap.org
7 years ago
3a973dbb23
Removed unused import
7 years ago
e9527cd0e5
Reuse the same Pattern instance when matching multiple key/values
7 years ago
dbf4c1cd76
Improved blacklist entries editing operations :
...
- Fixes issue #160 : handle properly syntax exceptions with a user
friendly message
- Fixes loss of information on multiple blacklist entries editions
- Fixes loss of entries when moving entries from one list to another
7 years ago
87077b8fb6
Adjust and move Language Navigator to be member of the navigatior plugin
...
list.
7 years ago
eb20589e29
Fixed issue #158 : completed div CSS class ignore in crawl
7 years ago
0cdee4e26a
Fixed loss of "meanCount" search param when using facets or page buttons
...
Then on new search queries, no suggestions at all could be displayed.
7 years ago
117a859879
Do not clear all search modifiers when unselecting one modifier.
...
Previously, when clicking a selected facet in the search results page to
unselect it, all other eventually selected modifiers/facets were also
removed.
7 years ago
33593c22e9
Fixed loss of other modifiers on keywords/tags search navigation links
7 years ago
a9dc0874c0
Remove old query terms from search results suggestions links.
...
Especially when old terms were misspelled, suggestions links then
provided most of the time empty results.
7 years ago
9412881230
Added basic support for autotagging microdata annotated item types.
...
With the appropriate vocabulary settings in Vocabulary_p.html page, this
can produce Vocabulary search facets displaying item types referenced in
html documents by microdata annotation.
Tested notably, but not limited to, vocabulary classes/types defined by
Schema.org and Dublin Core.
7 years ago
5a14d34a7d
Refactoring : documented and extracted autotagging processing functions.
7 years ago
58b9834729
Added HTML microdata typed items parsing capability.
...
This adds the possibility for the HTML parser to gather typed items URLs
annotated in HTML tags with itemscope and itemtype attributes (see
microdata specification https://www.w3.org/TR/microdata/ ), notably
Types from the schema.org vocabulary, but also Types/Classes from any
other vocabulary, such as the common ones listed in the RDFa core
context ( https://www.w3.org/2011/rdfa-context/rdfa-1.1.html ).
7 years ago
80fb1026d0
Create recrawl requests with the relevant crawl profile.
...
Recrawl default profile was previously effectively used for crawl
stacker acceptance check, but request entries were indeed still created
with the "snippetGlobalText" profile.
7 years ago
539925a275
Added an utility to generate/update XLIFF master file from lng files.
7 years ago
fa6d030b0b
Moved dbtest to the test source folder.
7 years ago
6cd3847d0a
Fixed NullPointerException case on Table init with relative file path.
...
Can occur for example when running dbtest with relative test table file
name (wihout explicit parent folder).
7 years ago
28883d8a71
Shutdown daemon threads at the end of dbtest
7 years ago
929e0d6eae
Replaced improper ByteBuffer.equals() implementation by Arrays.equals()
...
Renamed also ByteBuffer.equals() to startsWith() as this is the
appropriate function implementation semantics.
7 years ago
46b5249c20
Removed time condition on HostBalancer initialization in JUnit test.
...
Its initialization in main application usage remains asynchronous.
7 years ago
8b572b7337
Commit Solr index before simulating or starting recrawl job.
...
This ensures up-to-date simulation query results, and recrawl
processing.
7 years ago
733cacdbb8
Revised the RDFaParser main launcher for minimal proper operation.
...
This parser is still not enabled in the main text parsers list. More
would have to be done to make it functional.
7 years ago
7baa99f26f
Fixed stored URL in web cache when redirection(s) occurs.
...
Associate cached content to the last redirection location, instead of
the first URL of a redirection(s) chain :
- for proper base URL processing in parsers (fixes mantis 636 -
http://mantis.tokeek.de/view.php?id=636 )
- to prevent duplicated content in Solr index when recrawling a
redirected URL
7 years ago
9ddf92d143
Removed unncessary reflection usage for workflow tasks.
...
This improves code readability and maintainability (calls hierarchy are
easier to read) and eventually performance.
7 years ago
897d3d30cc
Added new recrawl job profile to the list of default crawl profiles
7 years ago
9624516bf8
Refresh recrawl job profile threshold date like other default profiles
7 years ago
b712a0671e
Added a specific default crawl profile for the recrawl job.
...
- with only light constraint on known indexed documents load date, as it
can already been controlled by the selection query, and the goal of the
job is indeed to recrawl selected documents now
- using the iffresh cache strategy
7 years ago
adf3fa493d
Added comments about crawl profiles recrawl cycles
7 years ago
luccioman