fcad2d0744
add uses of config constant INDEX_RECEIVE_ALLOW
9 years ago
226f81cfcf
declare poison pill url MultiProtocolURL() as protected to make sure not
...
used from outside.
After double checking use of poison url revert path init from commit
f8632ad292
9 years ago
f8632ad292
prevent string index out of bounds MultiProtocolURL.getPaths
...
as path maybe a empty string
+ init path to "" also in init for poison url (to guarantee success for
all existing uses of path w/o check for null)
9 years ago
35a7d57260
update lucenematchversion to current (5.2.0 -> 5.5.0)
...
there should be no need for reindex by the update
9 years ago
9b07bbf955
deprecate newurl(), not used and already replaced
...
instead of making it handle all supported the protocols
9 years ago
774b3906a9
fix GenericFormatter.parse ("time","timeoffset")
...
change: UTC offset internally expected in minutes
9 years ago
27163af0e1
improve detection of referenced links by taking http and https link protocol
...
into account
+ correct query start detection of commit f89d4eb51d
9 years ago
f89d4eb51d
fix MultiProtocolURL init (assign of host) for urls with '/' in query part
...
+ add to test case
9 years ago
87fcfc6d78
Adjusted hash computation and toNormalform for file:// protocol to deliver
...
same hash same file on Windows filesystem path with forward- and backslash in path.
Background see http://mantis.tokeek.de/view.php?id=671
+Test case
9 years ago
3811184abd
fix GSA servlet clientIP retrival
9 years ago
7ab41d4ff1
use directories original lastmodified date in file- & smbloader in response
9 years ago
708bcbb042
one more replacement to use cached hosthash vs. calculated
9 years ago
22db449f2a
to prevent crawler to concurrently access and alter same crawl queue
...
after restart, put hosthash in queue's filename (which is used as primary
key for crawl queue. Hint: initial hosthash from url and recalculated hosthash
from just hostname:port are not the same.
fixes http://mantis.tokeek.de/view.php?id=668 (partially)
9 years ago
50c5ddf1a1
Merge pull request #56 from luccioman/LibreJS
...
LibreJS compliance : YaCy JavaScript license information
9 years ago
7466d390b2
small refactoring + do not accept too old peers during bootstrap
9 years ago
8d58a48029
remove wrong log line in CrawlSwitchboard
...
+ don't allow CrawlSwitchboard to exit application
making network param unused
9 years ago
5aaa057c65
ignore empty input lines in FileUtils.getListArray() to poka joke blacklist read.
...
equalizes behavior with getListString()
improves: case were blacklist file contained a undesired empty line, not
fixed by blacklist-cleaner.
9 years ago
41c36ffd75
exclude rejected results from result count
...
(by using the resultcontainer.size instead of input docList.size)
skip waiting for write-search-result-to-local-index
(by removing the Thread.join - which will bring a small performance increase)
9 years ago
d4da4805a8
internal wiki code, require header line to start with markup
...
(to allow something like "one=two" as text)
+ incl. test case
9 years ago
e952e355a2
have Translator servlet adhoc apply added translation by translating a single file
...
+ fix NPE in Translator, coming from translation read by TranslatorXliff
which allows null content for not translated key's
9 years ago
b119ff65be
clean out not used Switchboard variables
...
counter indexedPages, const xstackCrawlSlots
9 years ago
223071337b
Translator to take caution of word boundaries to identify text portion to
...
be translated. To avoid key="TEST" sourcetext="this is a myTESTcase for it"
translation of partial terms/words.
Add check of word boundary before and after sourcetext (incl. take care
of current praxis for key to be delimetered by > <
+ add test case
9 years ago
009657791e
Merge remote-tracking branch 'origin/master' into LibreJS
9 years ago
a73c9327a5
JavaScript License fixes for LibreJS compatibility
9 years ago
0c40401d28
fix MessageBoard test for null data
9 years ago
5b22c63030
Adjust TranslatorXliff to load default 1st and merge downloaded or modified local translation.
...
process 1. load default from locales/*.*
2. load and merge(overwrite) from DATA/LOCALE/*.* (can be partial translation as it is merged)
- include all entries from DATA/LOCAL to be edited in Translator servlet
and save just modifications (instead of full list) to DATA/LOCALE
This shall make it easy to share modifications.
9 years ago
a2e0f00456
optimize Translator
...
- translateFilesRecursive: load translation once (reduce io), return true on complete success
- remove resulting unused translateFiles() variant
- translate: use StringBuilder parameter (skip toString conversion)
- remove not needed static declaration
- upd some javadoc
9 years ago
a6ba1faa80
introduce a translation edit servlet Translator_p.html YaCy's UI text translation
...
This is the 1st rudimentary approach to support the translatio utilities.
It allows currently to edit untranslated text and save it in a local translation file
in the DATA/LOCALE directory.
+ refactor Translator (less static's) to leverage on class overrides and support garbage collection for this 1 time routine
+ adjust TranslatorXliff to check for local translations in DATA/LOCALE,
this includes storing manually downloaded translation files in DATA as well
(to keep default untouched)
+ on 1st call of Translator_p a master tanslation file is generated, checking
the supported languages for missing translation text (later this masterfile is planned to part of the distribution, to harmonize translation key text between the languages)
Outlook: the local modifications (possibly as translation fragments instead of complete file) to be shared with maintainer using xlif features.
9 years ago
b3c9041f79
remove with localHostNames redundant (but unused) publicIPv4HostNames and publicIPv6HostNames
...
to free unused resources
9 years ago
bd8f7c11f5
Use transparent addToCrawler in AutoSearch instead of addToIndex
...
This would likely also be of advantage for RSS import/schedule as
following bug-reports suggest
http://mantis.tokeek.de/view.php?id=569
http://mantis.tokeek.de/view.php?id=655
9 years ago
f23d8ab47b
fix 2 more servlet RuntimeException in intranet mode thrown due to seed.getIP()
...
returning null in intranet mode (in servlets: ConfigSearchBox, Load_PHPBB3
+remove unused (const ∅) seed.IPTYPE
9 years ago
bb0076c3dd
fix: assure close inputstream in TranslatorXliff after reading xlf file
...
by using try-wiht-resource block
9 years ago
6384b7d82e
fix NPE in Load_MediawikiWiki servlet in intranet mode
...
- in intranet mode getip returns null causing a NPE
- adjust starturl (which was set to http://localip/repository ) which is never the start url for the Mediawiki
+ correct javadoc for seed.getIP()
9 years ago
596b5dfa59
add the JRE version in the seed. Purpose: identify if it is possible to
...
migrate to new JRE version
9 years ago
4cc38e979d
add InputStream close after reading input file (Vocabulary_p servlet)
9 years ago
6bf9c55584
adjust Solr select servlet to lates bugfix for boostquery (bq param)
...
to split query into multiple parameter on line separator in input query.
e.g. split "crawldepth_i_0^10.0 \n crawldepth_i:1^5.0"
but allow "url_file_ext_s:jpg OR url_file_ext_s:png" to be unsplitted
9 years ago
9a18e2297b
Merge pull request #51 from JeremyRand/multiple-boost-query
...
Fix multiple boost queries
9 years ago
f0d7b93372
make use and activate autodetect charset in Vocabulary input from file
...
+ revert mistake of empty cn.lng
9 years ago
433217b33e
Properly support multiple Boost Queries. (Previous code was broken because it concatenated multiple Boost Queries together rather than passing Solr an array.)
9 years ago
58824dfa6c
Refactor escaping in config file read/write code. Now it uses Apache Commons StringUtils instead of RegEx.
9 years ago
9e94989237
upd to PDFBox 2.0.1
9 years ago
d0a571bed2
del cytag trail for own index.html (save resource not used by default)
9 years ago
de46879637
fix SeedDB.get(byte[]) hash string compare (for returning own seed shortcut)
9 years ago
24b0fa2a38
extend snapshot Html2Image.pdf2image to use PDFBox image export capability
...
if no external tool installed (and for Win)
Resulting jpg are not always perfect (if graphic included) but imho sufficient.
9 years ago
eb2a00b1d8
fix NPE on missing crawldepth_i
9 years ago
efb9f1a8b7
save resource for unused blacklistFiles map
9 years ago
5f113be760
cleanup connectPeer & yacyVersion.latestRelease usage
...
obsolete since
527b3decde
9 years ago
7097dcbdbd
cleanup hack for partial Solr update on multivalued datefields
...
has been fixed in Solr http://issues.apache.org/jira/browse/SOLR-8050
9 years ago
f10ea3c155
clean-out unused SwitchboardConstants
9 years ago
ef24593347
delete obsolete SEARCHRESULT busythread constants
...
not used since 29.05.2013 18:27:27
0c1a018bbd
9 years ago
reger