aa9ddf3c23
Added control over Robots.txt active threads maximum number.
...
When starting a crawl from a file containing thousands of links,
configuration setting "crawler.MaxActiveThreads" is effective to prevent
saturating the system with too many outgoing HTTP connections threads
launched by the crawler.
But robots.txt was not affected by this setting and was indefinitely
increasing the number of concurrently loading threads until most ot the
connections timed out.
To improve performance control, added a pool of threads for Robots.txt,
consistently used in its ensureExist() and massCrawlCheck() methods.
The Robots.txt threads pool max size can now be configured in the
/PerformanceQueus_p.html page, or with the new
"robots.txt.MaxActiveThreads" setting, initialized with the same default
value as the crawler.
8 years ago
3092a8ced5
Fixed thread name consistency for improved monitoring.
...
Some tasks were modifying the current thread name without restoring it
once finished as it is effectively done elsewhere.
8 years ago
eec5779889
Added a name prefix to pooled threads for easier monitoring.
...
Using JVM monitoring tools, it is then easier to identify tasks running
inside thread pool with a custom prefix rather than the generic one :
"pool-".
8 years ago
baf6d21cfe
ConfigSearchPage, move protocol navi up to better simulate actual design.
...
Because here btn-group-justified screws up table column width (Explorer
and Firefox) bootstrap btn-group is used.
8 years ago
176f7c2aab
RWI Ranking Configuration page : fixed missing required dependencies.
...
Fix for mantis 707 (http://mantis.tokeek.de/view.php?id=707 )
8 years ago
59130777a6
add high scored items first to YearNavigator (to make sure to be included
...
in sorted view)
8 years ago
0ba5a838f7
Added charset meta to Solr HTML writers.
...
Non-ASCII characters are thus correctly rendered in browsers.
This is a fix ro mantis 706 : http://mantis.tokeek.de/view.php?id=706
8 years ago
08a0acc35d
make a YearNavigator availabel, useable as SearchEvent.naviator plugin.
...
It can take any Date field of the index and displays a list of year strings
in reverse order by the year (not the score/count).
To allow to define the index field to use, the fieldname (and title can be
appended to the navi's name "year" e.g. year:load_date_dt:LoadDate
It works also with dates_in_content_dts field (from the graphical date
navigator). Here the query parameter from: to: are used on selection as
Query modifier (for other dates currently no query parameter available, so
selection won't work to filter search results).
Not included in the UI Searchpage layout config so far (for experiment with
it manual change to conf needed).
8 years ago
7742579ca4
make a LanguageNavigator availabel, useable for the SearchEvent.naviator
...
plugin (not activated yet).
8 years ago
0d3bef659b
implement RequestHeader.setCharacterEncoding for legacy header,
...
make sure .getProtocol returns a http version
move the patch for Set-Cookie to ResponseHeader (applies only here)
8 years ago
5320209963
Merge branch 'master' of https://github.com/yacy/yacy_search_server.git
8 years ago
83f5e3d715
added+disabled a federate search experiment
8 years ago
4eeb448eb3
use DigestURL in UrlProxyServlet as parameter to pass requested url to
...
handler.
UrlProxyServlet splits url in parts to pass it on as parameter and
HeaderFramework constructs a url from param parts. This is obsolete if
already created url is used (makes HeaderFramework.getRequestURL obsolete
= removed)
8 years ago
3630fcc458
adjust messages_p servlet to standard header.getPathInfo, too
...
(replacing non standard http header)
8 years ago
bad8f87998
remove old/obsolete clear text "adminAccount" credential entry from init
...
and setConfig (.,empty) from servlets/code
8 years ago
811cf637f8
fix Jetty9YaCySecurityHandler, length check of Basic credential,
...
add comment to SwitchboardConstants.AdminAccount const
8 years ago
fdcf33f08f
fix Domain.stripToHostName for some IPv6 cases
...
add unit test for it
8 years ago
ac6e198bd1
add unit test for Domains.stripToPort,
...
simplify ipv6 check
8 years ago
f27531f5ec
fix Domains.stripToPort, make ipv6 save
8 years ago
67744a8038
fix HeaderFramework.getRequestURL on host with port considering ipv6 host
8 years ago
66cc0dd173
refactor: move GSA specific date formatter to GSAservlet
...
adjust return type to String for HeaderFrameWork.getSingle
8 years ago
d525967999
refactor: move convertHeaderFromJetty to ProxHandler (only used with active proxy
...
not needed for standard servlets)
8 years ago
20c9b0138e
let User servlet detect static admin with (newer) md5 encoded pwd
...
(complete a old todo)
8 years ago
60ba5c117c
fix legacy getHeaderCookies to work with cookies from original
...
HttpServletRequest, by moving to RequestHeader.
8 years ago
30f8d1e2d7
let RequestHeader.logout succeed w/o throwing exception by default
8 years ago
b449b0b660
remove login request directly after logout,
...
and add logout from servlet container
make logout button red
8 years ago
28afd3a2f8
fix UserDB.proxyAuth from header string
...
(take care of prefix "Basic" in header entry)
8 years ago
fd22d8c08b
Upgraded Bootstrap to 3.3.7 and upgraded its related js dependencies.
...
Upgraded the following JavaScript libraries dependencies :
- bootstrap-switch to 3.3.2
- html5shiv to 3.7.3 and switched to minified version
- typeahead to 0.10.5
- jQuery to 1.12.4
Removed unused bootstratp-rtl.css and bootstrap-rtl.min.css.
Tested non regressions on the following systems :
- Debian Jessie :
- Firefox 45.4.0
- MS Windows 10 :
- Chrome 54.0.2840.99
- Firefox 50.0
- Edge
- Emulated IE 11, 10 and 9
8 years ago
0806de8fdc
Ensure file input stream are closed in both normal and error cases.
8 years ago
a0dfbaca6a
FileUtils : added some JavaDocs and unit test cases
8 years ago
59448461d3
make use of userInRole for quick login verification
8 years ago
2a4d826d9e
adjust servlet RequestHeader.getLocale
...
init jvm defaultLocale matching UI language
8 years ago
200e7a7292
upd to jsoup-1.10.1 : fixed to Eclipse classpath and Ant configuration.
8 years ago
c55172f629
upd to jsoup-1.10.1.jar
8 years ago
9db68acb4f
remove obsolete X_YACY... header declarations
...
not in use (no writes, only remove and try to read).
Obsolete parameter setupHttpClient
8 years ago
f72c99474e
Network Access servlet : render pure relative HTML links.
...
For better YaCy integration behind reverse proxy with subfolder.
Tested on Debian Jessie with an apache2 reverse proxy.
See related mantis issues http://mantis.tokeek.de/view.php?id=106 and
http://mantis.tokeek.de/view.php?id=701
8 years ago
8e9aece786
more use of RequestHeader constant referer, authorization
...
in Jetty9YaCySecurityHandler
8 years ago
d631fbc019
make more use of the new ServletRequest interface methodes
...
getScheme, getServerPort (in QuickCrawlLink_p & YaCyDefaultServlet)
8 years ago
395f2e8946
Make ServletRequest implement the standardized HttpServletRequest interface,
...
to make all readily available information from the original ServletRequest
available to YaCy servlets (without converting data to internal structures).
The implementation of the common interface allows easier integration of
YaCy servlets with the servlet standard (e.g. shared login service with
the servlet container etc.)
8 years ago
62f75417ef
Updated Pattern JavaDoc links to current minimum (1.7) JDK version.
8 years ago
ca4c38a5ba
Updated links to external Java and Solr docs to currently used versions.
8 years ago
f45945cada
increase use of header const for custom "EXT" header
8 years ago
6a0b218ae5
Absolute URLs in yacysearch.* : ensure no downgrade from https to http
...
Also removed unnecessary use of deprecated Seed.getIP().
8 years ago
17fc09036e
fix cutoff text in button and adjust formatting
8 years ago
812abfc868
Converted one more set of URLs to pure relative ones.
...
Easier YaCy peer configuration behind a reverse proxy subfolder : no
need for the reverse proxy to rewrite HTML links or URLs in css files.
Tested on Debian Jessie with an apache2 reverse proxy.
See related mantis issues http://mantis.tokeek.de/view.php?id=106 and
http://mantis.tokeek.de/view.php?id=701
8 years ago
74fec066f4
Converted more URLs to pure relative ones.
...
Easier YaCy peer configuration behind a reverse proxy subfolder : no
need for the reverse proxy to rewrite HTML links or URLs in css files.
Tested on Debian Jessie with an apache2 reverse proxy.
See related mantis issues http://mantis.tokeek.de/view.php?id=106 and
http://mantis.tokeek.de/view.php?id=701
8 years ago
95a60d177e
bring back/fix: search results and logo without external icon
8 years ago
0f0393e5e3
Merge branch 'master' of https://github.com/yacy/yacy_search_server.git
8 years ago
7296e3884f
Switched even more URLs to pure relative ones.
...
Thus a YaCy peer can run behind a reverse proxy subfolder without need
for the reverse proxy to rewrite HTML links (a CPU costly operation).
Tested on Debian Jessie with an apache2 reverse proxy.
See related mantis issues http://mantis.tokeek.de/view.php?id=106 and
http://mantis.tokeek.de/view.php?id=701
8 years ago
49eae79c01
fix Tables.hasIndex check for tablename = key
...
apply same functionality to hasHeap (to not create new table on call hasHeap)
8 years ago
luccioman