c637955e67
fix for navigation steering / p2p mode
...
see also:
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5198&p=29958#p29958
11 years ago
f98ccf952f
Improved Blacklist API:
...
*) added JSON support
*) fixed Exception in case of missing parameters
*) renamed parameter for items in "add entry" and "delete entry" from
"entry" to "item" to match term in XML
11 years ago
91bd384cf6
fix input-group layout on index.html
...
see bug http://mantis.tokeek.de/view.php?id=391
11 years ago
0d88f292dc
Key for parameter "blacklist name" is "list" in all servlets now.
11 years ago
80e0ee92e5
adjust search page layout - search box to current style
11 years ago
a81dfc27eb
remove obsolet css class bookmarkfieldset
11 years ago
0898f0be17
input-group for main search input window
11 years ago
9bb616d778
enhanced HostBrowser buttons and fixed text input alignment
11 years ago
4a818ad72c
fix for strange fail reason
11 years ago
a2fba6584f
use submitted default userAgent if cloning a crawl
11 years ago
e0822fa008
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c97da1a0d8
First draft of a blacklist API.
11 years ago
312972c586
add display filter (active/disabled) to IndexSchema_p.html config
...
for easier overview of schema fields
11 years ago
d79d7dde55
fix for result display
11 years ago
362c988c05
design fixes to better use the new colours
11 years ago
bbadccbd8d
better buttons
11 years ago
a9963d5c95
bootstrap update
11 years ago
4e57000a40
remove redundant javascript & id in index.html
...
to set focus to query field in IE11
11 years ago
121d25be38
recover sax fatal error on OAI-PMH import of xml with entity error
...
this allows to continue loading next resumptionToken even if import file caused sax parser error
fix http://mantis.tokeek.de/view.php?id=63
11 years ago
81dc2aa536
add current css to HTMLResponseWriter to fix metadata view
...
(using css from metas.template except js links)
11 years ago
c6f0bd05f8
better removal of stored urls when doing a crawl start
11 years ago
469e0a62f1
added new button to terminate all crawls
11 years ago
4ee4ba1576
fix for NPE in IndexCreateParserErrors_p.html caused by bad handling of
...
lazy value instantiation of 0-value in crawldepth_i
11 years ago
727dfb5875
refactore URIMetadataNode to further unify interaction with index
...
- URIMetadataNode extending SolrDocument
- use language as stored (String), reducing conversion to string
- optimize debug code in transferIndex
11 years ago
2dabe2009d
- remove unused manual http KeepAlive config
...
(reducing references to obsolete httpdemon)
- add port info to settings_http
11 years ago
10cf8215bd
added crawl depth for failed documents
11 years ago
b4b0d14c04
fix for display bug
11 years ago
9a5ab4e2c1
removed clickdepth_i field and related postprocessing. This information
...
is now available in the crawldepth_i field which is identical to
clickdepth_i because of a specific crawler strategy.
11 years ago
da86f150ab
- added a new Crawler Balancer: HostBalancer and HostQueues:
...
This organizes all urls to be loaded in separate queues for each host.
Each host separates the crawl depth into it's own queue. The primary
rule for urls taken from any queue is, that the crawl depth is minimal.
This produces a crawl depth which is identical to the clickdepth.
Furthermorem the crawl is able to create a much better balancing over
all hosts which is fair to all hosts that are in the queue.
This process will create a very large number of files for wide crawls in
the QUEUES folder: for each host a directory, for each crawl depth a
file inside the directory. A crawl with maxdepth = 4 will be able to
create 10.000s of files. To be able to use that many file readers, it
was necessary to implement a new index data structure which opens the
file only if an access is wanted (OnDemandOpenFileIndex). The usage of
such on-demand file reader shall prevent that the number of file
pointers is over the system limit, which is usually about 10.000 open
files. Some parts of YaCy had to be adopted to handle the crawl depth
number correctly. The logging and the IndexCreateQueues servlet had to
be adopted to show the crawl queues differently, because the host name
is attached to the port on the host to differentiate between http,
https, and ftp services.
11 years ago
dd12dd392f
introduction of a data structure for HyperlinkEdges which should use
...
less memory as it does no double-storage of source links for each edge
of the graph.
11 years ago
a37d067692
refactoring
11 years ago
95780eed32
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
6bd8c6f195
fix for wrong status codes of error pages
11 years ago
9e503b3376
also delete the robots.txt file from the cache when a new crawl is
...
started
11 years ago
67501c9dda
Merge branch 'master' of git@gitorious.org:yacy/rc1.git
11 years ago
1c21b3256d
fix for robots.txt handling: delete old entry before starting a new
...
crawl.
11 years ago
c250fac9f4
linkstructure refactoring to get more options for clickdepth analysis
11 years ago
bd886054cb
new structure and enhancements for link graph computation:
...
- added order option to solr queries to be able to retrieve document
lists in specific order, here: link length
- added HyperlinkEdge class which manages the link structure
- integrated the HyperlinkEdge class into clickdepth computation
- extended the linkstructure.json servlet to show also the clickdepth
and other statistic information
11 years ago
c8d4a63604
eliminating the word 'Facet' from the interface because it is ugly. If
...
people do not know what search navigation is, then they also do not know
what a 'facet' is.
11 years ago
e8ddd415a8
enhanced the new link structure graph
11 years ago
8443255e18
better link structure limit calibration
11 years ago
7f5733638b
fix for linkstructure computation: now also detecting dead links
11 years ago
18f9c40302
moved Edge class out of linkstructure servlet as this does not work on
...
non-eclipse driven environments (all non-dev cases)
11 years ago
a6bb9be97e
- added d3.js for visualizations using embedded svg
...
- added a servlet api/linkstructure.json which generates a link graph
information in json
- added a javascript link graph renderer hypertree.js using d3 and the
new servlet linkstructure.json
- embedded the new link graph in the crawler monitor and the host
browser
11 years ago
c64c10ef00
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
48fbfa60c1
bugfix to inbound/outbound identification
11 years ago
227c42bc96
eleminate obsolete URIMetaDataRow class
...
by joining it with/into URIMetaDataNode.
11 years ago
cca851a417
introduced new solr field crawldepth_i which records the crawl depth of
...
a document. This is the upper limit for the clickdepth_i value which may
be shorter in case that the crawler did not take the shortest path to
the document.
11 years ago
d321b0314e
added missing servlet html
11 years ago
b1ba764d81
fix for first start options and added german translation for popup texts
11 years ago
043d274af5
fixed crawl start path for cloned crawls
11 years ago
1b9ec9a1c5
- added popover to p2p/stealth mode button to explain the peer mode and
...
privacy issues.
- added popover to first-time use case to explain that specific servlets
are only visible after customization and/or crawl starts
11 years ago
8d35fcb1c7
transition.js is also included in bootstrap.js
11 years ago
3abc3c4c4c
removed alert.js, modal.js and tooltip.js as these libraries are all
...
included in bootstrap.js
11 years ago
898f78258e
fix for naming bug
11 years ago
39b641d6cd
added tutorial mode - some menu items will only appear if you 'qualify'
...
for them. Thus, the first-time user will only see four menu items. The
other items will unfold as the user interacts.
11 years ago
7a49f72480
fix for crawler column width
11 years ago
46a1a15441
added more bootstrap libraries
11 years ago
5ccbfeb803
show host list by default in host browser
11 years ago
ba0e3fb0dc
fixed crawl start links after renaming them in latest commit
11 years ago
d29b6db270
made crawl start pages public since they do not reveal individual
...
information and they are also not used as servlet to actually start the
crawl (which is Crawler_p.html).
11 years ago
e41db47cac
added (again) underline to a tags
11 years ago
ff82a80eb3
Integrated HostBrowser back to administration interface; it can appear
...
with and without navigation bar.
11 years ago
94366ba2e5
added template for latest commit
11 years ago
701df02ead
Complete redesign of administration top-level menu. This follows two
...
principles:
- provide an easy tutorial-like "what should I do first" menu
- provide all elements which are subject to most first questions to YaCy
exibition people on top level: Resource limitation, Parser and Ranking
settings
I apologize to everyone who are used to the old style and need to find
the menu items (again) after this change. I hope that this will make the
interface more usable for new users who see a web indexer/crawler the
first time.
11 years ago
a3b7366aee
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
6b66bb7109
redesign of search page integration menu structure
11 years ago
92811d7850
fix: 3 more links pointing to old /xml path
11 years ago
c183d66d40
fix: blacklist xml export path to xml template
11 years ago
656e2ce62a
replacing direct html table cellspacing with css set-up for cellspacing
11 years ago
e11504309f
adding a hint to javascript browser short cut on Url-Proxy page (AugmentedBrowsing_p.html)
11 years ago
7f29eee9ac
fix: cut-off button in WatchWebStructure_p.html
...
(by header css dd hight/line-hight)
11 years ago
2953ebe701
fix: port in local target adress
...
& button style
11 years ago
fda591695c
fixed visibility of custom icon
11 years ago
a9b9950d7f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
bd54b85d46
fix for relative sitemap urls
11 years ago
8b44fcf0f4
added missing @Override annotation
11 years ago
1898175566
fix: button style IndexReIndexMonitor_p.html
11 years ago
bb21d825f9
fix for thread dump line spacing
11 years ago
a7bc130e27
removed performance settings
...
- they are incomplete and buggy
- it was not easy to explain
- it did not comply with a KISS strategy
- setting a performance of low priority actually caused crashing of a
peer
- there was nobody who would maintain that functionality
11 years ago
3290375493
changed the protocol navigator: turned into switch buttons
11 years ago
3272c885ea
fixed auto-activation of search field
11 years ago
cbdfef7ce1
changed protocol facet to show also all other counts if one facet is
...
selected
11 years ago
36e92e898f
moved host facet up
11 years ago
d873304381
added typeahead library to re-implement search suggestions for bootstrap
...
framework.
11 years ago
f686ae30a4
- added (previously removed) autocomplete
...
- removed autocomplete from yacyinteractive.html
- todo: replace old jquery-1.3.2.min.js with latest which is already a
part of bootstrap.css
11 years ago
809b4e1fd9
Team added support for URLs with unicode characters in host part to
...
blacklist. Punycode is used to handle unicode characters.
11 years ago
af82b57a2b
corrected line-height in tagcloud
11 years ago
8424d2b743
fixed search field
11 years ago
b87a94da56
fix for embedded header: the new css framework always need a headline,
...
which is the location for the search input frame. The new embedded
header is a clone of the simpleHeader with removed menus and without a
link to the index administration.
11 years ago
be93bff281
added /date -ranking button
11 years ago
e3cb0ffe16
- added text/image/audio/video/app search option to new navigation bar
...
- changed colors of privacy selector
11 years ago
395837b3b6
from the administration search bar, do global searches
11 years ago
6d17468f21
fix: set text color black in select field in table.th in CrawlResults
...
(default inherite white)
11 years ago
721178dc84
misc style bugfixes
11 years ago
e6dec011ea
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
69d1d067ac
- do not show external link icon on navbar icon
...
- do not show navbar icon on index.html
11 years ago
22618e3ba2
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
936d2ef656
bring api icons to the front
11 years ago
d1091e79f8
- added stealth button to navigation menu
...
- more fixes to progress bar
11 years ago
f0f22e68bb
fix for page navigation bar
11 years ago
deae992d47
fixes to progess bar
11 years ago
e2f2586429
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
c297de5145
remove check for unused virtual path /currentyacypeer/
...
- del jqueryheader.template (not used)
11 years ago
9125bedb87
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
617dd9c97b
- added new input field in index.html
...
- changed progress bar in yacysearch.html
- moved pagination navigation to page bottom
- moved search term input field to headline
11 years ago
c53c52e210
ConfigSearchPage.html first adjustment to new layout
11 years ago
f8f88d4e81
replaced pdblue-homebrew buttons with bootstrap standard buttons
11 years ago
3c8d6e1eee
added adminAccount switch to ConfigAccounts_p servlet to switch on
...
protection of all pages; some refactoring as well
11 years ago
f8e998d660
added bootstrap-switch to turn checkboxes into toggle switches
11 years ago
3d6bcbaa22
reduce number of facets to 10
11 years ago
ed7ad2ef0a
replaced old navbar with bootstrap pagination
11 years ago
a25799f735
removed underline for anchor refs and replaced it with
...
underline-on-hover. Please update your style (switch and switch back)
11 years ago
1fe26550a0
remove AugmentedBrowsing_p.html augmented browsing switch
...
(has no function in code, previously used in conjuction with http://reflect.ws )
11 years ago
381781613b
enhanced api menu entries
11 years ago
5ea5bb3549
redesign of menus; usage of bootstrap buttons and dropdown-menus
11 years ago
21f1d1a7df
added dropdown menu in test dashboard
11 years ago
dc796ebc8d
fixed bold label setting in bootstrap which causes a bad look in YaCy
11 years ago
9d35455f3f
use default header for user front-ends and inverted header for
...
administration front-end
11 years ago
1944df6df1
enlarged facet column slightly
11 years ago
523dde9160
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
ab0e6ba9a6
- removed usage of jquery and yacy/ui in standard search interfaces as
...
possible
- re-implemented formerly jquery-based tagcloud with simple html
- fixed minimized menu for mobile/minimized view (try that: make window
very narrow...)
11 years ago
453bfd0f17
removed unused variables and warnings
11 years ago
dbd7f77568
removed superfluous import of portalheader.template
11 years ago
353d9c523f
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
92655c7fd9
- added bootstrap css framework
...
- adopted all YaCy administration pages to new framework
- created new search page layout (working, but still work in progress)
- old skin files are fully appliable! (and looking good)
- target is a new style based on bootstrap examples, see /test.html
- icons in YaCy may be replaced by glyphicons (to be done)
11 years ago
e972b87a8a
remove AugmentedBrowsingFilters_p.html as none of the settings are used currently
...
config settings frome the page also removed from yacy.init
augmentation.reflect
augmentation.addDoctype
augmentation.reparse
interaction.overlayinteraction.enabled
11 years ago
05655d98df
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
9f02d2c47b
fix: remove link to triplestore in Vocabulary_p (triplestore does not longer exist)
...
- should be investigated in more detail to look for additional implications
Remove "yacyaction" from proxyservlet as it was only needed for removed interaction routines.
11 years ago
e9abb25b03
tried javascript hack to make statistic divs equal height
11 years ago
d052bbdfe1
prevent exception on Site Crawl if no start url is given
11 years ago
85a427ec54
support for multiple sitemaps in robots.txt
11 years ago
b08375da33
fix for bad/missing values of size_i
11 years ago
8cf4d04ae0
fix for facets in search rss result: some facets had been missing, all
...
had missing url properties
11 years ago
51800007c4
- added concurrency to postprocessing of webgraph document
...
- bundeled separate webgraph postprocesing steps into one
11 years ago
351c2be68d
fix: make sure adminAccount changes made via ConfigAccounts_p are effective immediately
...
force to remove current credentials from knownuser cache
11 years ago
5c9dcc269d
improve OAI-PMH import identifier recognition
...
- find best fittng identifier (url) by checking all given dc:identifier in record (many entries proviede several identifiers)
as identifier is currently a multivalued field use "getParams" in preference of splitting the 1st string by ";"
- add resolve DOI:... identifier via http://dx.doi.org/
11 years ago
bcd9dd9e1d
enhanced concurrent loading by using a fixed set of concurrent loader
...
processes in favor of throwaway-processes. The control mechanism does
less often report a 'queue full' message to the busy loop which then
does not perform a long busy waiting; instead all requests are queued
and new loader processes are started if necessary up to a given limit
(as set before)
11 years ago
c7ea0f123b
Added German translation for Config Search and Ranking RWI and Solr
11 years ago
61e51d47a5
fix: unused / incorrect default username parameter
...
(removed setting)
11 years ago
a87aef83b4
Improved Image Collage and WatchWebStructure page display with CSS changes
11 years ago
0813f5f82e
Added German translation for AccessGrid_p.html, CrawlCheck_p.html, RegexTest.html and env/templates/submenuTargetAnalysis.template
11 years ago
a2e5ea2026
status panel link to set max mem
...
+url proxy same error text as in transparent
11 years ago
fdaeac374a
- enhanced postprocessing speed and memory footprint (by using HashMaps
...
instead of TreeMaps)
- enhanced memory footprint of database indexes (by introduction of
optimize calls)
- optimize calls shrink the amount of used memory for index sets if they
are not changed afterwards any more
11 years ago
1245cfeb43
small change to crawler monitor to fit in larger translations
11 years ago
1960aafd6c
better height for statistic windows
11 years ago
b0e3e2100d
better width for Progress table
11 years ago
d2f7ce27a7
fix for switch back to localhost (why did this not work? it worked all
...
the time)
11 years ago
a632b0d2a4
added a forced commit to index deletion to enable synchronized index
...
updates
11 years ago
9c41527e9c
Merge branch 'master' of gitorious.org:yacy/icewindxs-rc1
11 years ago
orbiter