532c7cf827
added physics experiment to the graph plotter. not active by default
13 years ago
aba9b1bfa0
better names for elements of a linked graph
13 years ago
2fc8ecee36
ConcurrentLinkedQueue has a VERY long return time on the .size() method.
...
See
http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html
and the following test programm:
public class QueueLengthTimeTest {
public static long countTest(Queue<Integer> q, int c) {
long t = System.currentTimeMillis();
for (int i = 0; i < c; i++) {
q.add(q.size());
}
return System.currentTimeMillis() - t;
}
public static void main(String[] args) {
int c = 1;
for (int i = 0; i < 100; i++) {
Runtime.getRuntime().gc();
long t1 = countTest(new ArrayBlockingQueue<Integer>(c), c);
Runtime.getRuntime().gc();
long t2 = countTest(new LinkedBlockingQueue<Integer>(), c);
Runtime.getRuntime().gc();
long t3 = countTest(new ConcurrentLinkedQueue<Integer>(),
c);
System.out.println("count = " + c + ": ArrayBlockingQueue =
" + t1 + ", LinkedBlockingQueue = " + t2 + ", ConcurrentLinkedQueue = "
+ t3);
c = c * 2;
}
}
}
13 years ago
8aba045ba1
if a new pop-up page is set in config portal, then this page applies
...
also to the default page configuration for the httpd if no path is
given.
13 years ago
fa7b3481b3
better navigation in file search: less results by first try, but much
...
faster. after the first search is done, buttons appear to get more
results for the same search
13 years ago
8c06925984
animation of the web structure picture
13 years ago
99c74699de
removed scroogle (scroogle is dead)
13 years ago
6e51a00a2f
Revert "fix for page navigation: show only as much pages as are available for given navigation constraints, not as given by total results size"
...
This reverts commit 73f5a9e8b3
13 years ago
73f5a9e8b3
fix for page navigation: show only as much pages as are available for
...
given navigation constraints, not as given by total results size
13 years ago
9c51dc0f13
fixed a bug with navigation: if a navigation was applied to file type or
...
protocol, then it was not possible to remove that again. This is the fix
for that.
13 years ago
8bfc987374
enhanced hint how to enter file:// urls
13 years ago
c6c61be3f0
fix for http://bugs.yacy.net/view.php?id=148
13 years ago
edaa8ac94c
Merge commit 'e15e633a0128b8d31011283a65b4ef26a6dddcd8'
13 years ago
e15e633a01
Bugfix for IE9 (doesn't accept html form within form)
...
changes of API schedule row data changed form input form to unique field names
using row pk.
Fix for issue 96 http://bugs.yacy.net/view.php?id=96
IE9-64bit doesn't interprete iframe with align parameter as desired
misaligns following content (in CrawlProfileEditor_p.html)
13 years ago
a9b4d49b75
removed debug output
13 years ago
8d63a5887c
bugfixes
13 years ago
9ad1d8dde2
complete redesign of crawl queue monitoring: do not look at a
...
ready-prepared crawl list but at the stacks of the domains that are
stored for balanced crawling. This affects also the balancer since that
does not need to prepare the pre-selected crawl list for monitoring. As
a effect:
- it is no more possible to see the correct order of next to-be-crawled
links, since that depends on the actual state of the balancer stack the
next time another url is requested for loading
- the balancer works better since the next url can be selected according
to the current situation and not according to a pre-selected order.
13 years ago
e2f8f263e8
changed storage of search words: keep order
13 years ago
c166eb68b6
fixes in solr schema file
13 years ago
2e5cd6a1b2
fixed parser extension deny list generation and usage
13 years ago
f077b11d38
Merge branch 'master' of git://git.gitorious.org/yacy/rc1.git
13 years ago
29675d9766
more label on search options (usability)
13 years ago
b4bc1e2875
remote search does not do snippet generation
13 years ago
335a776351
xss hardening on Status.html
13 years ago
4c5edab1ec
added option to have exception search result windows
13 years ago
32adad7dd5
show less navigation by default
13 years ago
88b86afc89
no DoS protection for intranet mode
13 years ago
0f443ac755
automatic switching off of navigation that is not useful
13 years ago
852ce43d99
better rules for default open/close of navigation objetcs
13 years ago
e8d24fd802
author navigator can be switched off
13 years ago
558ab7bd4e
made the protocol navigator reversible
13 years ago
96cb75f1d4
made the filetype navigator be able to deselect the search constraint
13 years ago
30891d026f
added a remove-navigation for vocabularies
13 years ago
d5ead5314d
changed navigation links: now using checkboxes.
...
This looks better and allows that negative checkboxes (such that remove
the navigation) are possible. These are not yet implemented (comming
next)
13 years ago
ef5192f8c9
using the generic document parser for crawl starts instead of the html
...
parser. This makes it possible that every type of document can be a
crawl start point, not only text documents or html documents. Testet
this with a pdf document.
13 years ago
ce620be783
for for crawl start with smb url
13 years ago
18227790ca
variable search works at least on Firefox again
13 years ago
c73af39e54
refactoring of tray icon class,
...
now uses Java 6 methods natively
13 years ago
5afec4af39
fix for http://bugs.yacy.net/view.php?id=131
13 years ago
dc165275ad
bugfix for usage of multiple vocabulary navigators
13 years ago
190b77c55e
added Ukrainian translation
13 years ago
7f9b6b7a0c
added switches to ConfigParser to accept/deny documents by their
...
extension
13 years ago
83009d86f7
added the vocabulary navigator. It can be very simply tested by
...
switching on the locale dictionaries.
13 years ago
37d43e5589
jump back to where we want to modify the scheduler in api steering after
...
submit
13 years ago
26e227f17d
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
7053f8ab46
added automatic generation of a solr schema.xml file
13 years ago
c7f88f3fd1
fix for http://bugs.yacy.net/view.php?id=101 - the default crawl
...
depth for bookmarks is now editable.
13 years ago
c602eaaf46
enhanced search process
13 years ago
a71f751cc8
Merge branch 'master' of git://gitorious.org/~apfelmaennchen/yacy/apfelmaennchens-rc1
13 years ago
a94ced7cb2
Solved ( http://bugs.yacy.net/view.php?id=124 ) accessibility issue with
...
page 'Ranking_p.html'
13 years ago
eff966f396
fix for search process (it was aborted too early during remote search)
13 years ago
6d8faa8066
small fix
...
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
1dc5d9f0f3
make ConnectionInfo comparable and sort list of connections in Connections_p
...
ConnectionInfo compare by initTime
Connections_p implement wish to sort connections, descending
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
c1973c78a3
Blog + BlogComments method propose() make similar + fix anonymous poster's IP resolve in Blog
...
this method share very much common code in both classes,
would be nice to unify
+fix some ugly code
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
7189a70a62
BlogComments dateString() - use localized date/time format
...
use same method from Blog class, reduce code duplication
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
f727c35722
Blog dateString() print in localized date/time format
...
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
8318103f25
Blacklist addBlacklistEntry() - dont add an empty entry
...
Signed-off-by: Marek Otahal <markotahal@gmail.com>
13 years ago
972d411d99
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
bd40a10230
added autotaggig stub .. only reading and parsing of vocabularies at
...
this time
13 years ago
f214f6ebb4
added no-load queues to the crawler monitor
13 years ago
2ee8cbeb2c
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/search/Switchboard.java
13 years ago
992dbdf4bb
added noload statistic to servlets
13 years ago
216a287a85
Merge commit '6d4e08ed06c5cd28c45981b2ebe31c7f7ec6fd83' into quix0r
...
Conflicts:
source/de/anomic/crawler/CrawlQueues.java
13 years ago
d18095dc48
Patch fuer Issue 0000102
...
and fixes to Patch (private peer status is a property of a peer, not a
status)
13 years ago
ce289eb64f
remove heuristics if index receive is switched off
13 years ago
fa08ed5ae5
Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check
13 years ago
1d66496362
added more properties to opensearchdescription, see also:
...
http://bugs.yacy.net/view.php?id=114
13 years ago
1cf0f35621
the link to the path shall be the path
13 years ago
87c16fb67f
show server list as server side include
13 years ago
5bfb287753
make a bad fix even worse
13 years ago
4656bb0365
allow html in greeting line since that is set by administrator only.
...
html in that line may be wanted
13 years ago
85bd4cc8bc
better lookup for peer names
13 years ago
20e3084bd4
redesign of fining of peers by ip: more leightweight method to read the
...
seed databases
13 years ago
b09988c9f8
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
c67c30db34
added language selector for russian and chinese in ConfigBasic
13 years ago
9e5894c784
Removed handling of components objects for URIMetadataRows.
...
This is a preparation to replace this rows with nodes from the node
store.
13 years ago
c04bfaa51b
refactoring
13 years ago
e9dc99fe15
added rules to set specific RWIs as private RWIs which are not
...
transmitted to remote peers. This will be used for private index copies
and phonetic indexes.
13 years ago
0bcef2d156
added feature as requested in
...
http://forum.yacy-websuche.de/viewtopic.php?f=18&t=3461
The search can now be configured with a non-display host list.
the search will always exlude the given list of host unless they are
requested directly using the host navigation
13 years ago
66e4832989
work around for overflow in YMarks.html import tab
13 years ago
d6e6f7715b
added "about" box configuration
13 years ago
204c29f010
small bugfixes for search result display and cache display
13 years ago
9aa73a13a8
stop words are on by default in site crawl. This causes normally nothing
...
since the stopwords are empty by default.
13 years ago
17f962fceb
translator updates:
...
- config string for chinese
- do not copy the language file to DATA/LOCALE any more (and do not use
them there, this is really confusing for new translators)
13 years ago
ac5b1e891d
new
13 years ago
391fc9bd57
Merge branch 'master' of gitorious.org:yacy/rc1
13 years ago
23afee58fe
Merge branch 'master' of git://github.com/f1ori/yacy
13 years ago
ff19fcdb28
bugfix for YMarks XBEL import and export; thanks to Dominic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8138 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c715d19c09
fixes for dependency on svn
13 years ago
0d1042363c
showing /16 and /24 subnet option instead of 'bigrange' in network
...
scanner
13 years ago
584f853c4b
added YBR computation process as servlet
13 years ago
864c8d5f05
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
13 years ago
f14faf503b
better ranking because we wait a very little time during the search
...
process more to get better remote sear results into the ranking priority
stack
13 years ago
89dc04115f
fix for stretched network image
13 years ago
2548a9cb04
Fix for http://bugs.yacy.net/view.php?id=93 , Added a link to Bugtracker on YaCy Webinterface.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8137 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
402e9d71ef
changed ording on release files: main criteria is not the svn any more; releases are now ordered by
...
- release number
- date
- svn number
additionally there is a new option to remove the svn number completely
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8135 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
11729061f2
added an option in the bookmark import process to put everything into the crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8134 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
77317a88e0
Added nice jquery tagsinput to bookmarks dialog - similar to delicious.com ;-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8133 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
8f30d288e9
small change to mouse over text for crawl starts within bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8131 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
4b8ff84705
- search bugfixes (page counter and number of results per page; recognition of new search)
...
- experiments to speed-up the network image production (commented out)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8130 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
35a9e8f307
- fixed network graphic
...
- debuged evaluation tables
- changed cache settings in template engine
- some speed hacks
- changed int angles for peer positions in network graphic to double angles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8124 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
6dad227bdf
more...
...
(yes this does not scale unlimited)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8123 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
6c9320e82a
fix for latest navigation feature...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8121 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
80ca673ecd
automatically activate checkbox for set-password if password field is activated to prevent user mistakes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8120 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
9b0879c184
added a hint that the interactive search is only searching in the local index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8116 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
5b2e68b60d
fixed page navigation counter
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8113 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
8fa7f5656e
corrected the changelog link
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8111 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
ebd840ebf6
- enhanced description on search front page
...
- fixed language and heuristic modifier
- added hint to crawl start that we can do also ftp and smb crawls
- added a protocol extension to remote crawls to transport all search modifiers to remote peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8108 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
2512119e5f
added search operator help
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8107 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
29e97f94f2
small optical enhancements to ymarks treeview
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8106 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
77a080ced9
smaller fixes for YMarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8105 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
e22f8497c9
- tested the ARC methods
...
- removed strict authentication (if password is empty; this was buggy and not useful; can be switched on if necessary globally and not for each interface method)
- increased speed of CrawlResults page (no dns lookup any more)
- increased speed of favicon display (removed dns lookup)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8104 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
5a55397f99
some last-minute performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
dd1482aaf5
further update to YMarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8100 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c882c0d421
fixed link to development change log
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8099 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c9216d5adf
fixed secondary remote search (the process that finds distributed join situations)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8098 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
64fd20b857
new default ranking profile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8097 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
0cf9ebc3b0
speed enhancements when parsing RWI rows (makes search slightly faster)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8096 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
da55a359e9
addon to http://bugs.yacy.net/view.php?id=72
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8095 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
ee8b1d4de1
fixed unresolved pattern and unwanted local/global switch when using votes on search results
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8093 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c584db991f
creating a bookmark from the search results now works again .. with new YMarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8092 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
ac5bda205f
- removed lower page navigation (it never looks nice)
...
- added visibility of metadata and parser in search results since that shows what YaCy can do in a nice way
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8091 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c659310e89
- removed option to search for audio, video and applications. These things are still experimental and should not be shown to new users since this would cause them to argue that YaCy does not work. The functions are stil available, because:
...
- added a configuration option in ConfigPortal to swtich the search media types on or off
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8090 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
e799ee55b9
fix for web structure if there is no web structure
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8088 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
6cd27473f5
- better default values for caching and cache usage
...
- set new caching and verification behavior according to use case automatically
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8087 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
1019c36dad
bug fixes and speed enhancements for search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8085 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
507c9d478d
much better timing when search globally; less blocking; more results earlier!
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8084 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
8e0b2c5832
fixed cluster search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8083 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
368b51ed5b
argh.. fixed bad SVN 8080
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8081 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
eb4436defb
removed limitation to cluster peers if peer is asked remotely. This enables single-linked clusters which naturaly is there first if a new cluster is created
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8080 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
d0b4e5fa4d
added grafics for tutorial page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8079 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
7463fafabe
moved the Save button
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8078 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
ccd29e9ee3
moved the help menu point (it is not any more related to search but to YaCy itself)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8077 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
4fa8072b5b
replaced old help page with tutorial movies
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8075 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
5866c73a09
fix for compare search: use scroogle instead of bing and get a default search if configured search engine is not available
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8074 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
9f3ea1d0f4
removed google from compare search because google does not allow embedding of the search page in iframes (it just does not work any more)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8073 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
564374d1fe
- included YMarks in addition to old bookmarks in yacysearchitem.html; don't get confused by the old bookmark dialog, the ymark is automatically added silently beforehand.
...
- reworked bookmark creation on crawlstart
- many smaller adjustments to ymarks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8072 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
05f34a3fa7
added a full, complete, database insert, update and delete API for the tables.
...
Please see this example:
list all database tables:
http://localhost:8090/api/table_p.xml
now create a new table and insert some values into 'mytable'
http://localhost:8090/api/table_p.xml?table=mytable&pk=&commitrow=&col_termin=Release%20Machen&col_datum=24.11.2011&col_status=ongoing
list the table content:
http://localhost:8090/api/table_p.xml?table=mytable&pk=
update the table and change a single value inside. You must refer to the row using a primary key 'pk'
http://localhost:8090/api/table_p.xml?table=mytable&pk=000000000001&commitrow=&col_datum=29.11.2011
you can also select rows using a search operator
http://localhost:8090/api/table_p.xml?table=mytable&pk=&count=10&search=
now lets delete the row:
http://localhost:8090/api/table_p.xml?table=mytable&pk=&deleterows=pk_000000000001
and we can also delete the complete table:
http://localhost:8090/api/table_p.xml?table=mytable&deletetable=
You can use this to administrate the robots, bookmarks and API steering using an outside application!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8071 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
d9d6ed3400
added api link to api page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8068 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
393cdc517e
added api link to YMarks.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8067 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
8e2bb17532
added api icon in table administration which appears automatically if a table was selected
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8066 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c461c1eebf
fixed xml output for table retrieval
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8065 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
d449547023
fix for http://bugs.yacy.net/view.php?id=72
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8064 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c93f10417a
add a bookmark automatically each time a new crawl is started
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8063 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
e4a82ddd8b
produce a bookmark entry from every crawl start. these bookmarks are always private.
...
these bookmarks will be used to get a source reference for the search in case of intranet or portal searches.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8062 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
257c399d29
added a 'fork me' banner to motivate people to get hands on the code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8061 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
6287c2b4a9
YMarks:
...
- introduced tag manager - a quite powerful tool (still not 100% stable, so be careful)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8060 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
5581be12fb
YMarks:
...
- added backend and api for tag management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8058 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
804e48888b
smaller bug fixes for search behavior; should produce less unnecessary removals and an exact number of results as shown in counter
...
should also be a little bit faster
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8057 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
a3eebfdcba
YMarks:
...
- show active/running crawls
- execute crawls (works currently only if API entry is available)
- various smaller fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8056 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c50f8f9a06
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8055 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
84c3fc9d97
local/global fixes in search, better abstraction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8054 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
aca0f33f08
enhancements for extended search options
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8053 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
4f95f72124
YMarks:
...
- working direct importer for YaCy Crawl Starts
- working direct import for old bookmarks.db
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8052 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
a635e43f40
fix for global search attribute when selecting extended search options
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8051 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
017a01714d
- enhanced logging in robots.txt parser for remote debugging
...
- robots.txt is now more robust against database operations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
5a7cec59f3
moved ynetSearch to get all files out of htroot/api/util/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8042 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
a410cfd7f3
- flexigrid images didn't load last time
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8041 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
a8dfe787ed
- updated to jquery flexigrid 1.1
...
- YMarks.html automatically recognizes if a bookmark is a crawl start
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8040 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
cef8ebc41d
getpageinfo: Checks if there is a OAI repository behind the URL.
...
This check is only performed if oai parameter is set when calling e.g. getpageinfo_p.xml?actions=oai
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8039 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
eb1c7c041d
write info about robots.txt evaluation into getpageinfo_p.xml
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
f8b8c82421
- refactoring of getpageinfo_p.xml (moved out of util)
...
- added more logging in getpageinfo_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8037 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
abba31f02e
- bugfix for correctly sorting ymarks
...
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
ff32469272
added a link to /api/util/getpageinfo_p.xml as API to crawl start info and to ViewFile.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8035 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
3a15e58e28
- increased stability when opening the robots table
...
- increased stability when deleting tables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8034 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
5f7dbe1c42
- some refactoring (ymarks)
...
- improvement for autotagger (is now able to create/detect multi word tags e.g. 'open source')
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
2f03186252
- small bug fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8030 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
1e55e50c49
- removed unused code from search widget
...
- added more comments for documentation
- ALT key now submits global search
- various smaller bug fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8029 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
f0820a9d02
- more improvements for search widget (portalsearch)
...
- added proper error handling
- greatly increased robustness
- greatly increased usability of navigators
- some smaller speed improvements
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8028 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
78ce3b13be
typo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8027 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
9067ab20b2
- included missing image for portalsearch.tar.gz in build.xml
...
- compressed (minify) yacy-portalsearch.js for better performance
- removed language selector, as it doesn't work really well (at least for me)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8026 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
c7d117505c
- portalsearch.js some fixes for paths, when remote loading method is used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8025 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
a90a72a76b
- some smaller changes to search widget
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8024 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
a425fbd8d6
- created new target 'portalsearch' in build.xml to generate yacy-portalsearch.tar.gz for static hosting
...
- some refactoring for search widget and jquery
- update for ConfigLiveSearch.html to refelct latest changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8023 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
42425c8003
fixed directDocByURL (has now effect if switched off)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8022 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
0d858d48ec
replaced String with StringBuilder in suggestion process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
d871812621
fix for http://bugs.yacy.net/view.php?id=68 as well as for a far more serious bug in navigator handling in the portal search widget. Navigators are now quite usable, but the GUI has still some flaws...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8019 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
3a807e10cf
- added a cache for active crawl profiles to the crawl switchboard
...
- moved the domain cache for domain counter from the crawl switchboard to the crawl profiles. the crawl domain counter is now therefore relative for each crawl start, not for the whole crawler.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8018 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
37e35f2741
normalization of url using urlencoding/decoding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8017 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
e58438c01c
- added a new retry connector for solr (for cases where solr responses are slow)
...
- added a new exist property into the metadataRepository which includes solr entries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8016 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
62e674af50
fix for http://bugs.yacy.net/view.php?id=69
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8015 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
4d7ae76017
- update to jquery 1.7 (does not apply to all jquery code, old version is additionally kept for compatibility)
...
- update to jquery-ui 1.8.16 (includes themes)
- introduced new portalsearch (as default)
- old portalsearch is still available and accessible, but will eventually be removed
- jquery and portal search is now loaded by special header templates for maintenance reasons
- update to new autocomplete, solves bug: http://bugs.yacy.net/view.php?id=29
- many improvements to YMarks GUI and API...more to come anytime soon
Sorry, this is a rather large commit, I hope it doesn't break anything essential, but I need to consolidate some of my efforts in order to move ahead. Especially the update to the portalsearch widget might not be welcomed, but the old one is simply incompatible with newer jquery and jquery-ui libraries, sorry. The code tree /yacy/ui/... is obsolete and will be removed in the future. At that point all productive portalsearches should have migrated to the new version.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8014 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
f121f4bb45
fix for link in Supporter and Suftipps page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8010 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
94eab08794
- updated opensearchdescription text and icon
...
- removed automatic setting of maxitems during search (can be set now elsewhere)
- updated RSSMessage.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8009 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
ba41a869a7
set default number of search results in ConfigPortal.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8008 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
279482a76d
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8007 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
d260b25457
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8006 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
2adc30d335
suppressing size if size unknown
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8005 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
1b86d06d1e
fix for http://bugs.yacy.net/view.php?id=62
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8004 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
1b8b989744
*) set maxlength of input field for country code filter to value > default text length (old value caused warning in Opera)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8002 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
9e4875230f
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8001 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
b5b09b329c
BOOSTED the image search function. The result page now shows the images as embedded image link from the original source and not from the
...
built-in image buffering and re-sizing servlet. The result is shown much faster now not because YaCy does not need to re-size the images but
for a very strange other reason: because of RFC specification (http://tools.ietf.org/html/rfc2616#section-8.1.4 ) a browser does not open more than
two connections to the same server at the same time. If the YaCy image servlet is used, then the target host is the YaCy host for all images
and that prevents a parallel computation of the image loading.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7998 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
a9838f8b99
fix for http://bugs.yacy.net/view.php?id=59
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7997 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
204e98db3a
added a protection against rwi flooding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7993 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
a7df70221e
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
cf4fd525ee
added directDocByURL attribute in crawl profile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7985 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
c61e4cfd78
- fix for incomplete clear() in balancer
...
- renamed Parser Errors to Rejected URLs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7984 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
813f297a95
another performance hack: re-use of known host addresses for isLocal property; avoids look-up in local hash
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7983 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
035ebfbf3b
- performance hacks (should affect the crawl balancer and reduce CPU load during crawl stack re-fill)
...
- this may have also (good) performance side effects on other parts of YaCy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7982 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
9c131adeb6
show IP of crawled host and country in CrawlResults
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7981 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b250e6466d
implemented crawl restrictions for IP pattern and country lists
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7980 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
5ad7f9612b
added crawl settings for three new filters for each crawl:
...
must-match for IPs (IPs that are known after DNS resolving for each URL in the crawl queue)
must-not-match for IPs
must-match against a list of country codes (allows only loading from hosts that are hostet in given countries)
note: the settings and input environment is there with that commit, but the values are not yet evaluated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7976 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
2c3161b4ac
refactoring:
...
RankingProcess -> RWIProcess
ResultFetcher -> SnippetProcess
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7974 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
d2ea250d99
refactoring:
...
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
fabda9ad31
added script that can be used to delete a single url from the index
...
call:
bin/deleteurl.sh <url>
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7969 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
30d340563e
fix in result count display
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7967 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
6b02b696b0
- add number of search results to end of rss and json output to reflect latest status of retrieval
...
- distinguish search access with different verify state in access of search cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7965 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
97045022fa
* pass cookies to Server Side Includes
...
* User.html a bit more usable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7963 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
6fba6e7cee
fix: follow link target setting on image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7962 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
a6bb0f9af4
fixed missing menu entries in access tracker
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7960 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
734059d33e
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7955 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
ffd848c7a9
moved the log, memory, processes and the messages into a new computation monitor main menu item
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7951 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
ef72fdac79
added keyboard-based search result page navigation:
...
- page-up or tab switches to next search result page
- page-down switches to previous search result page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7950 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
e48ce5d80e
- style change for search box: larger font, selected by default
...
- style change for search results: by default no parser, size, image info
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7949 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
5905392ca3
redesign and simplification of main menu; bundling of some sub-menues
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7948 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
e5a93a1742
fix for image name
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7947 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
5fd4f3fef8
fresh look for yacy icons
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7946 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
95790b82d9
replaced old-style favicon
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7945 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
85a5487d6d
YaCy can now use the solr index to compute text snippets. This makes search result preparation MUCH faster because no document fetching and parsing is necessary any more.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7943 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b0b4886618
try to avoid the unresolved pattern in search result
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7940 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
cec3836e73
added reference limitation to IndexControlRWIs_p.html servlet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7936 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
656286347e
fix for javascript error during search (not ready yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7923 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
0229029dcf
a bit protection against search result bugs in interactive search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7920 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
2d03dc1804
removed unnecessary warning
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7916 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
cf8e3b0df8
small fix for count: overXX includes the count
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7915 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
6db8921a0f
enhanced termlist
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7914 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b5252ef91f
added new word recommendation library in DictionaryLoader_p.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7913 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b00e69c5df
removed test output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7911 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
ce11b7b6d2
*) Changed action to "" instead of "yacysearch.html". This should not do any harm, but helps a lot if the page is accessed not by its original name but by a different name which can be done by adding a symbolic link to the file system of the peer. (See http://www.yacy-forum.org/viewtopic.php?f=2&t=464 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7909 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
c660f8862a
*) changed links to be underlined again since lots of links were not obvious anymore
...
*) added SVN properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7906 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
24e76a7b69
*) Replaced occurrences of "Wikimedia" with "MediaWiki" where applicable. (Thanks to the folks of 0x20.be for pointing this out.)
...
*) Added description of where to place MediaWiki dump for import.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7905 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
d40a177c05
Generation Memory Strategy fine tuning
...
add some log-output in termlist_p
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7904 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
a5541751a8
- added memory computation to termlist_p.xml
...
- added option to delete terms in termlist_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7901 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
9bdee5c71c
added a servlet that produces a list of term hashes that appear more than 10000 times
...
see /api/termlist_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7898 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
5dd2efc9a2
- bugfixes in html parser
...
- new fields in solr
- extended file viewer to debug parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7897 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
2c595a6a47
added new methods to count the number of objects in RWIs. lots of refactoring was necessary to introduce new Rating class and to unify naming of methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7896 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
22d69a6368
refactoring in cora: added sorting package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7890 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
51cf697acd
refactoring: moved all score-related classes to new ranking package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7889 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
4fec99115b
Implementation of strategies for controlling memory resources.
...
You can toggle between previous (standard) and new (generation) strategy at PerformanceMemory_p.html.
The generation memory strategy is implemented with the objective of running more robust
but with the cost of early stopping some tasks (eg. dht) while running low on memory.
This new strategy does respect the generational way a heap is organized on most used jvms.
These changes run fine on my 3 peers for weeks now, but as I'm human, I may fail.
Please be carefull using generation memory strategy and report errors by naming
OS, jvm and java_args.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7886 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
c39d63e7ad
by default show only domain navigator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7884 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
2c58af6874
- added a short memory status simulation mode
...
- added a button in PerformanceMemory_p.html to set the simulated short memory status
- bugfix: added a missing lowercase in KeyList
- better concurrency in loader dispatcher
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7883 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
6361f1d875
select the search window on focus so its easy to type in another query
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7878 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
ca09081341
better interaction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7875 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
3f0349e362
added a 'loading...' message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7874 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
feac494f26
switch off real-time search if index is large
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7873 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f970670a7c
- bugfix in ServerScannerList
...
- speed up of generation of scanner list avoiding forced dns lookup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7871 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
8e03b8ee8b
better integration of server list in interactive search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7870 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
606c5a9b40
added a serlvet that shows all scanned servers inside of the yacyinteractive search page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7869 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
77a9af99f1
same values for Xmx and Xms: memory extension may be difficult if the OS has not the remaining memory available and may kill the jvm. If the memory is reserved at the start but never used the OS may handle that as well and leave non-used space in swap area (and never swap)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7867 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
594d8f546a
#cccamp11 maintenance fix: anons may find up to 1000 items in interactive search (was: 100)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7866 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
3aa6528ed0
the form value was not correctly interpreted
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7864 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
59b767eebd
stop loading via http at defined maximum of bytes - even size is unknown before loading
...
using max-file-size of type int for parsing documents
(since content is used as byte-arrays, 'integer' should be maximum)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7855 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
e131bf804d
correct value @ low memory warning
...
see: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3317
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7854 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
916d79111e
Runtime.maxMemory() DOES change @ runtime:
...
I wondered getting Total-ram > Max-ram and MemoryControl.available() < 0
MemoryControl.available() < 0 causes some errors where its value is used for dimension of buffers for eg.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7852 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
11dc653de3
added a visualization of peer pings to the performance graphic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7837 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
3a191cdf14
because newbies are scared about the memory consumption in the performance graph and arguments about high memory consumption according to bad knowledge about java garbage collection techniques, the memory display had been removed from the performance graph shown on the Status.html page. The memory graph can still be seen on the Performance page where the memory graph is just like it was.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7836 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
768c59740c
- replaced solrj 3.1 with solrj 3.3
...
- updated also slf4j
- added authentication for solrj
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7829 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
c7b95e8c81
*) Invalid crawl profiles (containing invalid mustmatch/mustnotmatch filters) will be moved from active crawls to invalid crawls (new file: DATA/INDEX/freeworld/QUEUES/crawlProfilesInvalid.heap). This file can not be edited yet, but it shoudl be easy to extend the CrawlProfileEditor accordingly.
...
*) Corrupt crawlProfilesPassive.heap would cause crawlProfilesActive.heap to be deleted. Don't know if this ever happend, but will not happen anymore.
*) Cleaned up a little bit.
*) Added some comments.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7827 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
fd02d6d9f8
fixed solr scheme table view
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7825 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
2d4bb139d3
- added counting of links with noindex tag for solr index
...
- bugfixes for solr index
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7820 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
68681a9576
hint for proxy scraping
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7816 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b6f09a475d
- added an index profile editor in the /indexFederated_p.html servlet for solr indexes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7811 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
fdc84d8319
small pi link on index page to administration pages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7804 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
9ebc75db4b
fix for channel authorization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7803 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
6d9e5865ee
faster appearance of search result page (but complete search time is the same)
...
this was inspired by http://bugs.yacy.net/view.php?id=37
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7801 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
4fe1329de2
*) trying to at least fix symptoms of http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3293#p22791
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7799 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
d8072d1866
added more info to DNS cache in /PerformanceMemory_p.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7798 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
4999740790
added new navigation to search trailer json and xml files which causes that these navigation is also available in the search widget
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7796 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
84c9658644
added a file type navigator
...
added a protocol navigator
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7795 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
31283ecd07
- added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search.
...
for example search for "passwd /ftp". This can also be done with /http /https and /smb
- fixed some search throttling processes that should protect your peer against search DoS or strong search load
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7794 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
4b425ffdd2
fix for http://bugs.yacy.net/view.php?id=41
...
added another RSS channel "PROXY". the rss feed for peer news filters this channel if there is not an authorized access on that channel
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7792 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
a65ecffef6
fix for http://bugs.yacy.net/view.php?id=42
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7791 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
07cbb6cb5f
display cache hit/miss values in correct column
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7781 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
115abc8917
- more attributes for search progress bar
...
- moved cache strategy to cora package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7778 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
bce280a308
update on options for interface graphics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7775 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
72a3cd5832
equalize lock icon for Status.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7773 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
2683162ec5
- added more options to access grid picture, web structure picture and network graphics
...
- remove test class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7770 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
7fea51ecee
check filter to bee a correct pattern on edit CrawlProfiles
...
see; http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3277&p=22662#p22660
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7764 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
af63aa1d0e
added fresh links to java regular expression api-doc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7763 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
535b6b953c
more hacks to omit superfluous string object allocation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7757 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
87082f407e
less String object creation during search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7756 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
4bea3f9714
hack to reduce resource contention caused by massive UTF8 decodings which use java.nio resources:
...
used a ASCII String <-> byte[] conversion wherever possible. Many Strings in YaCy are hashes which are pure ASCII (base64 hashes).
The new ASCII String <-> byte[] conversion method have less computation overhead than the UTF8 conversion.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7746 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
e28bd0d038
fix for some possible causes of memory leaks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7741 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
8d9b5dda3b
disabled did-you-mean computation for json and rss search results where this info is not used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7739 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
10e2f588f8
- enhanced ybr ranking computation
...
- many speed/performance hacks
- added solr charding and new charding web interface
- added option to switch off the yacy index when using solr
- added new fail-url categories which are used to make a distinction which fail-urls to be sent to solr
- refactoring/renaming of some method names to distinguish host/url hashes better
- a large number of bug/npe fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7738 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
3ed4a09368
small features, some bug fixes and performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7733 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
3ec94d87c4
show dom counter only for active crawls where the dom counter is enabled within the crawl profile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7731 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b45701d20f
this is a re-implementation of the YaCy Block Rank feature
...
This time it works like this:
- each peer provides its ranking information using the yacy/idx.json servlet
- peers with more than 1 GB ram will load this information from all other peers, combine that into one ranking table and store it locally. This happens during the start-up of the peer concurrently. The new generated file with the ranking information is at DATA/INDEX/<network>/QUEUES/hostIndex.blob
- this index is then computed to generate a new fresh ranking table. Peers which can calculate their own ranking table will do that every start-up to get latest feature updates until the feature is stable
- I computed new ranking tables as part of the distribition and commit it here also
- the YBR feature must be enabled manually by setting the YBR value in the ranking servlet to level 15. A default configuration for that is also in the commit but it does not affect your current installation only fresh peers
- a recursive block rank refinement is implemented but disabled at this point. it needs more testing
Please play around with the ranking settings and see if this helped to make search results better.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7729 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
123375bfba
added a new yacy protocol servlet 'idx'. This returns an index to one of the data entities that is stored in YaCy.
...
This servlet currently only serves for indexes to the web structure hosts. It can be tested by calling
http://localhost:8090/yacy/idx.json?object=host
This yacy protocol servlet is the first one that returns JSON code and that also shows index entries in a readable format. This will make the development of API applications much easier. This is also an example implementation for possible json versions of the other existing YaCy protocol interfaces.
The main purpose of this new feature is to provide a distributed block rank collection feature. Creating a block rank is very difficult if the forward-link data is first collected and then one peer must create a backward-link index. This interface provides already a partial backward index and therefore a collection of all these indexes needs only to be joined which is very easy. The result should be the computation of new block rank tables that all peers can perform.
To reduce load from peers this servlet buffers all data and refreshes it only once in 12 hours. This very slow update cycle is needed because the interface will be called round-robin from all peers once after start-up.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7724 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
d326f1486a
added timeout setting to scanner interface
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7723 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f0d5ddfa92
*) preventing potential NPE which occured if user deleted DATA/RELEASE manually and opened ConfigureUpdate_p.java then
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7722 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
5c981762c6
added bigrange option for network scan
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7721 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
bade61696f
speed-up of network port scanner
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7719 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
b04382bc59
added topmenu as defined for search to wiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7718 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
229df8b626
restart link after memory changed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7717 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
1d8b0f74f4
one more fix for SVN 7713
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7716 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
0960261769
fix for svn 7713
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7715 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
7e368000c8
transparent progress bar
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7714 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
5b579e21a3
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7713 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
fcd4b03892
show progress of search after display of results is finished
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7712 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
Michael Peter Christen