Michael Christen
71649a1296
added an api to retrieve the new citation.index with the
...
webstructure.xml api. This api will respond with details about a single
URL if requested with 'webstructure.xml?about=[url|urlhash|host]'.
13 years ago
Michael Peter Christen
9ad1d8dde2
complete redesign of crawl queue monitoring: do not look at a
...
ready-prepared crawl list but at the stacks of the domains that are
stored for balanced crawling. This affects also the balancer since that
does not need to prepare the pre-selected crawl list for monitoring. As
a effect:
- it is no more possible to see the correct order of next to-be-crawled
links, since that depends on the actual state of the balancer stack the
next time another url is requested for loading
- the balancer works better since the next url can be selected according
to the current situation and not according to a pre-selected order.
13 years ago
Michael Peter Christen
e2f8f263e8
changed storage of search words: keep order
13 years ago
Michael Peter Christen
c166eb68b6
fixes in solr schema file
13 years ago
Lotus
335a776351
xss hardening on Status.html
13 years ago
Michael Peter Christen
ef5192f8c9
using the generic document parser for crawl starts instead of the html
...
parser. This makes it possible that every type of document can be a
crawl start point, not only text documents or html documents. Testet
this with a pdf document.
13 years ago
Michael Peter Christen
ce620be783
for for crawl start with smb url
13 years ago
Michael Peter Christen
7053f8ab46
added automatic generation of a solr schema.xml file
13 years ago
Michael Peter Christen
2ee8cbeb2c
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
source/net/yacy/search/Switchboard.java
13 years ago
Michael Peter Christen
992dbdf4bb
added noload statistic to servlets
13 years ago
Roland 'Quix0r' Haeder
fa08ed5ae5
Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check
13 years ago
Michael Christen
9e5894c784
Removed handling of components objects for URIMetadataRows.
...
This is a preparation to replace this rows with nodes from the node
store.
13 years ago
Michael Christen
c04bfaa51b
refactoring
13 years ago
Michael Christen
e9dc99fe15
added rules to set specific RWIs as private RWIs which are not
...
transmitted to remote peers. This will be used for private index copies
and phonetic indexes.
13 years ago
Michael Christen
204c29f010
small bugfixes for search result display and cache display
13 years ago
apfelmaennchen
ff19fcdb28
bugfix for YMarks XBEL import and export; thanks to Dominic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8138 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
11729061f2
added an option in the bookmark import process to put everything into the crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8134 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
8f30d288e9
small change to mouse over text for crawl starts within bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8131 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
29e97f94f2
small optical enhancements to ymarks treeview
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8106 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
77a080ced9
smaller fixes for YMarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8105 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
e22f8497c9
- tested the ARC methods
...
- removed strict authentication (if password is empty; this was buggy and not useful; can be switched on if necessary globally and not for each interface method)
- increased speed of CrawlResults page (no dns lookup any more)
- increased speed of favicon display (removed dns lookup)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8104 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
5a55397f99
some last-minute performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
dd1482aaf5
further update to YMarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8100 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
564374d1fe
- included YMarks in addition to old bookmarks in yacysearchitem.html; don't get confused by the old bookmark dialog, the ymark is automatically added silently beforehand.
...
- reworked bookmark creation on crawlstart
- many smaller adjustments to ymarks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8072 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
05f34a3fa7
added a full, complete, database insert, update and delete API for the tables.
...
Please see this example:
list all database tables:
http://localhost:8090/api/table_p.xml
now create a new table and insert some values into 'mytable'
http://localhost:8090/api/table_p.xml?table=mytable&pk=&commitrow=&col_termin=Release%20Machen&col_datum=24.11.2011&col_status=ongoing
list the table content:
http://localhost:8090/api/table_p.xml?table=mytable&pk=
update the table and change a single value inside. You must refer to the row using a primary key 'pk'
http://localhost:8090/api/table_p.xml?table=mytable&pk=000000000001&commitrow=&col_datum=29.11.2011
you can also select rows using a search operator
http://localhost:8090/api/table_p.xml?table=mytable&pk=&count=10&search=
now lets delete the row:
http://localhost:8090/api/table_p.xml?table=mytable&pk=&deleterows=pk_000000000001
and we can also delete the complete table:
http://localhost:8090/api/table_p.xml?table=mytable&deletetable=
You can use this to administrate the robots, bookmarks and API steering using an outside application!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8071 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
c461c1eebf
fixed xml output for table retrieval
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8065 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
c93f10417a
add a bookmark automatically each time a new crawl is started
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8063 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
6287c2b4a9
YMarks:
...
- introduced tag manager - a quite powerful tool (still not 100% stable, so be careful)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8060 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
5581be12fb
YMarks:
...
- added backend and api for tag management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8058 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
a3eebfdcba
YMarks:
...
- show active/running crawls
- execute crawls (works currently only if API entry is available)
- various smaller fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8056 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
4f95f72124
YMarks:
...
- working direct importer for YaCy Crawl Starts
- working direct import for old bookmarks.db
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8052 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
017a01714d
- enhanced logging in robots.txt parser for remote debugging
...
- robots.txt is now more robust against database operations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
5a7cec59f3
moved ynetSearch to get all files out of htroot/api/util/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8042 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
a8dfe787ed
- updated to jquery flexigrid 1.1
...
- YMarks.html automatically recognizes if a bookmark is a crawl start
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8040 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
cominch
cef8ebc41d
getpageinfo: Checks if there is a OAI repository behind the URL.
...
This check is only performed if oai parameter is set when calling e.g. getpageinfo_p.xml?actions=oai
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8039 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
eb1c7c041d
write info about robots.txt evaluation into getpageinfo_p.xml
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
f8b8c82421
- refactoring of getpageinfo_p.xml (moved out of util)
...
- added more logging in getpageinfo_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8037 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
abba31f02e
- bugfix for correctly sorting ymarks
...
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
5f7dbe1c42
- some refactoring (ymarks)
...
- improvement for autotagger (is now able to create/detect multi word tags e.g. 'open source')
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
4d7ae76017
- update to jquery 1.7 (does not apply to all jquery code, old version is additionally kept for compatibility)
...
- update to jquery-ui 1.8.16 (includes themes)
- introduced new portalsearch (as default)
- old portalsearch is still available and accessible, but will eventually be removed
- jquery and portal search is now loaded by special header templates for maintenance reasons
- update to new autocomplete, solves bug: http://bugs.yacy.net/view.php?id=29
- many improvements to YMarks GUI and API...more to come anytime soon
Sorry, this is a rather large commit, I hope it doesn't break anything essential, but I need to consolidate some of my efforts in order to move ahead. Especially the update to the portalsearch widget might not be welcomed, but the old one is simply incompatible with newer jquery and jquery-ui libraries, sorry. The code tree /yacy/ui/... is obsolete and will be removed in the future. At that point all productive portalsearches should have migrated to the new version.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8014 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
a7df70221e
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
d2ea250d99
refactoring:
...
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
2d03dc1804
removed unnecessary warning
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7916 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
cf8e3b0df8
small fix for count: overXX includes the count
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7915 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
6db8921a0f
enhanced termlist
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7914 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
sixcooler
d40a177c05
Generation Memory Strategy fine tuning
...
add some log-output in termlist_p
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7904 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
a5541751a8
- added memory computation to termlist_p.xml
...
- added option to delete terms in termlist_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7901 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
9bdee5c71c
added a servlet that produces a list of term hashes that appear more than 10000 times
...
see /api/termlist_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7898 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
sixcooler
916d79111e
Runtime.maxMemory() DOES change @ runtime:
...
I wondered getting Total-ram > Max-ram and MemoryControl.available() < 0
MemoryControl.available() < 0 causes some errors where its value is used for dimension of buffers for eg.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7852 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
9ebc75db4b
fix for channel authorization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7803 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago