cominch
cef8ebc41d
getpageinfo: Checks if there is a OAI repository behind the URL.
...
This check is only performed if oai parameter is set when calling e.g. getpageinfo_p.xml?actions=oai
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8039 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
eb1c7c041d
write info about robots.txt evaluation into getpageinfo_p.xml
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
f8b8c82421
- refactoring of getpageinfo_p.xml (moved out of util)
...
- added more logging in getpageinfo_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8037 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
abba31f02e
- bugfix for correctly sorting ymarks
...
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
ff32469272
added a link to /api/util/getpageinfo_p.xml as API to crawl start info and to ViewFile.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8035 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
3a15e58e28
- increased stability when opening the robots table
...
- increased stability when deleting tables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8034 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
5f7dbe1c42
- some refactoring (ymarks)
...
- improvement for autotagger (is now able to create/detect multi word tags e.g. 'open source')
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
2f03186252
- small bug fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8030 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
1e55e50c49
- removed unused code from search widget
...
- added more comments for documentation
- ALT key now submits global search
- various smaller bug fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8029 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
f0820a9d02
- more improvements for search widget (portalsearch)
...
- added proper error handling
- greatly increased robustness
- greatly increased usability of navigators
- some smaller speed improvements
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8028 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
78ce3b13be
typo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8027 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
9067ab20b2
- included missing image for portalsearch.tar.gz in build.xml
...
- compressed (minify) yacy-portalsearch.js for better performance
- removed language selector, as it doesn't work really well (at least for me)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8026 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
c7d117505c
- portalsearch.js some fixes for paths, when remote loading method is used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8025 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
a90a72a76b
- some smaller changes to search widget
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8024 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
a425fbd8d6
- created new target 'portalsearch' in build.xml to generate yacy-portalsearch.tar.gz for static hosting
...
- some refactoring for search widget and jquery
- update for ConfigLiveSearch.html to refelct latest changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8023 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
42425c8003
fixed directDocByURL (has now effect if switched off)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8022 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
0d858d48ec
replaced String with StringBuilder in suggestion process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
d871812621
fix for http://bugs.yacy.net/view.php?id=68 as well as for a far more serious bug in navigator handling in the portal search widget. Navigators are now quite usable, but the GUI has still some flaws...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8019 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
3a807e10cf
- added a cache for active crawl profiles to the crawl switchboard
...
- moved the domain cache for domain counter from the crawl switchboard to the crawl profiles. the crawl domain counter is now therefore relative for each crawl start, not for the whole crawler.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8018 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
37e35f2741
normalization of url using urlencoding/decoding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8017 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
e58438c01c
- added a new retry connector for solr (for cases where solr responses are slow)
...
- added a new exist property into the metadataRepository which includes solr entries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8016 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
62e674af50
fix for http://bugs.yacy.net/view.php?id=69
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8015 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
apfelmaennchen
4d7ae76017
- update to jquery 1.7 (does not apply to all jquery code, old version is additionally kept for compatibility)
...
- update to jquery-ui 1.8.16 (includes themes)
- introduced new portalsearch (as default)
- old portalsearch is still available and accessible, but will eventually be removed
- jquery and portal search is now loaded by special header templates for maintenance reasons
- update to new autocomplete, solves bug: http://bugs.yacy.net/view.php?id=29
- many improvements to YMarks GUI and API...more to come anytime soon
Sorry, this is a rather large commit, I hope it doesn't break anything essential, but I need to consolidate some of my efforts in order to move ahead. Especially the update to the portalsearch widget might not be welcomed, but the old one is simply incompatible with newer jquery and jquery-ui libraries, sorry. The code tree /yacy/ui/... is obsolete and will be removed in the future. At that point all productive portalsearches should have migrated to the new version.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8014 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
f121f4bb45
fix for link in Supporter and Suftipps page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8010 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
94eab08794
- updated opensearchdescription text and icon
...
- removed automatic setting of maxitems during search (can be set now elsewhere)
- updated RSSMessage.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8009 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
ba41a869a7
set default number of search results in ConfigPortal.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8008 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
279482a76d
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8007 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
d260b25457
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8006 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
2adc30d335
suppressing size if size unknown
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8005 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
1b86d06d1e
fix for http://bugs.yacy.net/view.php?id=62
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8004 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
low012
1b8b989744
*) set maxlength of input field for country code filter to value > default text length (old value caused warning in Opera)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8002 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
9e4875230f
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8001 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
b5b09b329c
BOOSTED the image search function. The result page now shows the images as embedded image link from the original source and not from the
...
built-in image buffering and re-sizing servlet. The result is shown much faster now not because YaCy does not need to re-size the images but
for a very strange other reason: because of RFC specification (http://tools.ietf.org/html/rfc2616#section-8.1.4 ) a browser does not open more than
two connections to the same server at the same time. If the YaCy image servlet is used, then the target host is the YaCy host for all images
and that prevents a parallel computation of the image loading.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7998 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
a9838f8b99
fix for http://bugs.yacy.net/view.php?id=59
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7997 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
204e98db3a
added a protection against rwi flooding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7993 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
a7df70221e
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
cf4fd525ee
added directDocByURL attribute in crawl profile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7985 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
c61e4cfd78
- fix for incomplete clear() in balancer
...
- renamed Parser Errors to Rejected URLs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7984 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
813f297a95
another performance hack: re-use of known host addresses for isLocal property; avoids look-up in local hash
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7983 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
035ebfbf3b
- performance hacks (should affect the crawl balancer and reduce CPU load during crawl stack re-fill)
...
- this may have also (good) performance side effects on other parts of YaCy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7982 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
9c131adeb6
show IP of crawled host and country in CrawlResults
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7981 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
b250e6466d
implemented crawl restrictions for IP pattern and country lists
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7980 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
5ad7f9612b
added crawl settings for three new filters for each crawl:
...
must-match for IPs (IPs that are known after DNS resolving for each URL in the crawl queue)
must-not-match for IPs
must-match against a list of country codes (allows only loading from hosts that are hostet in given countries)
note: the settings and input environment is there with that commit, but the values are not yet evaluated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7976 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
2c3161b4ac
refactoring:
...
RankingProcess -> RWIProcess
ResultFetcher -> SnippetProcess
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7974 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
d2ea250d99
refactoring:
...
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
fabda9ad31
added script that can be used to delete a single url from the index
...
call:
bin/deleteurl.sh <url>
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7969 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
30d340563e
fix in result count display
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7967 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
orbiter
6b02b696b0
- add number of search results to end of rss and json output to reflect latest status of retrieval
...
- distinguish search access with different verify state in access of search cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7965 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
f1ori
97045022fa
* pass cookies to Server Side Includes
...
* User.html a bit more usable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7963 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago
lotus
6fba6e7cee
fix: follow link target setting on image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7962 6c8d7289-2bf4-0310-a012-ef5d649a1542
13 years ago