karlchenofhell
6265d321bd
- more constants
...
- display why global search is not available on search page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3839 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
7921f07c9d
userDB fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3837 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
7b2e1bb8f2
Feedparser with reflection.
...
TODO: This needs a special build.xml entry
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3832 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
8bff810d19
- fixed logging output of serverMemory.request()
...
- don't start up if DATA/yacy.running exists as this is usually a sign of an already started yacy-instance
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3831 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
18a5380ee3
*) situation-dependent lock-buttons for search-page
...
*) removed one unused import and a double definition of "ogg" as media-type
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3817 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
f05ca43780
- the wiki-parser works for remote wiki-code now, not displaying links anymore as if they were local (ViewProfile comment)
...
- fixed wrong link to CrawlStart on Status-page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3816 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
9d6605a83c
- fixed NPE in Blacklist Cleaner during deletion of more than one double entries
...
- don't display responseHeader1.db in CacheAdmin_p anymore
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3814 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
594ff95955
:-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3801 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4ca797401e
fix for ConcurrentModificationException
...
see http://www.yacy-forum.de/viewtopic.php?p=36566#36566
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3800 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7b904e0077
integrated robots.txt crawlDelay into the crawl balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3797 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
52cb033f01
- slightly different painting of web structure picture:
...
hosts that have many own connections are painted farer away (this is not yet cato's idea, this will be implemented in another step)
- doc update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3796 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
30c3d909b1
- fixed charset problem in ConfigProfil_p.html (use accept-charset="UTF-8" in forms)
...
- fixed wrong XML output if no peers are known in Network.xml
- simplified parsing of table properties in wikiCode and ZTableToken
- reimplemented GC heuristics. They are needed to constantly ensure that an amount of free memory is available which is higher than Java's max. limit for performing a Full GC (please use serverMemory.request(long, boolean) rather than serverMemory.available(long, boolean) to provide data for averaging over the last GCs)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3793 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
6c9df13552
more debugging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3791 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
4392ee0c51
BugFix for typo and wrong include
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3789 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
d1e1580223
Surftips Blacklist
...
Blacklists List Hardcoded instead of only updated on firststart / migration.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3788 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e1a5babff1
*) Logging GUI handler: line-size is now set to max-size if max-size was exceeded
...
See: http://www.yacy-forum.de/viewtopic.php?p=36355
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3786 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
94cc9f05f5
*) Improvements for restart via update wrapper
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3785 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
44bac7dea1
*) blog-comments can now be moderated
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3778 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
2ab020445a
bugfix, i think - http://www.yacy-forum.de/viewtopic.php?t=4059
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3777 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
f89517203d
*) SOAP: new function to get the Performance Settings of Queues and Processes
...
No items left in the yadmin SOAP-TODO :-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3776 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
92351c4dcb
*) SOAP: bookmarks list now indicates if a bookmark is private (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3775 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
957a25afff
getRight(rightName) instead of get...Right()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3774 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
e0e46d3aec
*) SOAP: new function doGarbageCollection (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3773 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
1efe607c34
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3771 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
a0149317ac
*) fixed bug where headlines were added to directory of a wiki page multiple times ( http://www.yacy-forum.de/viewtopic.php?t=4034 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3762 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
ef24bed406
Sorry...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3760 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
a29cb2e1af
blupp
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3759 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
651b05ba43
*) wsdl file updated (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3758 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0ae6664ad8
enhanced web structure picture
...
- hand-over of get properties from web front-end to graphics generation
- added depth-control buttons
- added marking of anchor-points to highlight relation order
- enhanced ymage graphics library
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3757 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a585b4d41b
added web structure image
...
see http://localhost:8080/WatchWebStructure_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3747 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
33ad0c8246
added a web structure computation and logging:
...
- all web page parsing operations will now increase a web structure file
- the file is computed in memory and dumped at shutdown-time to PLASMASB/webStructure.map in readable form (not a database)
- the file can be used externally to analyse the link structure of the crawled pages
- the web structure can also be retrieved using a xml-interface at http://localhost:8080/xml/webstructure.xml
- the short-term purpose is the computation of a link-graph image (before linuxtag!)
- a long-term purpose could be a decentralized computation of the citation rank
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3746 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
7904175338
- sorry for typos
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3743 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
baa9402b97
- wiki-parser is now configurable via the config setting wikiParser.class which holds the class-name for the parser to use
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3742 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
601fc7d1c5
- added source to J7Zip-modifed.jar and it's license (changelog is still to come)
...
- moved HTML-*replace-methods from wikiCode to de.anomic.data.htmlTools
- prepared use of different wiki parsers as suggested here: http://www.yacy-forum.de/viewtopic.php?p=34444#34444
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3741 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
0a64047081
- plasmaParserDocument can process subdocuments now (other archive-parsers may want to use this method)
...
- added 7zip parser
- added 'text/sgml' to realtime parseable mimetypes (sometimes returned by the mime type parser)
- added new cached output stream class, very suitable for parsers because of limited memory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3740 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b1680ab71f
*) bugfix for ArrayIndexOutOfBoundsException in robots-parser (thanks to low012)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3739 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b30e64daab
*) passing homepath to serverLog.configureLogging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3738 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
9a4375b115
*) robots.txt: adding support for crawl-delay
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3737 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
11ac7688d5
reverted a part of last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3736 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b3f97b5c38
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3735 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3c5ff7f735
adopted kelondroBytesIntMap to kelondroIntBytesMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3734 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5551ff5306
enhanced index storage data structure kelondroBytesIntMap
...
this stores now two index structures, one for data that is aquired during start-up
and one for data that is aquired during run-time. This reduces the grow factor, and should reduce the memory amount in case that a index-reorganisation happens.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3733 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
872eb46cb9
some redesign of the handling of the index for kelondroFlexTable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3732 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
086239da36
- added servlet: remote crawler queue overview
...
- added servlet: crawl profile editor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3731 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
8ba81e0995
- added some comments (will get more in the near future)
...
- added missing <label> to the search field in Network.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3728 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
65a8a9fc58
fix for nullpointer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3726 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b05e2314cf
another dht selection fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3725 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
03c6551b0c
- fix for http://www.yacy-forum.de/viewtopic.php?t=3747
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3724 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b28e5d0ee9
protection against wrong word hash length
...
see http://www.yacy-forum.de/viewtopic.php?p=35657#35657
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3723 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e897eb9b4a
fix for DHT selection target
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3722 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0384b8771b
fix for http://www.yacy-forum.de/viewtopic.php?p=35700#35700
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3719 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
578c2ef130
release 0.52
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3715 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
756a27049b
last-minute-feature 'newbie-selection' for workshop purpose:
...
for remote search, always select all peers that are less than a day old
(should be removed someday in the future if load is too high, which could mean when pph > 100)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3712 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
46367afaaa
update of memory-protection values
...
see http://www.yacy-forum.de/viewtopic.php?p=35539#35539
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3709 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
ea87fe5d78
*) Updated German translation
...
*) Changed "Lost Handle" error to warning (masses of it if deleting crawl-profile)
*) Removed unnecessary code from Windows script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3708 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
85035dc319
addition to svn 3699: check send/receive if p2p-mode is activated
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3701 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
26f05d1fd0
avoid division by zero if search is done for no words
...
this case is relevant if the bluewords (yacy.blue) are used
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3698 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2fa8b50e54
reverting svn 3691+3692
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3696 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
139c59ebbd
- fixed dht selction problem: the seed tables used a wrong ordering
...
- cleaned some code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3693 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
22a0e9f117
more timeout-control
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3692 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
24db55a541
added timeout for httpd-sockets during read
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3691 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f3fee4d445
fix for http://www.yacy-forum.de/viewtopic.php?p=35322
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3689 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7f56c8d4aa
fixed some seed selection details
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3685 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e602436fda
fixed problem with cluster routing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3684 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0831034e07
fixed non-termination bug for robinson remote crawl peer selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3681 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
68f5d64ae6
replaced yacy logo by better version
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3675 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d6480dc670
fix for long transfer pauses
...
see http://www.yacy-forum.de/viewtopic.php?p=35243#35243
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3672 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
cb43ae11ba
*) Bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3668 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
0b5fc3c28c
*) moving date functions to serverDate class
...
*) Sitemap-parser
- logging added
- parsing of modDate added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3667 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
6f46245a51
*) Bookmarks: Ajax icon is displayed while loading title
...
*) First version of a sitemap parser added
- currently only autodetection of sitemap files is supported
*) DB-Import restructured
- pause/resume should work again now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3666 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
269d5ca45b
*) Bugfix for IllegalMonitorException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3664 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e8610f049a
*) new method allowing the updater to wait until yacy has finished startup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3662 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
111ba9e359
- fixed some width problems in new status page
...
- fixed deadlock in dns cache
- added termination security for DHT peer selection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3660 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
74dd6cac95
*) signal yacy shutdown to updater
...
*) some javadoc added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3658 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
43748f87fb
*) changes required for the uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3655 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
e12e934ade
*) Fixed broken compile process.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3650 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
d6811ac243
*) Moving tar.jar from libx to lib
...
*) Enhanced interface
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3649 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
469583ea80
*) new interface class. should be implemented by the updater to allow communication between the updater and yacy
...
(not yet functional)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3648 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
191ef16499
fixed wrong ordering that caused bad dht selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3646 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7cf8981a98
- added debugging code for wrong DHT target iterator
...
- restricted distance constraint from 0.4 to 0.2
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3644 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
abb63e3289
*) disabled system.exit() in case of YaCy shutdown as it kills the whole VM (including updater and other management threads)
...
*) UpdateCheck-Thread now pauses for given interval correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3642 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dd44a1394f
disabled automatic performance setting change
...
- during crawl start
- each indexing cycle
- for delay values
- for short memory cycles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3634 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b9add5cf37
some bugfixes:
...
- dht iterator start point
- wordIndex synchronization
- surftipps url check
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3633 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
06b6e35484
fix for a null pointer exception if clusters are not defined
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3632 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
35c660654d
more debugging lines to fix bug for
...
http://www.yacy-forum.de/viewtopic.php?p=34935#34935
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3629 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
47e90f31b2
fix for deadlock in plasmaWordIndex.addPageIndex
...
synchronization for class method not necessary
see also: http://www.yacy-forum.de/viewtopic.php?p=34959#34959
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3628 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
81844e85b2
- fixed more cluster routing problems
...
- fixed a problem in remote search when balancer caused shift process to wait too long
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3627 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
304ed3f4d2
fix for remote crawl requests in clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3626 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1979a167d3
fixed problem with cast
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3625 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e48189c710
enhanced cluster routing
...
- cluster definitions can now contain an addition for local ip addresses
- cluster-cluster communication uses the local ip address instead the global address, if one is given
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3624 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e9d87b2fce
*) changes required for the uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3621 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b33cef421e
better routing for public clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3620 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
7c902996b5
*) changes required for the uploaderWrapper
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3618 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f73e1e3af9
fixed bugs in remote search setting for public clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3615 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
485bf1ea83
bugfix for robinson/remote crawl bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3614 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
62c947b4aa
next try to fix deadlock in plasmaWordIndex
...
see also:
http://www.yacy-forum.de/viewtopic.php?p=34821#34821
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3607 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
871ee1ce0f
one step closer to automatic updates:
...
automatically aquire release information from download archives
web pages from latest.yacy-forum.net and yacy.net are retrieved, parsed,
links wihin are analysed, sorted and the most recent developer and main
releases are provided as direct download link on the status page, if it was
discovered that a more recent version than the current version is available.
This process is done only once during run-time of a peer, to protect our
download archives from DoS by YaCy peers.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3606 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ec225f9ab6
*) SOAP: adding methods to get the comment and MD5 checksum of a single file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3604 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
2399ed817c
*) robots.txt parser now extracts the sitemap-URL (will be used later)
...
*) some javadoc added
*) junit testclass for robots.txt parser added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3602 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
fa012789b2
tried to fix a deadlock problem durin shutdown
...
see also:
http://www.yacy-forum.de/viewtopic.php?p=34753#34753
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3601 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e192f616a2
collection of small bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3600 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
64a6d6e5e6
added new set iterator (needed for last commit)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3599 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f8de19fb2f
robinson cluster: added client-side protocol implementation
...
- the network configuration page shows a new option: robinson clusters
- when a global search is made, all robinson peers are excluded, but:
- robinson peers/clusters that provide peer tags and where search words match
such tags, they are included in global search. Therefore, robinson peers/clusters
support the global yacy network with their indexes, without doin DHT-exchange
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3598 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
50e1e61fa5
*) SOAP: adding functions to rename and move files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3595 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
63a004abff
*) bugfix for Nullpointerexception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3594 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
29fe2beac7
possibly fixed a deadlock
...
cannot find forum link now for that
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3593 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
4f4d3d71dd
*) Faster appearance of ConfigBasic by bypassing UPNP-scan in case of existing external connects
...
*) Marked two deprecated source-points
*) Added possibility to dump words from indexing to file. Should not affect performance in the current form.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3592 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
657585fe0d
network functions for robinson peers: server-side protection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3591 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
89c1511738
- added new Network Configuration menu, can be found in basic settings
...
- new cluster functions will be available in this menu, but currently not enabled,
because corresponding interface methods are not ready yet
- shifted remote crawl settings to new network configuration menu
- shifted DHT distribution/receive to the new network configuration menu
- adopted some string constants
- added cluster configuration settings to yacy.init
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3589 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
62b79aa0a9
bugfix for http://www.yacy-forum.de/viewtopic.php?p=34558#34558
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3586 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2f3b518169
temporary patch for startup-problem:
...
http://www.yacy-forum.de/viewtopic.php?t=3854
This is a serious problem that is caused by the database bug between 0.511 - 0.513
which produced a large number of double-entries in the RWI index. The uniq()-method
tries to fix this, and it does not terminate when the index is large and the number
of double-occurrences is also large. This patch does simply implement a time-controlled
termination, which does not heal the inconsistency problem. The uniq-method itself
is correct and does not need a bugfix, the non-termination is simply caused by the large number
of data that is shifted during the process. It was possible to reproduce this behaviour
in a test environment.
A real fix would need to:
- enhance the uniq()-method by using a recursive, binary segmentation of the array to be fixed
- uniq() must report the entries that are double
- the double-entries must be deleted from the collection index (from the index and the collections) to heal the problem
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3583 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
e6fb6426a3
*) Some cosmetical changes and corrections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3582 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ba525ebf52
- re-enabled path optimization that was disabled during testing
...
- re-implemented index load/extend optimization that was removed from kelondroFlexTable,
this is now part of kelondroIntBytesIndex
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3580 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
595ee10468
fixed datatabase inconsistency bugs
...
inserted many debug lines
added a huge number of asserts
extended database test methods
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3579 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ca79362b9d
disabling auto-setting of remote crawl performance
...
see also http://www.yacy-forum.de/viewtopic.php?t=3849
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3577 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7a7a1c7c29
fight against problems with remove-methods and synchronization
...
- some bugs may have been fixed with wrong removal operations
- removed temporary storage of remove-positions and replaced by direct deletions
- changed synchronization
- added many assets
- modified dbtest to also test remove during threaded stresstest
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3576 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b6a5f53020
removed double synchronization from kelondroRecords.USAGE to prevent thread locking.
...
The method synchronization should be sufficient
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3574 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
6186185775
*) Moved some comments to javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3573 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
063063aa0c
fix for 100% cpu bug during dht selection
...
see also: http://www.yacy-forum.de/viewtopic.php?p=34068#34068
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3570 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
michitux
4990909178
Some bugfixes, new layout/style for image search results:
...
* removed divide by zero bug when 20_dhtdistribution_busysleep is 0
* replaced German comment with wrong charset in source/de/anomic/plasma/plasmaCrawlBalancer.java by an English one
* replaced the table-fix for floating behind snipped images by a br with clear
* removed unnecessary old xhtml-files (were not in use, they were created when we weren't having xhtml for testing)
* new layout for image-search results: replaced the old one with spans and tables inside (not valid) with new divs, now each image snippet container has the same size
TODO:
* the ids of the snippetLoading-divs aren't valid because ids must start with an alphabetic letter or an underscore, they have to be prefixed
* in the returned snippet-xml is an unresolved pattern for status (the status is only set for text snippets)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3566 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
78d04bcbcf
fixed bug in search statistics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3562 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b79b4082e2
completed search exclusion:
...
- exclusion on index-level (not only from search snippets)
- exclusion hand-over at remote search protocol
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3556 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
d66b0276e3
*) removed log-output for PPM-calc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3553 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
4400acc27d
*) created new 8 bit oldschool style font for possible future use
...
*) main method is generalization of main method of ymageFontGenerator:
it does not matter how many lines of how many bits a font is made of
as long as the values stay the same within the font -> use this class as
a template for your own font generators and be a happy camper
*) main method checks if font is valid (96 characters, all letters must have
same number of lines and same number of bits per line)
*) ***** I have not checked if the result is really a valid font so far. *****
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3552 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
25070822a5
fix for http://www.yacy-forum.de/viewtopic.php?p=33925#33925
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3551 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
06a7978730
moved url pattern matching for search to better place
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3550 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
159bd0cab5
diverses; b.o. fix for http://www.yacy-forum.de/viewtopic.php?p=33914#33914
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3549 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
cdc7b77a62
fix for http://www.yacy-forum.de/viewtopic.php?p=33916#33916
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3548 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
40c14a4f0e
- better implementation of search query properties
...
- basic protection against start-up problems when database files are corrupted
- auto-delete of not-critical databases during startup when load error occurs
- on-the-fly reset option for all database tables
- automatic on-the-fly reset for seed tables during enumeration exceptions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3547 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1696606b7f
*) changing loglevel of "PPM-Calculation" message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3545 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
f30bf1683e
*) corrected spelling of captcha
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3544 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
fcdf000fbc
bugfix for http://www.yacy-forum.de/viewtopic.php?p=33838#33838
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3543 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
d7edc9740b
*) added correct (c) and Last*-data
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3542 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
ee241f32e6
*) very basic capcha class (see coding sections of forum for more details)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3541 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6e7340ef52
added exclusion search
...
(you can now search and exclude words from the result with '-')
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3540 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e4734a8b6b
fix for fix in SVN 3537
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3539 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
356033aceb
fixed bug with continuous reset of balancer file index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3537 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ba2c307ab3
optimized memory allocation in kelondroRow.Entry
...
such an entry cannot be instantiated without allocation of new byte[]; instead
it can re-use memory from other kelondroRow.Entry objects.
during bugfixing also other bugs may have been solved, maybe the INCONSISTENCY problem
could have been solved. One cause can be missing synchronization during bulk storage
when a R/W-path optimization is done. To test this case, the optimization is currently
switched off.
More memory enhancements can be done after this initial change to the allocation scheme.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3536 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
24ea4ca631
*) adding first version of postscript parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3535 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
def0d6124e
*) trying to solve SecurityManager problem during init of soap engine
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3534 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
75eb65028a
*) adding a test if a seucrity manager is active
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3533 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
210ede8230
added a class for byte-array management. This was the result of a very large experiment
...
to replace byte[] objects within kelondro. Frequent System.arraycopy are common when
kelondroRow.Entry objects are handled. This class may be used to prevent this.
However, experimental replacement of byte[] by kelondroByteArray in kelondroRow.Entry
resulted in complete re-write of large parts of kelondro. This experiment did not
completely lead to a result, because then the interface to kelondro had to be changed
also from byte[] to kelondroByteArray, which may have caused a rewrite of large parts
of YaCy. The experiment is therefore abanonded, but this class remains here without
any function but possibly for future use.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3531 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1b7fda12ee
*) SOAP: separate function to get the active/passive/potential peer list
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3526 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6488ec8a80
no deletions in index in case that snippet-loading fails and there is no network connection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3525 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
847349358b
less memory usage during collectionIndex-rebuild
...
should also speed up that process a little bit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3524 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
8ef3ad12a7
*) fix for rare bug in PPM-calc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3523 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
00bc0c1b47
*) new logging for PPM-Calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3522 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
5941577076
*) added some logging to PPM-Calculation to find a rare bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3521 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5c3afb3202
added option to configure a path to a secondary index location.
...
this shall be used to store a fragment of the index on another physical device,
to split IO load and enhance access speed. The index is splitted in such a way
that the LURLs are stored to the secondary location, and the RWIs to the primary
location. This is especially useful for environments where symbolic links are
not possible and may cause IO access even if there is no write access to the
device which hosts the symbolic link.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3519 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
c2e6afbd69
*) bugfix: setting mimeType properly for dir listing with e.g. "?format=xml"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3516 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
242c19b480
completed TLD categorization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3515 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
b99f9d870d
*) fixed double selection of peers for the same DHT-chunk.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3513 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
f20b596dc0
*) adding servlet to display all deployed SOAP Services
...
- soap related servlets are located in htroot/soap
*) new serverContext class for soap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3511 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
75d90834a2
*) adding additional file extension for powerpoint
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3507 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2cb16824e3
removed support for old database structures.
...
The new collection index will be more generalized to support other indexes
i.e. YBR block-rank computation. A clean-up of the many conditions to support
the old database was necessary.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3506 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
81b4598487
*) peer profile can now be displayed as vcard
...
e.g. http://localhost:8080/ViewProfile.vcf?hash=localhash
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3504 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3688ec33e5
release 0.51
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3501 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1f61c13697
*) RSS-parser extracts the author tags now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3500 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
602ac42010
fix for OOM case when a kelondroTree Node cache grows
...
See also: http://www.yacy-forum.de/viewtopic.php?p=33275#33275
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3499 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b374812f01
*) adding rpm packager as author
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3498 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
beb772d6cd
fixed problem with broken notifier image, occurred only at initial start-up
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3497 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
40ce33e664
*) adding RSS feed for yacy news
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3496 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
589cbd8cbf
*) replacing all yacy-news-category strings with corresponding constants
...
Note: please use these constants from now on
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3495 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
f4af360f7c
bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3494 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7af188ff9a
fix for http://www.yacy-forum.de/viewtopic.php?p=33089#33089
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3491 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5bbf010107
removed synchronization of size() method from numerous classes to avoid thread locking
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3490 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6b9eea3932
- removed differentiation between longTitle and shortTitle; this cannot be used for search results,
...
and it is difficult to get both types from all document types
- added some author parsing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3489 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a738b57b31
added author tag to indexing content
...
enhanced composition of title tag
TODO: insert author information for external parsers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3488 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6be57983a8
another update to the crawl balancer
...
can now alternate between top and bottom of the crawl stack
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3487 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
91cdc1493f
removed query to NAT or responder in case that no other peer is there.
...
this is not needed any more, there are enough peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3486 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4783a30910
- fixed a flush problem in balancer
...
- return to idle divisor in RWI RAM cache flush
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3485 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
91c2a042a7
*) bugfix for wrong proxy traffic accounting
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3484 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
861f41e67e
redesigned NURL-handling:
...
- the general NURL-index for all crawl stack types was splitted into separate indexes for these stacks
- the new NURL-index is managed by the crawl balancer
- the crawl balancer does not need an internal index any more, it is replaced by the NURL-index
- the NURL.Entry was generalized and is now a new class plasmaCrawlEntry
- the new class plasmaCrawlEntry replaces also the preNURL.Entry class, and will also replace the switchboardEntry class in the future
- the new class plasmaCrawlEntry is more accurate for date entries (holds milliseconds) and can contain larger 'name' entries (anchor tag names)
- the EURL object was replaced by a new ZURL object, which is a container for the plasmaCrawlEntry and some tracking information
- the EURL index is now filled with ZURL objects
- a new index delegatedURL holds ZURL objects about plasmaCrawlEntry obects to track which url is handed over to other peers
- redesigned handling of plasmaCrawlEntry - handover, because there is no need any more to convert one entry object into another
- found and fixed numerous bugs in the context of crawl state handling
- fixed a serious bug in kelondroCache which caused that entries could not be removed
- fixed some bugs in online interface and adopted monitor output to new entry objects
- adopted yacy protocol to handle new delegatedURL entries
all old crawl queues will disappear after this update!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3483 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
9b5fb3908d
*) a peer-message are now created when a blog-comment is written
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3480 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
581db87237
more debug code for
...
http://www.yacy-forum.de/viewtopic.php?p=33009#33009
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3479 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
81c4cc6bf7
better debugging of balancer failure
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3478 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dd06d4cada
more logging to better trace bug
...
http://www.yacy-forum.de/viewtopic.php?p=33001#33001
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3477 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
96b79bf86d
redesigned remove method in kelondroRowSet
...
This should fix also numerous bugs like
http://www.yacy-forum.de/viewtopic.php?p=31077#31077
(java.lang.ArrayIndexOutOfBoundsException in kelondroRowCollection.removeShift)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3476 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9f929b5438
better snippet handling in case of snippet load fail
...
see also http://www.yacy-forum.de/viewtopic.php?p=31096#31096
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3475 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
d451ad48d3
*) improved peerloadgraphic:
...
- unnecessary (0 %) pieces are removed
- percent-values of each thread displayed in legend
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3474 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a5d668c0c6
added speed-buttons for easy performance setting
...
appears in crawl start and on indexing monitor page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3473 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5b0a84ce09
fix for synchronization deadlock with flushMissNameCache.
...
see also: http://www.yacy-forum.de/viewtopic.php?p=32939#32939
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3472 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
e2ac5f62bd
- Code hübscher machen [von NNs TODO]
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3471 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
f04097c3dd
integrated tor-patch for crawling, if yacyDebugMode is set.
...
(replaces: http://yacy.deruwe.de/overlay/net-misc/yacy-tor/files/disable_dns_checks-svn3132.patch )
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3470 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
22fe14f292
*) first version of Peerload-graphic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3469 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
432d7d4e9c
better catch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3468 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8f7e8b6ee2
auto-delete for not-fixable db error in crawl stacker.
...
see also http://www.yacy-forum.de/viewtopic.php?p=32906#32906
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3467 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7a52b07fcc
better memory protection during freemen cycle
...
see also http://www.yacy-forum.de/viewtopic.php?p=32903#32903
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3466 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6faa262259
fix for NURL-fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3465 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
909d7a8ae9
fixed wrong implemented row iterator in kelomdroFlexSplitTables
...
this has no effect, until now this iterator was only used on
the Index Administration page.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3464 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a1fb8358b2
lets make a well-formed http link so that other crawlers don't have a problem to follow this link :-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3463 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4edb70f68b
added yacybot info-page from Roland
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3462 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3ef77d2030
fix for http://www.yacy-forum.de/viewtopic.php?p=29878#29878
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3461 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3bb3df3fc0
fix for http://www.yacy-forum.de/viewtopic.php?p=32298#32298
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3460 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
243a2f831b
fixed problem with not found NURL-hashes
...
The cause for this problem could still not be found, but the effect
is handled much better. The NURL-pop will continue automatically until
it found a hash that can be found.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3458 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6ad39bae1e
fixed shutdown problem
...
this fixes the 'inconsistency' messages during start-up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3457 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
38b93f8cb8
bugfix for my last commit:
...
iterator did not consider secondary start point in case of rotation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3456 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
264a82eec8
- fix for http://www.yacy-forum.de/viewtopic.php?t=3657
...
- fix for http://www.yacy-forum.de/viewtopic.php?p=32758#32758
- Diff takes any objects now, not only strings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3455 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d755a8026d
- better OOM protection
...
- better memory allocation for FlexTable indexes
- splitting between static index and dynamic index (only the dynamic part must grow)
- to enable a merge-iteration of new splittet index, a huge number of classes needed to be adopted for new iterator classes
- added new iterator classes that support cloneable iterators
- adopted all iterator classes to implement cloneable itarators
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3453 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
23338d2070
small fix for RAM computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3447 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
33f97cff7a
changed startup initialization sequence slightly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3446 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4e8eb1dbe3
some minor changes here and there
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3441 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
03c5906ae7
- minor bugfixes for url-fetcher & http://www.yacy-forum.de/viewtopic.php?t=3646
...
- PerformanceMemory_p.html is valid XHTML again
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3440 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3499a364ef
a little bit better memory protection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3439 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
313f6a7680
fix for http://www.yacy-forum.de/viewtopic.php?p=31553#31553
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3438 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
958ebea5c5
fix for http://www.yacy-forum.de/viewtopic.php?p=32470#32470
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3437 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5d5e6ebfcc
fix for http://www.yacy-forum.de/viewtopic.php?p=32631#32631
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3436 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1cba31de43
redesigned ram organization for database caches
...
- each cache can now allocate as much memory as is available
- no more fixed limits
- replaced old performance memory monitor by new one
- added supervision methods as static functions into the classes that provide cache functionality
- steering of ram allocation is done with two simple limits that are ram availability-relative
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3434 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
26450a1d9a
*) avoid nullpointerException on seed.getAddress() (reported by netbude)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3431 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
db235f2d61
added some memory protection in collection index multiple merge
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3429 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
c72605ecab
*) adding a function to determine if a given URL is bookmarkt
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3428 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
bd03c6b874
*) bugfix in bookmarksDB:
...
- NullpointerException when trying to get an unknown bookmark
- bookmarks can either start with http or https
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3427 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b466baa574
added some memory protection
...
too large collection arrays are now avoided. By default, the biggest
collection index is 7. larger collections are dumped into a commons
directory, but cannot yet be used. Bevore doing a dump, the collection
is splittet into a part which has only root-references, and stored back
to the collection; the remaining part goes to commons
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3426 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
ce360ef43e
*) no more HTML in plasmaCrawlProfile.java anymore
...
*) <br> will not be displayed in items in Auto Filter Content on WatchCrawler_p.html anymore
*) removed unnecessary replaceHTML()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3425 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
88245e44d8
- improved version of robots.txt (delete your old htroot/robots.txt before updating):
...
- robots.txt is a servlet now
- no need to rewrite the whole file each time a section is added or removed
- user-defined disallows, added manually, won't be overwritten anymore
- new config-setting: httpd.robots.txt, holding names of the disallowed sections
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
9623bf7bbe
- removed call of java 1.5 method
...
- added config servlet for local robots.txt
- removed YPStats_p as it is of no use anymore
- supertemplates use XHTML now
- quick-fix for http://www.yacy-forum.de/viewtopic.php?p=32296#32296
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3422 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
51e12049fa
third generation of R/W head path optimization
...
- data from collection arrays are read in order
- merged data is written in order
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3419 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
a1d68fe092
- use .class rather than Class.forName for classes in class-path
...
- added Bost's patch for Diff.findDiagonale() from: http://www.yacy-forum.de//files/patch_685.txt
- fixed minor bugs in Blog
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3416 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
10a3c20b8d
some more enhancements to R/W Head path optimization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3415 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f4cfd19835
second Generation of collection R/W head path optimization:
...
- permanent cache flush is switched off. The optimized cache flush
works better if it is a large number of collections that is flushed
together
- the flush size can be configured instead the flush divisor. There is
only one size for all flushes
- collection records that shall be removed during collection transition
(jump from one collection file to another) are now not really removed
but only marked in RAM. add-operations to the collection use these
marked collection spaces
- index bulk write operations are now separated for each file of a kelondroFlex
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3414 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1fda50fd3c
correct R/W head positioning in kelondroFlex
...
and some enhancements
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3409 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
304412a049
first generation of collection index R/W head path optimization
...
- collections are now hand-over as collection lists to collection index for merge opertations
- collection index lists are separated into 'new' and 'extend' lists
- lists are written separately
- write operations are done into array sets and array indexes. These are now serialized
- write operations into index files are sorted by index;
that means that a R/W head does not need to go forward
and backward, only forward
More enhancements are possible
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3407 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
54fef3574f
*) missing files for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3406 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
cb89c74d52
*) added blog-comments
...
*) removed debug-output when deleting news
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3405 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
6fbe31425a
- some code-cleanup (no more syntax-warnings here)
...
- added deletion from loadedURLs of URLs to be blacklisted in IndexControl_p
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3404 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
32867580ee
update to kelondroRecords needed fo last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3403 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e3480d4ad3
fix for warning in crawl balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3402 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8668ac5d91
preparations for collection index cache flush optimization
...
(hand-over commit, no functional change to current code)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3399 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
39a2000d8b
- added support for [[Bookmark:$bookmarkTag|description]]-link-listings (requested by theli) to wiki-parser
...
- added support for <pre>-tags to wiki-parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3393 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
619653c054
- fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3392 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
26f5757b40
- added support for multiple paths per domain to default-blacklist
...
warning: an interface-change had been neccessary:
- remove(String, String) has been renamed to removeAll(String, String), because it removes all path-entries for the specified host
- remove(String, String, String) has been added to delete only a path-entry
- geBlacklistType(String) has been renamed to getBlacklistType(String)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3391 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
a5a36d9252
- hopefully last fix fo 1.5 methods (sorry for that, eclipse isn't that helpful in identifying those methods)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3387 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
e97b6f0458
- we still use Java 1.4 ...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3386 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
0c7b8cf632
- added first version of new wiki-parser
...
- added blacklist support to manual URLFetcher stack fill
- fix for NPE: http://www.yacy-forum.de/viewtopic.php?t=3559
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3385 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f7803a6ce4
enhanced crawl balancer
...
- new domains now get a chance to get crawled early
- less IO operations
- new balancing method
- better dump order at shutdown time
- bugfixes regarding not found url hashes (no more superfluous cache kill)
- domain access time is now shared over all balancer stacks
- viewing the stack does no more disturbish the balancing algorithm that much
- intelligent selection of best next domain using domain access times
- extra double-check (to double-check the double-check)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3384 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
801eea8849
*) Fixed bug where pairReplace() got caught in infinite recursion. ( http://www.yacy-forum.de/viewtopic.php?t=3466 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3383 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c3e8c23f5d
fix for 'CANNOT FETCH ENTRY: hash is null' bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3380 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
badab8d924
fixed some more bugs in new db handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3379 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e72d253577
fixed problem with initial cache load
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3378 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2d8e472cfd
emergeny bugfix for last commit
...
(kelondroTree should work again)
the cache prefill is broken and will be fixed later
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3377 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dc0c06e43d
PLEASE MAKE A BACK-UP OF YOUR COMPLETE DATA DIRECTORY BEFORE USING THIS
...
redesign for better IO performance
enhanced database seek-time by avoiding write operations at distant
positions of a database file. until now, a USEDC counter was written
at the head-section of a kelondroRecords database file (which is the
basic data structure of all kelondro database files) to store the
actual number of records that are contained in the database. Now, this
value is computed from the database file size. This is either done
only once at start-time, or continuously when run in asserts enabled.
The counter is then updated only in RAM, and written at close of the
file. If the close fails, the correct number can be computed from the
file size, and if this is not equal to the stored number it is a strong
evidence that YaCY was not shut down properly.
To preserve consistency, the complete storage-routine had to be re-written.
Another change enhances read of nodes in some cases, where the data-tail
can be read together with the data-head. This saves another IO lookup during
each DB node fetch.
Includes also many small bugfixes.
IF ANYTHING GOES WRONG, ALL YOUR DATA IS LOST: PLEASE MAKE A BACK-UP
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3375 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
c016fcb10f
- added streaming-support to CrawlURLFetchStack_p servlet
...
- bug for NPE in list.java
- use more constants
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3373 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
d114a0136e
- crawl profile: don't add null-values
...
- added some settings and statistics for url-fetcher 'server'-mode
- added own stack for fetchable URLs
- added possibility to fill stack via shift from peer's queues, via POST (addurls=$count and url$num=$url) or via file-upload
- added "htroot" to classpath of linux start-script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3370 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
b2a9d32f29
why do I always forget some lines? sorry...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3368 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e1edb23689
*) Bugfix for IllegalMonitorStateException
...
See: http://www.yacy-forum.de/viewtopic.php?t=3522
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3358 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
bf69a721cb
more protection against mis-use of YaCyHop interface:
...
- target must not be at port 80
- target access not more than every 3 seconds
- requester may not access more than every 10 seconds
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3357 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a15963ff98
better balancing: if element from top would force a busy waiting,
...
an element from the bottom of the stack is used instead.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3356 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dda24fcb85
ups
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3355 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8c1d2e0227
protection against crawl balancer failure:
...
a minimum of 500 milliseconds distance between two acesses
to the same domain is now ensured
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3354 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1f1f398bfa
enhanced speed of RAM cache flush by factor 20 (twenty times faster)
...
- the speed was doubled by avoiding read access during the dump
- the speed was dramatically increased at least by factor 10
by using a temporary ram-file where the structures are flushed to
before it is dumped then as a whole byte-chunk to the file system.
The speed enhancements also affects some other parts of the database.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3353 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
30d79d69a6
fix for wrong display of search statistics
...
see http://www.yacy-forum.de/viewtopic.php?p=31242#31242
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3352 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
daf2e15f59
some storage process enhancements (write without preceding read)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3348 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9c2101a852
small enhancement to cache dump
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3346 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c464157a6e
replaced some toString()
...
see http://www.yacy-forum.de/viewtopic.php?p=31151#31151
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3345 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7673f0869b
minor enhancements
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3344 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b4aa195c27
added user-agent check for yacy-hop proxy authentication
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3343 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2d7f7da7ce
fix for null pointer exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3342 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d25caa07bf
redesigned some parts of http authentication
...
added another access check for peer hops
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3340 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
9184113284
*) fixed News deletion. News are now removed if they are no longer in a news-stack. This does not effect News-entires in the news-db that have no stack-entries.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3336 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
e218940293
The copyright sign "\u00A9" is already replaced by "©". String "(C)" is not a unicode sequence!
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3334 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
1bc4d8d470
*) If there is more than one pair of patterns in a line, all of them (and not only one pair) will be replaced.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3333 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
ea7a8cf7aa
*) <hr> and <br> tags are XHTML compliant now.
...
*) Avoid superflous trailing blank in non-proportional sections.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3332 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d03cd41266
fix for http://www.yacy-forum.de/viewtopic.php?t=3411
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3331 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
f2e6f19b90
- added versioning to Wiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3327 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
10ebcb5e1e
fixed bug in disconnection time
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3326 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
02a73dce87
- added Diff-class for wiki-versioning (forthcoming, first need suitable serverObjects.put() for it)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3325 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
2401e748a3
- fixed wrong replacement of POST-parameters in httpd ('<' and '>' are still replaced, don't know why): http://www.yacy-forum.de/viewtopic.php?t=3466
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3324 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b2f4087400
redesign of last-seen fieln inside seed:
...
the field contains now a time in UDC-0 (instead relative to local UDC offset)
this fixes a bug in peer selection, where an iteration over all seeds
ordered by lastseen did not work correctly.
Problems may occur because the new meaning of this field may mix with
the different meaning of that field in older peers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3322 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
e68cdeeeb3
- reverted parseArg(String) to use a byte-array to handle correct UTF-8 parsing
...
- arguments aren't passed html-escaped to the servlets anymore, bug-fix for http://www.yacy-forum.de/viewtopic.php?p=30573
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3321 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e00e850a98
removed constants (no connection with yacySeed.dna identifier)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3320 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
fcc11391a8
some redesign attempts because sorting of lastseen does not work correctly
...
not finished yet
target: better selection of peer-ping targets, which should enhance stabilization of the net
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3319 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c2d6edf21d
integrated number of remote targets as 'partitions' into remote search protocol
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3317 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f696d3c1eb
added double computation to kelondroMapObjects
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3316 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
819ff21c92
fixed QPM output
...
QPM is temporarily called QPH (until more search requests are present?)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3313 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
89e7af037a
*) used more switchboard-vars instead of config-vars
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3310 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
306c50ac40
QPM (queries per minute) statistic stub
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3308 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7598e1243e
removed unused variables/imports
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3306 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
27e4429645
*) fixed minimizeurldb-function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3305 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
47ab83a7c0
added flag for YaCyHop - proxy access for all paths that start with /yacy/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3304 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
9f74b128dd
- added many more commented constants (please use constants rather than i.e. config-setting strings directly)
...
- not all constants may be located correctly in the switchboard. Please relocate if you know the appropriate place for them
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3303 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e4910f03d1
tag storage fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3302 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
991182b29b
more space for bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3299 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
88fa764b64
implemented new kelondroObjects into bookmarkDB
...
- Bookmark-Objects are stored inside the kelondroObjects cache
- removed superfluous classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3298 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9c05e2a820
re-design ob kelondroMap
...
- this class is replaced by an object that can hold any type of object
- this object must be defined as a class that implements kelondroObjectsEntry
- the kelodroMap is now implemented as kelondroMapObjects
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3297 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
669c21db05
first version of abstracted kelondroMap Cache.
...
get returns a kelondroCachedObject(or in most cases a subclass of it),
or a map, which can be used to construct a kelondroCachedObject.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3295 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
98cb777e18
abstract wikiCode in putWiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3293 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f25c0e98d1
- replaced String by StringBuffer in condenser
...
- added CamelCase parser in condenser
- added option to switch on or off indexing for proxy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3292 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
14f2068daf
some more bookmark changes towards multiuser bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3291 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
25c7d4e25e
fix for form (cookie) login
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3284 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
ff79c52fc0
bookmark users can now edit bookmarks.
...
TO COME: tag bookmarks with username, list bookmarks of a special user, filter private bookmarks for users.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3274 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
49972b6193
*) SOAP: getPeerHash returns the proper value now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3273 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
15f0334cd3
- fixed IllegalThreadStateException in LogParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3265 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
f40169fcd7
preparing multiuser bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3256 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
814a09a0ed
*) reversed r3250 and parts of r3252 (nanotime() is an java1.5 function)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3253 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
d311e258f8
- adjusted LogStatistics to nano-seconds
...
- removed patches of SVNs 3184/3185 preventing fast DHT
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3252 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
f7623f5d24
*) added missing measuring points for Parser-Runtime
...
*) changed precision of Parser-Runtime from ms to ns
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3250 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f3f99b19c6
extended search statistics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3249 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c0851ee943
refactoring: moved and renamed de.anomic.data.searchResults to plasma package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3248 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
c39dda2374
finished refactoring of searchtemplates.
...
now plasmaSwitchboard.searchFromLocal calculates a searchResults structure,
which is parsed in the yacysearch/detailedSearch Servlets.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3244 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
35039982da
refactoring of search process: store results in a searchResults structure. At the moment, its just stored in it, and read from it again.
...
Next step: return searchResults instead of serverObjects, and parse the results in the servlets.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3241 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
76fab83395
fixed bugs in seach statistics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3240 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
139810d504
- truncate attributes of new newsRecords if too long
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3239 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
3c43e605ba
- don't accept malformed bookmarks, fix for: http://www.yacy-forum.de/viewtopic.php?t=3414 (first report)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3238 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d07b132a0d
- fixed colors of network grafic
...
- added option to activate write cache for seed-db
- did not activate write cache because it did not work
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3236 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
5d540b219e
- LogalizerHandler skips interfaces again
...
- added LogParser stats to LogStatistics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3234 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
e1fb3550ab
fix for profile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3231 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
6faf9b70b7
*) LogParserPLASMA now counts its total runtime.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3229 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
e5f854bc37
*) added LogalizerHandler-settings to yacy.logging.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3228 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
7c40197e42
- fixed error pages and <label>s for index.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3226 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
29aa7031d3
workaround for the snippets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3225 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
b4457763e5
fix for putSafeXML and supertemplates.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3223 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
77b73aa7a8
- log-entries 'Indexed' are parsed correctly now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3222 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
aea199cb7b
- IndexTransfer is working again
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3220 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
71112b1fe6
- added LogStatistics_p.html servlet based on the logalizer (indexing values not functional yet due to charset/regex problems)
...
add the following to DATA/LOG/yacy.logging:
---
# Properties for the LogalizerHandler
de.anomic.server.logging.LogalizerHandler.enabled = true
de.anomic.server.logging.LogalizerHandler.debug = false
de.anomic.server.logging.LogalizerHandler.parserPackage = de.anomic.server.logging.logParsers
---
and "de.anomic.server.logging.LogalizerHandler" to the list of global handlers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3219 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
0c81bd39d4
XSS-safe put as default.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3217 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
bdda9e802f
- added some commented string constants to ease use of the result-table
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3215 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5515571950
redesign of ymage classes
...
- less memory usage
- better usage of awt classes
- drawing abstractions: preparations for movable objects for animation class
- test applet for animations
- known bugs: wrong colours for network picture
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3214 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
52c6461e6b
some bugfix for statistics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3211 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
fe72b772cf
added a monitor page for search requests
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3206 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
b3a2c79fec
- fix for last fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3202 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
4b23e79f51
- quick-fix for http://www.yacy-forum.de/viewtopic.php?t=3358
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3201 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
4dce5ec261
- if mem is too low but former GCs helped, the word-cache limit is only decreased now, if a subsequent GC doesn't
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3197 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
c27e88104c
*) getResults() should now work and compile properly with Java 1.4
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3191 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
a2fb54afff
*) Quickfix for http://www.yacy-forum.de/viewtopic.php?p=29973#29973 getResults() used a java 1.5 function (Output is temporally disabled until a sulution with 1.4 functions is implemented)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3190 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
b873ad51ab
- fix for http://www.yacy-forum.de/viewtopic.php?t=3369
...
- merged netBude's alternative for tables in yacysearch.html & search results valid
- added statistic info to index.html as proposed here: http://www.yacy-forum.de/viewtopic.php?p=29762#29762
- fixed error-log in httpTemplate
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3189 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
7fb30de8a4
*) made combined2prettyversion less restrictive and made it again easier to read
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3187 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
3acd90033c
*) added functions to get results from log-parsers (not documented yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3186 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
1aa74bbd2b
update for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3185 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
23e613b2ab
CPU & IO reduce (Index Distribution)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3184 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
c67d22116e
added exists-check based only on RAM index lookup:
...
- faster double-check during crawling
- less IO
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3179 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
37e53b4a6a
replaced tree database structure for seed db by flex data structure
...
I don't know if this helps, we will find out...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3177 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
340dc52a9d
- ConfigProfile_p.html now transmits usable encoding for other than 7-bit ASCII charset, see TODO in httpd.parseArg(String)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3174 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
35fb671721
- updated DetailedSearch and ViewFile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3173 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
00aa9472d6
- added decode of HTML-entities in request lines
...
- removed Bookmark symbol on search pages and surftips if not authenticated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3172 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
d157201e08
*) IfesL for "Unexpected end of ZLIB" error message
...
See: http://www.yacy-forum.de/viewtopic.php?t=3327
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3169 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
0336480a3e
- the maxMemory-fix for the Sun JVM 1.4.2 wrongly also applied to 1.6, thx to NN
...
- added logging of reducing word-cache (log-level fine)
- disabled memprereq field in PerformanceQueues_p.html, because it is now set by the collections db
- minor changes to ConfigSkins / -Language
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3165 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
735c1f87b1
restart re-activated (windows)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3164 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
2c01508ada
*) fix for http://www.yacy-forum.de/viewtopic.php?p=29575#29575
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3162 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
6ff8359b98
possibility to use anonther bindPort than the externally reachable port.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3161 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
d2be3c674d
wrong cache values fixed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3159 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
df6281ba1f
- removed JS from DetailedSearch => valid
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3151 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
fb1d8b91af
*) changed Startpoints of IndexCleaner and IndexTransfer from ------------ to AAAAAAAAAAAA.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3150 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
d6eb699e8e
- fix for last commit (didn't know that the paragraph sign has an UTF-8-specific location)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3134 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
41bc31d2c2
- ConfigAdvanced_p => XHTML (no invalid IDs)
...
- removed unmappable characters from code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3133 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9b726ac366
release 0.50
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3132 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
036a0c828e
fix for auto-configuration of crawler thread memory
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3131 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
16f88c49d9
applied patch from http://www.yacy-forum.de/viewtopic.php?p=27392#27392
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3130 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2aa6f978c2
fix for http://www.yacy-forum.de/viewtopic.php?p=29125#29125
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3129 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a4e90bc1dc
fix + debug-code for http://www.yacy-forum.de/viewtopic.php?p=29126#29126
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3128 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
6b5f28b746
answer for last commit: no
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3126 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
d98ba7bc33
fix for memory limit computation ?
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3125 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
49ffedfd8b
patch from Fuchs against deadlock,
...
see http://www.yacy-forum.de/viewtopic.php?p=28804#28804
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3122 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
a4f63d187d
better map2string and NullPointer fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3119 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c48374d14a
new memory limit computation for indexing queue
...
shall better prevent outofmemory errors
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3118 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
a9f1d3a065
*) SOAP: adding function to get the local peer hash
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3117 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
9659c004c3
fix for wrong date (Lotus-1 Peer)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3111 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
08ac4c5ed0
bugfix for http://www.yacy-forum.de/viewtopic.php?p=29045#29045
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3110 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8e3bd17554
adopted DetailedSearch page to new ranking options
...
- fixed bug http://www.yacy-forum.de/viewtopic.php?t=3265
- more attributes on page
- attributes can be set as default for main search page
- option to re-set the attributes to built-in values
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3109 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
93a7e88245
more ranking parameter usage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3108 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2dbea612c9
fixed display bug for image search preview
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3107 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0a050bc043
enhanced ranking
...
- redesign of data storage in plasmaSearchRankingProfile
- profiles are extended by new ranking parameters
- new RWI ranking parameters are considered during ranking
- appearance attributes (i.e. emphasised text) is now considered
- faster ranking
- some attributes that had been checked during post-ranking can now be
checked during pre-ranking phase
- removed old ranking parameter on index.html page (will be replaced by profiles in the future)
- ranking can now consider appearances of media content
- snippet-loading for media types now work correctly (fetches only from the wanted media)
- ranking-profiles can be handed over the remote peers and apply there also
- re-search of same query with different domain now also re-triggers remote search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3105 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d0c32c6aeb
better protection against fraud peers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3104 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
61798f0ae6
added option to distinguish between text crawl and media crawl
...
- for each crawl start, there is now a flag for text and media
- the localCrawl flag is superfluous
- added new crawl profiles
- if an image search is done, only media links are crawled for the snippets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3100 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
be941c4475
- "javascript:"-URLs are recognized as well (as intended formerly I assume)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3097 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
a619ba3f49
- fix for String index out of range during URL parsing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3096 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
febe6b114a
design update of crawler monitor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3094 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
40049e0635
fixed media search snippet flow
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3092 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
5763d20795
- added DATA/yacy.running file to indicate whether YaCy is running or not. Helpful for i.e. the updater and linux init-scripts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3089 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
05d0464377
only do 16 checks if "address" starts with "172.";
...
better readably;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3087 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
1a525710c1
*) cursor jumps now to searchbox on searchpages again
...
*) added missing private IP-ranges for APIPA/Zeroconf and 172.16.0.0–172.31.255.255
*) Changed some seed-download-errors to warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3086 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
e17591acc3
- parse HTML arguments as UTF-8 strings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3085 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
d30932c7d8
- fix for fix... sry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3084 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
6118fb73ec
- added decode of UTF-16 escapes in url-arguments (%u0123), bugfix for http://www.yacy-forum.de/viewtopic.php?t=2762
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3083 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
782db9099d
version independent name for commons-pool lib
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3082 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7ff86d6ba6
- image search now shows thumbnails (in bad order, but it works)
...
- repaired DHT selection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3081 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ee3d91cb6b
print-out of links that result from contraint-filtering
...
in search result
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3078 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
f4737eebd6
Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3076 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
1fc75c0f67
better logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3075 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e4570bffaf
-implemented a specialized snippet-fetch for media content
...
-changed search result preparation for media search presentation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3073 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
694a6e4f44
*) better text snipptes: any possible searchword (welt, linux, tag) in welt-linux-tag will be marked correctly now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3072 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
bddc197453
reverted by-mistake removed change from low012/SVN 3068
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3070 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1377c53aa3
extraction of media links from search results
...
these links are mixed to the snippets for testing purpose
(a final version will handle this differently)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3069 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
586add4c6c
*) Better snippets: words like GNU/Linux will not prevent Linux or GNU from being marked if they are searchword (see http://www.yacy-forum.de/viewtopic.php?t=2891 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3068 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
29922d5dd2
changed writeMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3066 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
5b00a04359
changed writeMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3065 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
52abbd4131
- fix for wrong public IP if no hostname was set and IP was from range 192.168.*.*
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3063 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
8b7c543885
NullPointer fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3061 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
937ccd4e76
fix for snippet-generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3060 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
559f41a001
fix for http://www.yacy-forum.de/viewtopic.php?p=28607#28607
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3059 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
c086c71f17
*) fixed ArrayIndexOutOfBoundsException
...
--> http://www.yacy-forum.de/viewtopic.php?t=3210
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3058 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c93cfdc23a
fix for http://www.yacy-forum.de/viewtopic.php?p=28564#28564
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3057 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
93a5ace330
fix for http://www.yacy-forum.de/viewtopic.php?p=28544#28544
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3056 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
bf0d820659
- added correct flagging of word properties
...
- added self-healing to database in case that wrong free-pointers exist
- added presentation of media links in snippets (does not yet work correctly)
- code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3055 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
10d888e70c
- added a media search for images, audio, video and applications
...
- new search options on search page
- new option in ViewInfo to display all links of a file
- enhanced collection data structure
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3054 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a603c4d5e8
more code simplifications
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3052 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9a85f5abc3
cleanup
...
- removed 'deleteComplete' flag; this was used especially for WORDS indexes
- shifted methods from plasmaSwitchboard to plasmaWordIndex
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3051 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
fbe1ee402b
plasmaCrawlLURL$kiter cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3050 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
773ba1e91a
- generalized object order handling
...
- controlled object order for all database tables
- migrated DHT position computation to correct base64-decoded values
this also closed the 'gaps' in the dht positions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3049 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
15381cbf73
other bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3048 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
ad65cc9d2f
NullPointer fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3047 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago