orbiter
154ffd7c2c
fix for wrong http connection version and SSIs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3928 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1782ef57e5
- added SSI parser and include directive for <!--# include virtual="<file>" -->
...
- added chunked file transfer for non-yacy clients
- SSIs are streamed using chunked transfer, partly delivered pages can be seen in browser before transmission is finished
- added client-side network unit identification
- cleaned up code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3926 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6b4cfbd2d6
new network bootsraping method
...
- no more contact to yacy.net (no remote superseed any more)
- moved superseed file into new network unit definition
- fixed build; includes new network bootstraping files now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3922 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0e57a8062b
added network definition for different YaCy networks
...
(needs much more work)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3919 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
1d41ebf489
*) made age for deletion of too old seeds configurable
...
*) changed naming-scheme of seed-deletion-properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3918 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
52cb3208d0
*) old (lastseen > 7d) peers are now automatically removed from passive and potential seed-dbs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3917 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
815e3da62f
fix for http://www.yacy-forum.de/viewtopic.php?p=37353#37353
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3913 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
michitux
25529290ca
- 2 small changes in documentation
...
- hopefully fixed logging of GCs (in order to avoid things like "performed necessary GC, freed 18014398509481565 KB (requested/available/average: 4096 / 1631 / 2957 KB)") with the help of KoH
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3909 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
c59a7ce5c2
*) hopefully fixed a stupid bug (my fault of course) that sometimes messed up the marking of search words in the snippets (see http://www.yacy-forum.de/viewtopic.php?p=37329#37329 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3908 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6518bb6c08
changed release strategy:
...
we will provide two different releases in the future, one standard release and one 'pro'-release.
the 'pro'-release contains all additional parsers AND has different default performance values.
The pro-version differs therefore from the previous 'all'-version by this default values.
The pro-configuration is automatically choosen if the libx-folder exists. If a version is once initialized, its configuration stays independently from an existing libx folder.
The ant targets had been changed. There are now 3 different targets to create standard and pro-releases, and one target to upgrade:
- dist: creates a standard release (only, no libx target any more)
- distPro: creates a pro-release (includes the libx)
- distExt: creates a libx-release which includes the libx-folder only. It may be used to upgrade from standard to pro
Furthermore, the naming of 'dev'-releases had been removed.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3902 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
069562a14d
fixed problem with re-crawl; replaced error file-db with ram-db
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3900 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c7a614830a
several bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3899 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
465145cb6f
revert to insecure, but dau-proof defaults
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3898 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
7ad11ceaaa
security fix for peers without password. allow access only from localhost
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3897 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
2784820ee3
*) moving sleep to a better place
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3895 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
7a1b811d18
*) bugfix for SocketException:
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3893 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
71fd972ac0
- reduced default search time
...
- catched case when web structure cannot be painted because of too less data
- better logging when balance fails
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3892 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2b937abef1
slighlty different behavior in shutdown sequence for http server threads:
...
- first close streams
- make pause (that one that was made in httpdFileHandler)
- close sockets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3890 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e4aa8f2a08
disabled more sleep(200)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3889 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
cb38e57622
reduced httpd final waiting time
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3888 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b4585ad67d
im Sommer 2005 wurden die ersten pings zwischen YaCy-Peers ausgetauscht.
...
Das klappte aber merkwürdigerweise nicht immer. Um das Protokoll zu testen schrieb ich eine einfache message-Funktion, so wie sie heute noch in YaCy drin ist.
Aber auch die Messages funktionierten nicht richtig. Alex und ich haben lange Zeit gesucht, und den Fehler nie gefunden. Es stellte sich heraus das ein Timing-Detail das Problem lösen konnte, die Ursache haben wir bis heute nicht gefunden.
Die Lösung des Problems bestand aus einem kurzen sleep, kurz bevor der httpd Daten zum client zurück geschrieben hat. Das ist natürlich eine fürchterlich schlechte Lösung.
Bis heute war diese Sache im httpd. Mit diesem Commit habe ich den sleep auskommentiert, und es steht zu befürchten das wieder irgendwas nicht geht.
Wenn jetzt das Netz zusammenbricht, keine pings mehr ankommen oder so, war es dieses sleep, das es verhinderte.
Vorschläge willkommen.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3887 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f64d9f1c6c
removed forced termination in case that a previous bad termination is detected
...
this will cause many users to be unsure what to do next an leave them helpless
to simply delete the control file is the same thinig that the user is othervise forced to do
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3885 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
e1d809d5f1
- more detailed logging of MEMORY messages
...
- forced GCs don't contribute to heuristics anymore
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3881 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9f7765863b
bugfix for seed length control routine
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3879 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0b10ef64ba
better server access tracking
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3878 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4f5496062c
protection against too large seeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3877 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
684ded0e09
added new news types
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3876 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
669f840eab
- added ViewProfile / Impressum (default on) to local peer's robots.txt
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d7de0938a6
fix for http://www.yacy-forum.de/viewtopic.php?p=36587#36587
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3870 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
22ee85ca02
- specified exceptions thrown by ResourceInfoFactory and plasmaHTCache.loadResourceInfo()
...
- caught possible NPE in CacheAdmin_p and added more error-cases
- speeded up deletion of entries in the local crawl queue by crawl profile (it has been noted often that this deletion is slow)
- added a bit javadoc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3868 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dfd5e823c3
automatic limitation of web structure host count
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3867 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8b0aea6910
fixed automatic deletion of too many referenced hosts in web structure
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3866 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5dd9acc2a7
removed calls to deprecated methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3865 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
66ec8b63c1
added a httpd access tracker:
...
- all requests to the own httdp can now be listed in the access tracker menu
- the search statistics had been renamed to access tracker and extended by this tracker
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3861 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
71ca9aa6d4
- fix for changed blacklist types
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3857 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9a8a87612d
added new qph column to search tracker servlet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3854 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e07458bad4
added time-out function to web analysis
...
the default time-out is 1 second
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3852 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
4a1bc4743a
*)News-entries with blacklisted URLs are now ignored
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3849 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
6074264267
dynamic rights.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3847 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
99062c0c9e
*) SOAP should support authentication against the user-DB now (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3846 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
339153d40e
*) favicons that are specified in the document content via html link-tags
...
are now detected and displayed on the search page (requested by allo).
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3845 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
854eb1492f
.yacy /.yacyh urls for the feedreader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3844 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
7a5b22a0b8
Integration of FeedReader in Bookmarks.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3841 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
6265d321bd
- more constants
...
- display why global search is not available on search page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3839 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
7921f07c9d
userDB fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3837 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
7b2e1bb8f2
Feedparser with reflection.
...
TODO: This needs a special build.xml entry
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3832 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
8bff810d19
- fixed logging output of serverMemory.request()
...
- don't start up if DATA/yacy.running exists as this is usually a sign of an already started yacy-instance
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3831 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
18a5380ee3
*) situation-dependent lock-buttons for search-page
...
*) removed one unused import and a double definition of "ogg" as media-type
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3817 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
f05ca43780
- the wiki-parser works for remote wiki-code now, not displaying links anymore as if they were local (ViewProfile comment)
...
- fixed wrong link to CrawlStart on Status-page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3816 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
9d6605a83c
- fixed NPE in Blacklist Cleaner during deletion of more than one double entries
...
- don't display responseHeader1.db in CacheAdmin_p anymore
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3814 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
594ff95955
:-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3801 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4ca797401e
fix for ConcurrentModificationException
...
see http://www.yacy-forum.de/viewtopic.php?p=36566#36566
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3800 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7b904e0077
integrated robots.txt crawlDelay into the crawl balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3797 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
52cb033f01
- slightly different painting of web structure picture:
...
hosts that have many own connections are painted farer away (this is not yet cato's idea, this will be implemented in another step)
- doc update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3796 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
30c3d909b1
- fixed charset problem in ConfigProfil_p.html (use accept-charset="UTF-8" in forms)
...
- fixed wrong XML output if no peers are known in Network.xml
- simplified parsing of table properties in wikiCode and ZTableToken
- reimplemented GC heuristics. They are needed to constantly ensure that an amount of free memory is available which is higher than Java's max. limit for performing a Full GC (please use serverMemory.request(long, boolean) rather than serverMemory.available(long, boolean) to provide data for averaging over the last GCs)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3793 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
6c9df13552
more debugging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3791 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
4392ee0c51
BugFix for typo and wrong include
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3789 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
d1e1580223
Surftips Blacklist
...
Blacklists List Hardcoded instead of only updated on firststart / migration.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3788 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e1a5babff1
*) Logging GUI handler: line-size is now set to max-size if max-size was exceeded
...
See: http://www.yacy-forum.de/viewtopic.php?p=36355
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3786 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
94cc9f05f5
*) Improvements for restart via update wrapper
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3785 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
44bac7dea1
*) blog-comments can now be moderated
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3778 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
2ab020445a
bugfix, i think - http://www.yacy-forum.de/viewtopic.php?t=4059
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3777 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
f89517203d
*) SOAP: new function to get the Performance Settings of Queues and Processes
...
No items left in the yadmin SOAP-TODO :-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3776 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
92351c4dcb
*) SOAP: bookmarks list now indicates if a bookmark is private (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3775 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
957a25afff
getRight(rightName) instead of get...Right()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3774 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
e0e46d3aec
*) SOAP: new function doGarbageCollection (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3773 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
1efe607c34
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3771 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
a0149317ac
*) fixed bug where headlines were added to directory of a wiki page multiple times ( http://www.yacy-forum.de/viewtopic.php?t=4034 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3762 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
ef24bed406
Sorry...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3760 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
a29cb2e1af
blupp
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3759 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
651b05ba43
*) wsdl file updated (requested by KoH)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3758 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0ae6664ad8
enhanced web structure picture
...
- hand-over of get properties from web front-end to graphics generation
- added depth-control buttons
- added marking of anchor-points to highlight relation order
- enhanced ymage graphics library
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3757 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a585b4d41b
added web structure image
...
see http://localhost:8080/WatchWebStructure_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3747 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
33ad0c8246
added a web structure computation and logging:
...
- all web page parsing operations will now increase a web structure file
- the file is computed in memory and dumped at shutdown-time to PLASMASB/webStructure.map in readable form (not a database)
- the file can be used externally to analyse the link structure of the crawled pages
- the web structure can also be retrieved using a xml-interface at http://localhost:8080/xml/webstructure.xml
- the short-term purpose is the computation of a link-graph image (before linuxtag!)
- a long-term purpose could be a decentralized computation of the citation rank
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3746 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
7904175338
- sorry for typos
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3743 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
baa9402b97
- wiki-parser is now configurable via the config setting wikiParser.class which holds the class-name for the parser to use
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3742 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
601fc7d1c5
- added source to J7Zip-modifed.jar and it's license (changelog is still to come)
...
- moved HTML-*replace-methods from wikiCode to de.anomic.data.htmlTools
- prepared use of different wiki parsers as suggested here: http://www.yacy-forum.de/viewtopic.php?p=34444#34444
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3741 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
0a64047081
- plasmaParserDocument can process subdocuments now (other archive-parsers may want to use this method)
...
- added 7zip parser
- added 'text/sgml' to realtime parseable mimetypes (sometimes returned by the mime type parser)
- added new cached output stream class, very suitable for parsers because of limited memory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3740 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b1680ab71f
*) bugfix for ArrayIndexOutOfBoundsException in robots-parser (thanks to low012)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3739 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b30e64daab
*) passing homepath to serverLog.configureLogging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3738 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
9a4375b115
*) robots.txt: adding support for crawl-delay
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3737 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
11ac7688d5
reverted a part of last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3736 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b3f97b5c38
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3735 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3c5ff7f735
adopted kelondroBytesIntMap to kelondroIntBytesMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3734 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5551ff5306
enhanced index storage data structure kelondroBytesIntMap
...
this stores now two index structures, one for data that is aquired during start-up
and one for data that is aquired during run-time. This reduces the grow factor, and should reduce the memory amount in case that a index-reorganisation happens.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3733 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
872eb46cb9
some redesign of the handling of the index for kelondroFlexTable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3732 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
086239da36
- added servlet: remote crawler queue overview
...
- added servlet: crawl profile editor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3731 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
8ba81e0995
- added some comments (will get more in the near future)
...
- added missing <label> to the search field in Network.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3728 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
65a8a9fc58
fix for nullpointer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3726 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b05e2314cf
another dht selection fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3725 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
03c6551b0c
- fix for http://www.yacy-forum.de/viewtopic.php?t=3747
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3724 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b28e5d0ee9
protection against wrong word hash length
...
see http://www.yacy-forum.de/viewtopic.php?p=35657#35657
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3723 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e897eb9b4a
fix for DHT selection target
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3722 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0384b8771b
fix for http://www.yacy-forum.de/viewtopic.php?p=35700#35700
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3719 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
578c2ef130
release 0.52
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3715 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
756a27049b
last-minute-feature 'newbie-selection' for workshop purpose:
...
for remote search, always select all peers that are less than a day old
(should be removed someday in the future if load is too high, which could mean when pph > 100)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3712 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
46367afaaa
update of memory-protection values
...
see http://www.yacy-forum.de/viewtopic.php?p=35539#35539
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3709 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
ea87fe5d78
*) Updated German translation
...
*) Changed "Lost Handle" error to warning (masses of it if deleting crawl-profile)
*) Removed unnecessary code from Windows script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3708 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
85035dc319
addition to svn 3699: check send/receive if p2p-mode is activated
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3701 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
26f05d1fd0
avoid division by zero if search is done for no words
...
this case is relevant if the bluewords (yacy.blue) are used
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3698 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
2fa8b50e54
reverting svn 3691+3692
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3696 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
139c59ebbd
- fixed dht selction problem: the seed tables used a wrong ordering
...
- cleaned some code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3693 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
22a0e9f117
more timeout-control
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3692 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
24db55a541
added timeout for httpd-sockets during read
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3691 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f3fee4d445
fix for http://www.yacy-forum.de/viewtopic.php?p=35322
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3689 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7f56c8d4aa
fixed some seed selection details
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3685 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e602436fda
fixed problem with cluster routing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3684 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0831034e07
fixed non-termination bug for robinson remote crawl peer selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3681 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
68f5d64ae6
replaced yacy logo by better version
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3675 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d6480dc670
fix for long transfer pauses
...
see http://www.yacy-forum.de/viewtopic.php?p=35243#35243
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3672 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
cb43ae11ba
*) Bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3668 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
0b5fc3c28c
*) moving date functions to serverDate class
...
*) Sitemap-parser
- logging added
- parsing of modDate added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3667 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
6f46245a51
*) Bookmarks: Ajax icon is displayed while loading title
...
*) First version of a sitemap parser added
- currently only autodetection of sitemap files is supported
*) DB-Import restructured
- pause/resume should work again now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3666 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
269d5ca45b
*) Bugfix for IllegalMonitorException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3664 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e8610f049a
*) new method allowing the updater to wait until yacy has finished startup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3662 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
111ba9e359
- fixed some width problems in new status page
...
- fixed deadlock in dns cache
- added termination security for DHT peer selection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3660 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
74dd6cac95
*) signal yacy shutdown to updater
...
*) some javadoc added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3658 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
43748f87fb
*) changes required for the uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3655 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
e12e934ade
*) Fixed broken compile process.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3650 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
d6811ac243
*) Moving tar.jar from libx to lib
...
*) Enhanced interface
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3649 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
469583ea80
*) new interface class. should be implemented by the updater to allow communication between the updater and yacy
...
(not yet functional)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3648 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
191ef16499
fixed wrong ordering that caused bad dht selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3646 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7cf8981a98
- added debugging code for wrong DHT target iterator
...
- restricted distance constraint from 0.4 to 0.2
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3644 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
abb63e3289
*) disabled system.exit() in case of YaCy shutdown as it kills the whole VM (including updater and other management threads)
...
*) UpdateCheck-Thread now pauses for given interval correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3642 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dd44a1394f
disabled automatic performance setting change
...
- during crawl start
- each indexing cycle
- for delay values
- for short memory cycles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3634 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b9add5cf37
some bugfixes:
...
- dht iterator start point
- wordIndex synchronization
- surftipps url check
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3633 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
06b6e35484
fix for a null pointer exception if clusters are not defined
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3632 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
35c660654d
more debugging lines to fix bug for
...
http://www.yacy-forum.de/viewtopic.php?p=34935#34935
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3629 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
47e90f31b2
fix for deadlock in plasmaWordIndex.addPageIndex
...
synchronization for class method not necessary
see also: http://www.yacy-forum.de/viewtopic.php?p=34959#34959
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3628 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
81844e85b2
- fixed more cluster routing problems
...
- fixed a problem in remote search when balancer caused shift process to wait too long
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3627 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
304ed3f4d2
fix for remote crawl requests in clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3626 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1979a167d3
fixed problem with cast
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3625 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e48189c710
enhanced cluster routing
...
- cluster definitions can now contain an addition for local ip addresses
- cluster-cluster communication uses the local ip address instead the global address, if one is given
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3624 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e9d87b2fce
*) changes required for the uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3621 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b33cef421e
better routing for public clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3620 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
7c902996b5
*) changes required for the uploaderWrapper
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3618 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f73e1e3af9
fixed bugs in remote search setting for public clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3615 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
485bf1ea83
bugfix for robinson/remote crawl bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3614 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
62c947b4aa
next try to fix deadlock in plasmaWordIndex
...
see also:
http://www.yacy-forum.de/viewtopic.php?p=34821#34821
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3607 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
871ee1ce0f
one step closer to automatic updates:
...
automatically aquire release information from download archives
web pages from latest.yacy-forum.net and yacy.net are retrieved, parsed,
links wihin are analysed, sorted and the most recent developer and main
releases are provided as direct download link on the status page, if it was
discovered that a more recent version than the current version is available.
This process is done only once during run-time of a peer, to protect our
download archives from DoS by YaCy peers.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3606 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ec225f9ab6
*) SOAP: adding methods to get the comment and MD5 checksum of a single file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3604 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
2399ed817c
*) robots.txt parser now extracts the sitemap-URL (will be used later)
...
*) some javadoc added
*) junit testclass for robots.txt parser added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3602 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
fa012789b2
tried to fix a deadlock problem durin shutdown
...
see also:
http://www.yacy-forum.de/viewtopic.php?p=34753#34753
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3601 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e192f616a2
collection of small bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3600 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
64a6d6e5e6
added new set iterator (needed for last commit)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3599 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f8de19fb2f
robinson cluster: added client-side protocol implementation
...
- the network configuration page shows a new option: robinson clusters
- when a global search is made, all robinson peers are excluded, but:
- robinson peers/clusters that provide peer tags and where search words match
such tags, they are included in global search. Therefore, robinson peers/clusters
support the global yacy network with their indexes, without doin DHT-exchange
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3598 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
50e1e61fa5
*) SOAP: adding functions to rename and move files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3595 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
63a004abff
*) bugfix for Nullpointerexception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3594 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
29fe2beac7
possibly fixed a deadlock
...
cannot find forum link now for that
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3593 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
4f4d3d71dd
*) Faster appearance of ConfigBasic by bypassing UPNP-scan in case of existing external connects
...
*) Marked two deprecated source-points
*) Added possibility to dump words from indexing to file. Should not affect performance in the current form.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3592 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago