theli
339153d40e
*) favicons that are specified in the document content via html link-tags
...
are now detected and displayed on the search page (requested by allo).
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3845 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
6265d321bd
- more constants
...
- display why global search is not available on search page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3839 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
18a5380ee3
*) situation-dependent lock-buttons for search-page
...
*) removed one unused import and a double definition of "ogg" as media-type
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3817 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
9d6605a83c
- fixed NPE in Blacklist Cleaner during deletion of more than one double entries
...
- don't display responseHeader1.db in CacheAdmin_p anymore
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3814 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
594ff95955
:-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3801 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4ca797401e
fix for ConcurrentModificationException
...
see http://www.yacy-forum.de/viewtopic.php?p=36566#36566
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3800 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7b904e0077
integrated robots.txt crawlDelay into the crawl balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3797 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
52cb033f01
- slightly different painting of web structure picture:
...
hosts that have many own connections are painted farer away (this is not yet cato's idea, this will be implemented in another step)
- doc update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3796 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
6c9df13552
more debugging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3791 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
d1e1580223
Surftips Blacklist
...
Blacklists List Hardcoded instead of only updated on firststart / migration.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3788 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
94cc9f05f5
*) Improvements for restart via update wrapper
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3785 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
2ab020445a
bugfix, i think - http://www.yacy-forum.de/viewtopic.php?t=4059
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3777 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
ef24bed406
Sorry...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3760 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
(no author)
a29cb2e1af
blupp
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3759 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a585b4d41b
added web structure image
...
see http://localhost:8080/WatchWebStructure_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3747 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
33ad0c8246
added a web structure computation and logging:
...
- all web page parsing operations will now increase a web structure file
- the file is computed in memory and dumped at shutdown-time to PLASMASB/webStructure.map in readable form (not a database)
- the file can be used externally to analyse the link structure of the crawled pages
- the web structure can also be retrieved using a xml-interface at http://localhost:8080/xml/webstructure.xml
- the short-term purpose is the computation of a link-graph image (before linuxtag!)
- a long-term purpose could be a decentralized computation of the citation rank
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3746 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
7904175338
- sorry for typos
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3743 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
baa9402b97
- wiki-parser is now configurable via the config setting wikiParser.class which holds the class-name for the parser to use
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3742 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
0a64047081
- plasmaParserDocument can process subdocuments now (other archive-parsers may want to use this method)
...
- added 7zip parser
- added 'text/sgml' to realtime parseable mimetypes (sometimes returned by the mime type parser)
- added new cached output stream class, very suitable for parsers because of limited memory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3740 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
9a4375b115
*) robots.txt: adding support for crawl-delay
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3737 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
086239da36
- added servlet: remote crawler queue overview
...
- added servlet: crawl profile editor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3731 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b05e2314cf
another dht selection fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3725 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b28e5d0ee9
protection against wrong word hash length
...
see http://www.yacy-forum.de/viewtopic.php?p=35657#35657
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3723 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0384b8771b
fix for http://www.yacy-forum.de/viewtopic.php?p=35700#35700
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3719 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
578c2ef130
release 0.52
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3715 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
46367afaaa
update of memory-protection values
...
see http://www.yacy-forum.de/viewtopic.php?p=35539#35539
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3709 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
ea87fe5d78
*) Updated German translation
...
*) Changed "Lost Handle" error to warning (masses of it if deleting crawl-profile)
*) Removed unnecessary code from Windows script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3708 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
26f05d1fd0
avoid division by zero if search is done for no words
...
this case is relevant if the bluewords (yacy.blue) are used
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3698 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
139c59ebbd
- fixed dht selction problem: the seed tables used a wrong ordering
...
- cleaned some code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3693 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e602436fda
fixed problem with cluster routing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3684 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d6480dc670
fix for long transfer pauses
...
see http://www.yacy-forum.de/viewtopic.php?p=35243#35243
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3672 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
6f46245a51
*) Bookmarks: Ajax icon is displayed while loading title
...
*) First version of a sitemap parser added
- currently only autodetection of sitemap files is supported
*) DB-Import restructured
- pause/resume should work again now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3666 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
74dd6cac95
*) signal yacy shutdown to updater
...
*) some javadoc added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3658 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
43748f87fb
*) changes required for the uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3655 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
e12e934ade
*) Fixed broken compile process.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3650 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7cf8981a98
- added debugging code for wrong DHT target iterator
...
- restricted distance constraint from 0.4 to 0.2
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3644 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
dd44a1394f
disabled automatic performance setting change
...
- during crawl start
- each indexing cycle
- for delay values
- for short memory cycles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3634 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b9add5cf37
some bugfixes:
...
- dht iterator start point
- wordIndex synchronization
- surftipps url check
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3633 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
06b6e35484
fix for a null pointer exception if clusters are not defined
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3632 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
47e90f31b2
fix for deadlock in plasmaWordIndex.addPageIndex
...
synchronization for class method not necessary
see also: http://www.yacy-forum.de/viewtopic.php?p=34959#34959
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3628 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
81844e85b2
- fixed more cluster routing problems
...
- fixed a problem in remote search when balancer caused shift process to wait too long
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3627 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
304ed3f4d2
fix for remote crawl requests in clusters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3626 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e48189c710
enhanced cluster routing
...
- cluster definitions can now contain an addition for local ip addresses
- cluster-cluster communication uses the local ip address instead the global address, if one is given
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3624 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
485bf1ea83
bugfix for robinson/remote crawl bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3614 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
62c947b4aa
next try to fix deadlock in plasmaWordIndex
...
see also:
http://www.yacy-forum.de/viewtopic.php?p=34821#34821
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3607 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
871ee1ce0f
one step closer to automatic updates:
...
automatically aquire release information from download archives
web pages from latest.yacy-forum.net and yacy.net are retrieved, parsed,
links wihin are analysed, sorted and the most recent developer and main
releases are provided as direct download link on the status page, if it was
discovered that a more recent version than the current version is available.
This process is done only once during run-time of a peer, to protect our
download archives from DoS by YaCy peers.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3606 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
2399ed817c
*) robots.txt parser now extracts the sitemap-URL (will be used later)
...
*) some javadoc added
*) junit testclass for robots.txt parser added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3602 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
fa012789b2
tried to fix a deadlock problem durin shutdown
...
see also:
http://www.yacy-forum.de/viewtopic.php?p=34753#34753
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3601 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e192f616a2
collection of small bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3600 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f8de19fb2f
robinson cluster: added client-side protocol implementation
...
- the network configuration page shows a new option: robinson clusters
- when a global search is made, all robinson peers are excluded, but:
- robinson peers/clusters that provide peer tags and where search words match
such tags, they are included in global search. Therefore, robinson peers/clusters
support the global yacy network with their indexes, without doin DHT-exchange
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3598 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago