orbiter
a1fb8358b2
lets make a well-formed http link so that other crawlers don't have a problem to follow this link :-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3463 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4edb70f68b
added yacybot info-page from Roland
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3462 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3ef77d2030
fix for http://www.yacy-forum.de/viewtopic.php?p=29878#29878
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3461 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3bb3df3fc0
fix for http://www.yacy-forum.de/viewtopic.php?p=32298#32298
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3460 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b3ca177a5d
fix for http://www.yacy-forum.de/viewtopic.php?p=32797#32797
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3459 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
243a2f831b
fixed problem with not found NURL-hashes
...
The cause for this problem could still not be found, but the effect
is handled much better. The NURL-pop will continue automatically until
it found a hash that can be found.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3458 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6ad39bae1e
fixed shutdown problem
...
this fixes the 'inconsistency' messages during start-up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3457 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
38b93f8cb8
bugfix for my last commit:
...
iterator did not consider secondary start point in case of rotation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3456 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
264a82eec8
- fix for http://www.yacy-forum.de/viewtopic.php?t=3657
...
- fix for http://www.yacy-forum.de/viewtopic.php?p=32758#32758
- Diff takes any objects now, not only strings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3455 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
rramthun
045d758537
Avoid stopwords as topwords, configurable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3454 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d755a8026d
- better OOM protection
...
- better memory allocation for FlexTable indexes
- splitting between static index and dynamic index (only the dynamic part must grow)
- to enable a merge-iteration of new splittet index, a huge number of classes needed to be adopted for new iterator classes
- added new iterator classes that support cloneable iterators
- adopted all iterator classes to implement cloneable itarators
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3453 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
2be405e1e1
- fix for last two commits
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3452 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
de1b4a1731
- don't publish news if empty or equal page is submitted in wiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3451 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
dcc13abd59
- fixed small bug at home page, button "peer's console"
...
- fixed <fieldset><dl> for safari on many pages
- added Blog-link to Network page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3450 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
6596167277
*) bugfix for wrong RSS feed pubDate formats
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3449 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
0d178d00a5
*) adding RSS feed for peer messages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3448 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
23338d2070
small fix for RAM computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3447 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
33f97cff7a
changed startup initialization sequence slightly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3446 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
4f2e6ef47b
- WatchCrawler_p shows max. 80 characters of URLs now (maybe dynamically adjustable based on browser width?)
...
- typo in BlacklistCleaner
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3445 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
70cd391ea1
fix for dl/fieldset problem in Safari
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3444 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5741701b59
moved crawl start up, personal web pages down in main menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3443 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b627c77df6
- workaround for safari bug with definition lists inside fieldsets in ConfigBasic
...
- alternative can be seen in PerformanceMemory, where a dl is simulated with a table layout
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3442 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4e8eb1dbe3
some minor changes here and there
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3441 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
03c5906ae7
- minor bugfixes for url-fetcher & http://www.yacy-forum.de/viewtopic.php?t=3646
...
- PerformanceMemory_p.html is valid XHTML again
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3440 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3499a364ef
a little bit better memory protection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3439 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
313f6a7680
fix for http://www.yacy-forum.de/viewtopic.php?p=31553#31553
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3438 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
958ebea5c5
fix for http://www.yacy-forum.de/viewtopic.php?p=32470#32470
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3437 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5d5e6ebfcc
fix for http://www.yacy-forum.de/viewtopic.php?p=32631#32631
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3436 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
8e9bee12fc
*) adding guid to yacysearch.rss
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3435 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1cba31de43
redesigned ram organization for database caches
...
- each cache can now allocate as much memory as is available
- no more fixed limits
- replaced old performance memory monitor by new one
- added supervision methods as static functions into the classes that provide cache functionality
- steering of ram allocation is done with two simple limits that are ram availability-relative
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3434 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e934c5b09b
*) wrong blog rss feed titel
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3433 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ceed0364e2
*) Blog RSS: Image added
...
*) RSS Feed for YaCy Bookmarks added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3432 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
26450a1d9a
*) avoid nullpointerException on seed.getAddress() (reported by netbude)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3431 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
fc43007490
added .homeip.net
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3430 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
db235f2d61
added some memory protection in collection index multiple merge
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3429 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
c72605ecab
*) adding a function to determine if a given URL is bookmarkt
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3428 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
bd03c6b874
*) bugfix in bookmarksDB:
...
- NullpointerException when trying to get an unknown bookmark
- bookmarks can either start with http or https
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3427 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b466baa574
added some memory protection
...
too large collection arrays are now avoided. By default, the biggest
collection index is 7. larger collections are dumped into a commons
directory, but cannot yet be used. Bevore doing a dump, the collection
is splittet into a part which has only root-references, and stored back
to the collection; the remaining part goes to commons
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3426 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
ce360ef43e
*) no more HTML in plasmaCrawlProfile.java anymore
...
*) <br> will not be displayed in items in Auto Filter Content on WatchCrawler_p.html anymore
*) removed unnecessary replaceHTML()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3425 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
93e1ad2bca
- fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3424 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
88245e44d8
- improved version of robots.txt (delete your old htroot/robots.txt before updating):
...
- robots.txt is a servlet now
- no need to rewrite the whole file each time a section is added or removed
- user-defined disallows, added manually, won't be overwritten anymore
- new config-setting: httpd.robots.txt, holding names of the disallowed sections
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
9623bf7bbe
- removed call of java 1.5 method
...
- added config servlet for local robots.txt
- removed YPStats_p as it is of no use anymore
- supertemplates use XHTML now
- quick-fix for http://www.yacy-forum.de/viewtopic.php?p=32296#32296
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3422 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
daburna
f4c13b422c
*updated translation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3421 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
9b33562ed1
*) adding mimetype application/x-rar
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3420 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
51e12049fa
third generation of R/W head path optimization
...
- data from collection arrays are read in order
- merged data is written in order
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3419 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
1fe505f0b0
- adapted User_p to general web-interface style (and removed status-only page on changes)
...
- beautified WikiHelp.html + typos
- IP hasn't been set correctly in Blog.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3418 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
92b6bc0ad2
- fixed wrongly applied replacement of "<" and ">" in Blog and simplified the code a bit
...
- added check, whether active blacklist engine is supported by blacklist cleaner
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3417 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
karlchenofhell
a1d68fe092
- use .class rather than Class.forName for classes in class-path
...
- added Bost's patch for Diff.findDiagonale() from: http://www.yacy-forum.de//files/patch_685.txt
- fixed minor bugs in Blog
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3416 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
10a3c20b8d
some more enhancements to R/W Head path optimization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3415 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f4cfd19835
second Generation of collection R/W head path optimization:
...
- permanent cache flush is switched off. The optimized cache flush
works better if it is a large number of collections that is flushed
together
- the flush size can be configured instead the flush divisor. There is
only one size for all flushes
- collection records that shall be removed during collection transition
(jump from one collection file to another) are now not really removed
but only marked in RAM. add-operations to the collection use these
marked collection spaces
- index bulk write operations are now separated for each file of a kelondroFlex
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3414 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago