orbiter
90d569d70f
refactoring of index management:
...
url storage is part of index management; moved plasmaURL to indexURL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2122 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
f08e33680c
Added Blog-news-symbol as requested.
...
I think I will change the character distance a little bit later.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2101 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
ddfe0f0e27
*) don't try to parse referer string if it's null
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2090 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3e31820c3d
- corrections to PerformanceMemory display of object cache
...
- configuration of object cache size in kelondroTree initializer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2075 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
00a5d435e2
- fixed some bugs with domain filter
...
- added new ranking filter "prefermask": urls that match the filter are ranked better
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2022 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7a650d0023
several bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1971 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
59d52fb4a9
fixed some problems with crawl profiles
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1967 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
63f39ac7b5
added 3 new crawling steering options:
...
- re-crawl by age of page (enter in minutes)
- auto-domain-filter
- maximum number of pages per domain
NOT YET TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1949 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1f4412a146
adopted isListed to discussed new behavior as discussed (url, getFile)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1940 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
488a0ed580
replaced old keyIterator and rowIterator by buffered iterators
...
that are synchronized with database access
Main change is done in kelondroTree, other classes are only adoptions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1918 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dba02f399f
starting of re-design of kelondroTree iterator
...
- new access to iterator
- added many IOException handling in other Classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1914 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f02b426073
made kelondroTree.nodeIterator private
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1910 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
89286478e7
*) removing thread pool eviction for now. Not needed at the moment
...
See: http://www.yacy-forum.de/viewtopic.php?p=18290#18290
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1801 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fbbbf5f411
*) remote trigger for proxy-crawl
...
- remote crawling can now be enabled for the proxy crawling profile
See: http://www.yacy-forum.de/viewtopic.php?p=17753#17753
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1758 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
2da18ab359
*) correcting logging output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1667 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8ffc6e35ad
*) correcting logging output
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1665 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
ab7a911bb3
*) Trying to solve pool not open problem
...
See: http://www.yacy-forum.de/viewtopic.php?t=1798
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1482 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
d665f3c39c
*) fixed Threadnames for stackCrawl-Threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1480 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3d5347bc8e
*) changing loglevel for some messages
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1479 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b9c9eaeb44
*) next try todo a bugfix :-((
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1477 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
4b4b93c413
*) next try todo a bugfix :-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1476 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d9fbad71b9
*) next try todo a bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1475 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6da97bd2e4
*) next bugfix for threadpool problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1474 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
bea2b9edee
*) further redesign of threadpools to solve too many thread problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1473 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
784fd50437
*) more verbose thread names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1471 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
56e4dbeb71
*) displaying current active + current idle threads in PerformanceQueues_p.html now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1470 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
859c6a88f5
*) testing various thread pool eviction settings to avoid outOfMemory - Thread creation problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1467 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
6c02f889f7
Cosmetic changes.
...
Corrected version numbering as described in http://www.yacy-websuche.de/wiki/index.php/De:Versionsnummern
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1453 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b191f06d16
*) Adding additional logging message to locate problems with stackcrawl threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1452 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d9bcd73d93
*) Bugfix for exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1448 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
f5abfe8d57
*) more failsafe threadpools
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1446 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
c4487deba9
Minor changes collected over some time.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1319 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9544c47684
added some UTF-8 handling.
...
hope this will help somehow.. for shure not THE solution to our UTF-8 problem
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1308 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9086261476
refactoring of base64 encoding:
...
the kelondro database needs specific information about the order of
base64-encoded keys. Since no other package depends on base64
(only the httpd uses base64 for encryption, but does not need to encode these strings)
it is good to move base64 encoding to the new ordering classes in kelondro.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1284 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0c762daf4b
better startup failure handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1205 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
37f88b4017
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1176 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3d8a5ae652
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1166 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a04930f025
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1158 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
90b0eb144e
just a typo...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1155 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b35c5a48bf
*) First version of urlRedirector.pl script
...
- with this script it's possible to pass URLs from squid
to yacy via the squid redirector interface
- this URLs are then used by YaCy to feed the crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1141 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d0dfccdb77
*) Making CrawlStacker pool configurable via GUI and config file
...
See: http://www.yacy-forum.de/viewtopic.php?t=1448
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1087 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6f9f8ed8f8
*) Automatic Reset of Stack Crawler DB on startup errors
...
See: http://www.yacy-forum.de/viewtopic.php?t=1432
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1045 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fb766413d1
*) Changes on httpc dns caching
...
- Bugfix: old dns cache did not handle case insensitive hostnames correctly.
- adding a possibility to set domain name patterns defining hostnames that should not be cached by the httpc dns cache
e.g. borg-300.dyndns.org
This can be done by setting the new httpc.nameCacheNoCachingPatterns property
- using httpc.dnsResolve wherever possible within the sourcecode
[httpd.java,plasmaCrawlStacker.java]
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1044 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
00ab4d8723
cleaned, small change, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1026 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
27f180f24b
Update of YaWoStat to 0.2.
...
Now does not try to make 400000! operations to load a 4MB textfile :-/
Program is not finished yet.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1000 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
a98bafb939
Changes to german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@941 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c8a35a0130
*) Adding new connection tracking page (currently only for incoming connections)
...
*) Displaying statistic for incoming connections on status page
*) Bugfix for Loop-Access Bug when trying to access the yacy page while yacy is configured as proxy
See: http://www.yacy-forum.de/viewtopic.php?p=6826
*) Bugfix for Referer Bug
See: http://www.yacy-forum.de/viewtopic.php?p=11098#11098
*) Adding reverse Name lookup for yacy-domain names (used by the connection tracking page)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@916 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
579b22d8ff
small update to network drawing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@892 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
2b5829c3da
small fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@891 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4c7918f5b5
added shotdown to crawl stacker (moved from 882)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@889 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago