theli
28c5687ff9
*) Bugfix for "download of non supported file content" via crawler
...
See: http://www.yacy-forum.de/viewtopic.php?p=10724#10724
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@835 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2b3f964037
*) Bugfix: supportedFileExt Function didn't chop http parameters before trying to detect the file extension
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@834 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
d388292f24
*) adding function for user accounting which is called after each http request
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@827 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
c7b7c69484
*) adding some functions to userDB.Entry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@826 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
d9c08a47f9
*) Minor changes to userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@824 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
af7829df25
*) adding some functions to userDB.Entry
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@823 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
ff1d3d0680
Init of userDB
...
Pagelayout of User_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@822 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
19648702cc
*) adding iterator to userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@819 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
dd59c1ace9
*) Adding first version of yacy user DB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@817 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
595e0c7e56
*) Bugfix for ProxyErrormsg: Wrong base URL
...
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@815 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
9e1485c13b
new Class for UserAccounts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@813 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
9c4306e41e
fixed problem with htcache path
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@811 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
25a59a51ad
fixed problem created with last svn commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@810 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0054d3b1a6
added age in network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@809 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3dd7e90cdd
kbytes instead of bytes in performance settings; new default values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@808 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
1669eaaa1a
fixed svn 805
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@807 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
ca82d690a9
changed in SVN 805 one line too much
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@806 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
4bb1f849a0
Bugfix for http://www.yacy-forum.de/viewtopic.php?t=1233
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@805 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2c7b490e30
memory-logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@804 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
7fc822a59b
changed handling of time-zones
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@801 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
9b7f37fc37
*) Minor changes
...
- more debugging output: storageTime for indexed document is logged now
- saving memory in plasmaParserDocument.java, plasmaWordIndexEntryContainer.java (not a big deal)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@798 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3c1d968d29
fix-fix for 792 and small changes in ftpc/download/dir experiments
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@797 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
b5a8992d29
*) Setting some object fields to final
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@796 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
5f95a1cf62
*) Bugfix for ProxyErrormsg: Wrong http host header
...
See: http://www.yacy-forum.de/viewtopic.php?p=9905#9905
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@795 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
023be89586
*) Bugfix for "Robots.txt wird immer wieder geladen"
...
See: http://www.yacy-forum.de/viewtopic.php?p=10241#10233
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@794 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
35c6c5ead7
*) Bugfix for "Blacklist und Crawlen" Bug.
...
: Crawling continues even if URL is listed in Blacklist
See: http://www.yacy-forum.de/viewtopic.php?p=10279#10279
- missing return statement added. Thanks to allo for the
code review.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@793 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
dc474aa22f
various bug-fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@792 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
9e2fc7e5fe
load balancing of crawl target domains
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@791 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3fcc95a82c
integrated crawl-profiles db in memory-performance monitor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@788 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
72ce36baba
cleanup in kelondroRecords
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@787 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
fe6a6abc0b
*) Adding robots.txt db to Performance Settings for Memory menue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@785 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
3274ae725e
increased cache size of robots database; however, this should be integrated into new memory control
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@784 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
c6d2f50375
changed order of robots and double-check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@783 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
68d5ff2ef1
added stringbuffer in condenser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@782 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
90f02ea455
*) removing metainfo from serverargs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@780 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
70a5681a4f
*) Bugfix for inactive scp seed uploader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@779 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e380d4e55e
cleanup (no functional change)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@778 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
de0a58d79c
no more sync
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@776 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
495bc8bec6
removed cache-control from low and medium priority caches which reduces memory use and computation overhead
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@774 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
1dc94e7753
*) Adding support for gzip content-encoding of http post requests
...
used to transferRWIs and transferURLs.
See: http://www.yacy-forum.de/viewtopic.php?t=1167#10020
*) adding yacyVersion.java containing constants defining yacy versions
that support a given feature.
Needed to determine if a remote peer is able to decode gzip
content-encoded http post bodies properly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@772 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
a1f5027a88
finals;
...
cleaned;
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@770 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
979a3ee3c0
exceptions for better testing of bug http://www.yacy-forum.de/viewtopic.php?p=9852#9852
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@769 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
18d9e1a256
fix for http://www.yacy-forum.de/viewtopic.php?p=10026#10026
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@768 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
1dd7047af5
finals;
...
cleaned;
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@767 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
ef85fce661
change of memory-consumption constants (had been much too low)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@764 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
07f30931ec
various configuration options in memory performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@763 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2f732e32a2
enhancements to memory menue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@762 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e17df64b54
removed IS_ADMIN - feature. This was covered by plasmaSwitchborad.adminAuthenticated
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@760 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
96a5b6e8fb
removed yacy peer types from serverSwitch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@758 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
b990dc1ad1
*) Replacing jsch 0.1.19 lib with newer version 0.1.21
...
*) Replacing PDFBox 0.7.1 lib with newer version 0.7.2
*) Refactoring of classes httpd/httpc/httpHeaders to
make many methods for httpHeader/Requestline parsing
reusable for new icap implementation
*) adding chunked input stream support
- needed by new icap implementation
- needed by future httpc HTTP/1.1 support
*) httpd.java
- moving all connection property contants to class httpHeader
- moving readHeader function to class httpHeader
- moving parseQuery function to class httpHeader
- moving handleTransparentProxy function to class httpHeader
*) httpHeader.java
- adding new fuction to parse the http response line
- adding new function to converte http headers to a string that
can be send to the client
- adding a function that generates a proper url using all parsed
connection properties
*) ICAP Support
- yacy now supports handling of icap response modification requests
- this feature can be used by other icap enabled proxies to contact
yacy as icap server, and to handover the downloaded content to yacy.logging
for indexing
- functionality was successfully tested with squid 2.5Stable 10 + icap patch
- further icap services e.g. URL filtering based on yacy's blacklists are possible
*) plasmaSwitchboard.java
- htcache entries that are still needed for indexing are now properly registered
as in use after system restart
- extended logging: log message now shows parsing and indexing time for each sb. entry
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@757 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
6d1de8abfd
finals; cleaned;
...
Properties;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@756 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
af9021e956
fixed bug with news caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@754 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
14bc880fa4
fixed bug with crashed profile database
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@753 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
71a31f0902
integrated and extended new memory performance menu; found and fixed bug in DHT caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@752 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
fb52a82008
added new performance page for memory settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@751 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
9dfbd93c7b
Updated german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@748 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e6b30911c3
small changes to caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@747 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
42cd2cea65
added final constants, so that other class can reach it;
...
cleaned;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@741 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
900ab97422
change of memory-allocation blocking value for GC prevention
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@740 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0ffca99886
added priority-organization to kelondroRecord cache. This should virtually double the cache capacity.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@738 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2d22626386
automatic switch-off of cache control in kelondroRecords in case that cache is big enough (so that no cache-aging needs to be controled)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@737 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
50a9500035
fixed 100% CPU bug with news queue deletion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@735 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
cddd9aaa33
fixed SERIOUS bug with kelondroStack; affected all stack processing since 729
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@732 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
416c126815
fix for a profile = null problem and new monitor in crawl queue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@730 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
19547f1821
changed node manipulation methods in kelondro core to reduce object allocation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@729 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
debb207a74
removed file sync
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@725 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2148c0cf49
replaced kelondro storage core; much less objects in kelondro cache now; less IO from DB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@724 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
dff96601fe
*) Bugfix for transferURL:
...
URL list index was not incremented properly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@723 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
beefddf0e8
*) Adding option which allows to do a Index-Transfer without deletion of index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@722 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
4036ee812a
Updated german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@721 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
40925f4fb7
*) Improving complete index transfer performance by automatically increasing size of transfered word chunk
...
for fast connections (much similar to normal dht behavior)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@719 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
91ab4d044b
*) Adding automatic retry functionality to complete index transfer function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@718 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a62677f761
*) Adding additional logging output for complete index transfer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@717 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
10e7d6f02b
Bugfix for http://www.yacy-forum.de/viewtopic.php?t=1053
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@713 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
b991d2e7dd
*) Additional logging message for complete index transfer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@712 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3c00c5f6c7
*) Complete Index Transfer
...
See: http://www.yacy-forum.de/viewtopic.php?p=9622
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@711 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f783061414
*) Changing redirection code from 307 to 302
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@710 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a6a8af0f04
*) httpdFileHandler templateCache can now be disabled
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@708 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2cb084d426
*) Complete Index Transfer
...
See: http://www.yacy-forum.de/viewtopic.php?p=9622
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@707 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
d1de71e9f6
*) Suppress stacktrace on proxy error for "No route to host Exception"
...
See: http://www.yacy-forum.de/viewtopic.php?t=1153
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@704 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
56160cbd01
*) Bugfix for "YaCy verzählt sich ..." Bug.
...
See: http://www.yacy-forum.de/viewtopic.php?p=9559
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@701 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
43b42854a0
fix for null-entries and http://www.yacy-forum.de/viewtopic.php?p=8649
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@699 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
801e902795
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@698 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
019cc716db
*) Undoing last changes on yacySeed. Seems not to work properly.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@697 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
177e8af5b7
*) Bugfix for ConcurrentModification in kelondroAbstractRA.writeMap caused by yacySeed.getMap()
...
See: http://www.yacy-forum.de/viewtopic.php?p=9523
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@695 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
d3c923e6b9
*) Bugfix for "ConcurrentModificationException in hello.class"
...
See: http://www.yacy-forum.de/viewtopic.php?t=723
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@694 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
02c242ae22
minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@693 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
44b36d087e
*) Implementing a Cache for the servelet template files (.html)
...
should help to reduce IO
See: http://www.yacy-forum.de/viewtopic.php?t=749
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@690 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
394b4440d2
*) adding isLoggable function to serverLog class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@689 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
73ded2f0b6
*) Trying to fix bug for Seed-Upload-Failed
...
Bug may be caused because of timing issues
See: http://www.yacy-forum.de/viewtopic.php?p=9439
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@688 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3587407039
*) Fixing problems of list operation if index and queue size are both 0.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@687 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
51b48a10e8
*) Suppress stacktrace on proxy error for "ValidatorException: No trusted certificate found"
...
See: http://www.yacy-forum.de/viewtopic.php?t=1110
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@686 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
7809b382bf
*) Bugfix for Blacklist support for https (only initial connect)
...
See: http://www.yacy-forum.de/viewtopic.php?p=9419
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@684 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
7fe8784231
*) URLs pointing to a server having a private ip addess will not be indexed anymore
...
See: http://www.yacy-forum.de/viewtopic.php?p=9408
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@682 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
7c86c36210
undoing one part of the last commit. do not know, why it didn't work...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@681 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
a79913c6ea
updated german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@680 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
718950c5da
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@679 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
0aafb83edc
*) Bugfix for robots.txt isDisallowed Check.
...
Setting path to "/" if it is null or empty.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@677 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2cd695f376
*) Bugfix path-entries of robots.txt were not decoded correctly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@676 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
8260128ee9
changed getFreeSize();
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@675 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f8ad65eae1
*) First trial implementation of robots.txt support
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@674 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
9444852896
*) Correcting problems if the port number was set to -1, e.g. because of the usage of function
...
URL.getPort()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@673 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
0a57fbcde5
Added new HashSet filesInUse;
...
Added new Function getFreeSize();
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@672 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
8cd6a52dd0
Convention
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@671 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
c0e3d18bbf
*) remove import java.lang
...
*) Added Super()
*) replaced startsWith()
*) cleaned
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@670 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
b1cd1fa917
cleaned
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@669 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
da9c6857fb
*) changed a misunderstand, no BUG ;)
...
*) finals and other
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@668 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
fbac053c03
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@665 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
578f36ae18
*) Speedup of indexer. Proxy files will not be enqueued by the cachemanager
...
into the sb-queue anymore if the mimeType or fileExtension is not supported
by the installed parsers.
- Advantage: Avoiding unnecessary enqueueing and dequeueing from queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@664 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
1219ef99f0
*) Bugfix for NullpointerException in yacyDebugMode Init
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@663 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
9300689dde
bugfix *gr*
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@662 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
e09f1fe8e4
*) IfsL: Suppressing stacktraces on further proxy errors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@661 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
6c722706b7
*) Moving yacyDebugMode intialization to switchboard
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@660 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
ebc39a7b9a
minor fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@659 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
e19ededd66
*) prevents problems due to HTML code in profiles (my posting from 05.Sep.2005 http://www.yacy-forum.de/viewtopic.php?t=964 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@657 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
4e07828807
*) httpdProxyHandler.java
...
- harmonizing proxy exception handling
- adding malformed URL + blacklist check for http head method
- adding malformed URL check to http post method
- chunked encoding is now not used anymore for http post if clients
are http/0.9 or http/1.0 clients (same behaviour as already implemented for get)
- now an exception will be thrown on internal httpc errors to force an error output
to the client or a connection close. This should help to fix the "binary data in browser window" bug
*) plasmaSwitchboard.java
- fixing the following Bug
E 2005/09/03 18:02:42 PLASMA Could not index URL http://mis04.de/FAIL/snot.php : null
java.lang.NullPointerException
at de.anomic.plasma.plasmaSwitchboard.processResourceStack(plasmaSwitchboard.java:1000)
at de.anomic.plasma.plasmaSwitchboard.deQueue(plasmaSwitchboard.java:625)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at de.anomic.server.serverInstantThread.job(serverInstantThread.java:95)
at de.anomic.server.serverAbstractThread.run(serverAbstractThread.java:243)
This bug could occure if the cached responseHeader is null
- getting the mimeType now from the parsed document instead of the responseHeader because the
mimeType could have been changed during content parsing (e.g. because of the mimetypeParser)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@656 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
f90f699ab1
missing package line.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@655 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
89d7c6e45e
*) Bugfix: offset of 1 prevented correct replacement of HTML
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@654 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a7256e8f4e
*) Adding X-Forwarded-For Header
...
See: http://www.yacy-forum.de/viewtopic.php?t=1118&highlight=xforwardedfor
*) httpc.java: Bugfix for incorrect http response statuscode parsing
In some situations the statustext whas chopped
*) Adding a lot of fileheaders containing YaCy copyright and license
*) httpd.java: Adding additional debugging http header that should help du detect
the "binary data in browser window" bug.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@653 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
06a451768f
a simple robotsParser.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@652 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
81cb8feb15
back to 649 :/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@651 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
5194511e8e
*) attempt to find bug
...
See: http://www.yacy-forum.de/viewtopic.php?t=1121
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@650 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
8f3d19b331
*) Suppress stacktrace on proxy error for "Connection reset"
...
See: http://www.yacy-forum.de/viewtopic.php?t=1107
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@646 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
6991b9e2b9
*) Suppress stacktrace on crawler error for "Connection reset"
...
See: http://www.yacy-forum.de/viewtopic.php?p=9071
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@645 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a20814291f
*) Bugfix for "Race condition zwischen httpc und switchboard"
...
See: http://www.yacy-forum.de/viewtopic.php?p=9036
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@644 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
286853fd39
*) Bugfix for "YACY hängt sich beim Beenden auf" Bug
...
See: http://www.yacy-forum.de/viewtopic.php?p=8997
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@643 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a47f9238fe
*) Blacklist is now also used by the crawler
...
See: http://www.yacy-forum.de/viewtopic.php?t=1069
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@642 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
dc0a2d4c11
*) Bugfix for Loader Queue:
...
Job count was not displayed correctly
*) IndexingQueue:
- now it's possible to delete single entries from the queue
- now it's possible to clear the whole queue
See: http://www.yacy-forum.de/viewtopic.php?t=995
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@641 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
732a107160
*) Bugfix for "-UNRESOLVED_PATTERN-" Bug on IndexCreateWWWLocalQueue_p.html and "urlEntry.url() == null" Bug
...
- Logging message for "urlEntry.url() == null" is now displayed as info
- IndexCreateWWWLocalQueue_p.html now detects null entries while looping throug the list and removes them automatically
See:
- http://www.yacy-forum.de/viewtopic.php?t=532#8781
- http://www.yacy-forum.de/viewtopic.php?t=639
- http://www.yacy-forum.de/viewtopic.php?t=1071
- http://www.yacy-forum.de/viewtopic.php?t=338
- http://www.yacy-forum.de/viewtopic.php?t=980
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@640 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
33aaffbfc6
*) Displaying content size of each entry in indexing queue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@639 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
022c1ab179
performance fix for yacyDebugMode and useYacyReferer.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@638 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
286442fbc5
do not Use YaCy-Sites as Referer, if useYacyReferer = false
...
http://www.yacy-forum.de/viewtopic.php?p=8896#8896
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@637 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f9eb550fbc
*) Bugfix for NullpointerException in serverAbstractSwitch.setConfig
...
See: http://www.yacy-forum.de/viewtopic.php?t=692#5575
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@636 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
7626823519
BUGFIX for last 'commit'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@635 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
971756e8dd
the delete size is smaller
...
See: http://www.yacy-forum.de/viewtopic.php?t=1084
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@634 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
0471019606
*) IndexCreateIndexingQueue_p.html now also shows indexing jobs that are currently in process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@633 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3dc6845bef
*) Logging error message to logging output if no errormessage can be send to the user by the proxy
...
Note: This is only done if you set the logging level of PROXY to FINE
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@632 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3df5c7a6cf
*) Displaying an proxy error page instead of a white page if the server has closed
...
the connection before yacy was able to receive the http response line
See: http://www.yacy-forum.de/viewtopic.php?p=8866#8866
http://www.yacy-forum.de/viewtopic.php?t=704
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@630 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
cc493ef8c1
Added change from Hermes
...
See: http://www.yacy-forum.de/viewtopic.php?t=1050
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@629 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
4edb5b6f1e
*) Bugfix for "ProxyAccess logging" Bug
...
Loglevel was not set corretly for Proxy.access logger
See: http://www.yacy-forum.de/viewtopic.php?p=8875#8875
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@628 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
7a7254713d
*) Moving Logging directory per default to DATA/LOG
...
See: http://www.yacy-forum.de/viewtopic.php?t=940#7656
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@627 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
bead8a32aa
*) IndexCreate_p.java:
...
Crawler StartURLs will now also added to the errorURL-DB if an error occures on this url
*) kelondroStack.java, plasmaSwitchboardQueue.java
Adding method which returns a list of all entries in the queue. This list is used by IndexCreate_p.java
instead of an iterator to display the indexing-list.
Advantages: avoid concurrent modifications of the list while displaying it.
Speedup because now we have to access only one sync function instead of multiple ones
(one for each entry)
*) IndexCreateIndexingQueue_p.java
Using new list() function of plasmaSwitchboardQueue
*) httpdFileHandler.java
If a servelet returns the special value "LOCATION" the httpFileHandler does a Redirection of
the Browser to the URL specified by the servelet. This can e.g. be used when a http get request is
used insead of a post request, but a refresh should not be allowed.
*) IndexCreateWWWLocalQueue_p.html
Now it's possible to delete single entries of the local crawler queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@626 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
48aaf703cc
*) Adding additional logging output to detect crawling problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@625 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
59b8a98c7e
*) Bugfix for suppressing of stacktrace in log on crawler error "MalformedURLException"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8840
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@623 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
2ee4f9f4e5
*) added escape characters [= and =] to yacyWiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@622 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
c1d7527929
better cache cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@621 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2e6df95786
*) adding toString method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@620 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
ebbd063c92
*) Making mimeTable static final
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@619 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
60e77dcc60
*) Adding additional loglevel finer + finest according to Thread http://www.yacy-forum.de/viewtopic.php?p=8750#8750
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@618 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
fa54b5f38d
cleanup spaces
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@617 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago