theli
c2fe3a1670
*) Updating jMimeMagic Ruleset
...
- to detect some special formated html documents correctly
- adding rule to detect vCards
*) plasmaParser now supports parsing of files that have a supported fileExtension
but a unsupported mimeType because the webserver has set it incorrectly to text/plain
*) Adding vCard new Parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1107 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
88e3234393
fine-tuning of rci-generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1105 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a12759c1bf
first try to implement a rci-computation from cr-files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1103 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4a8e8f269e
refactoring of cr-processing; new kelondro class to handle the attribute file format
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1100 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
24dc0e0760
implemented cr-file processing and further transmission steps
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1099 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
5cd1e9cef4
*) fixed some dirty code, idea analog to bit stuffing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1098 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
022530df7e
small bugfix in kelondroTree
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1097 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9d9a87f445
limited htcache storage length
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1096 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8e308cf50e
*) Possibility to change the server port on-the-fly.
...
- Now it's possible to change the server port without the need to restart the whole server.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1089 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d0dfccdb77
*) Making CrawlStacker pool configurable via GUI and config file
...
See: http://www.yacy-forum.de/viewtopic.php?t=1448
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1087 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3631cb1f6d
*) deleting empty entities during index selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1086 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
ca26aab9b1
*) More debugging output for migrateWords
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1085 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9b35ae9027
*) Correcting wrong % values on IndexTransfer_p page
...
See: http://www.yacy-forum.de/viewtopic.php?p=12646
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1084 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
e6bf9d90a5
*) Fixing Problems with MalformedURLs during Word Selection
...
- removing (lurl.toString() == null) comparison because toString() is never null
- adding (lurl.url() == null) condition because url() is null if we have selected a word entry with
a malformed URL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1083 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
86a9210264
*) indexing queue slots are now configurable via config file
...
See: http://www.yacy-forum.de/viewtopic.php?t=1480
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1081 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3c11d7b81c
*) Bugfix for minimizeUrlDB
...
- function didn't work correctly because of new url hash structure
See: http://www.yacy-forum.de/viewtopic.php?p=12753#12753
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1080 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9913049009
fixed outOfMemory bug caused by loops in kelondroTree during enumeration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1079 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f8f9d509d5
removed dead Code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1078 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
5918d3985e
removed Debug Statements
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1076 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
bbb936b9ea
*) Bugfix for not human readable content of PDFs while viewing the URL Content via GUI
...
- This Bug also affects the snippet generation on non html/text documents
See: http://www.yacy-forum.de/viewtopic.php?t=1472
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1075 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
445e3a620f
*) Avoid rejecting of html content by the crawler when the file extension is not set properly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1074 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a3fd0069f5
fixed bug in kelondroTree node iteration (rotation did not work)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1072 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fd58d5f8e6
*) Adding possibility to specify the interface / IP-Address where YaCy should bind to.
...
- e.g. Port = 192.168.0.1:8080
Port = #eth0:8080
Port = 8080
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1071 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
889de6686c
Migration in yacyVersion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1070 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
444a5a9368
*) Bugfix for Entries with null url in GlobalQueue
...
See: http://www.yacy-forum.de/viewtopic.php?p=12675#12675
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1069 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
3bbb932fa2
Bugfix for nullpointerexception.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1067 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
4976f97956
Logout Function on User.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1065 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
b74e990c74
better Logout function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1064 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
ebac51df52
restore defaultRemoteProfile
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1063 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
4daa4b00e9
logout function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1062 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
5778428455
move cutUrlText to nxTools,
...
max length from URLs(title) on searchpage now 120 chars
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1060 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
9158845c3b
bugfix for snippet text null bytes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1059 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6e81f2580d
try to fix bug with storage of settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1058 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f763923e0a
added missing files for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1057 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9649d08171
*) More tolerant robots parser
...
- converting tabs to spaces
- cutting of '*' in the disallow section
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1056 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
79818a320f
introduced citation-rank transmission protocol and activate transport for anonymisation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1055 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6f8d7d3bcd
*) Adding first version of YaCy bookmarklet
...
- this can be used to easily crawl a webpage which is currently opened in the browser
- to get the bookmarklet javascript simply call http://localhost:8000/QuickCrawlLink_p.html
and drag and drop the link shown to your Browsers Toolbar/Link-Bar.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1053 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
7e0647f692
*) Bugfix for userDB usage during authentication
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1052 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
93cadb47b9
*) More tolerant robots parser for robots-files which missing empty lines between rule blocks
...
See: http://www.yacy-forum.de/viewtopic.php?p=12471
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1048 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
02f8013013
auto-delete of corrupted word files during word-migration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1047 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d2731418bf
added creation of global ranking files and changed url normal form usage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1046 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6f9f8ed8f8
*) Automatic Reset of Stack Crawler DB on startup errors
...
See: http://www.yacy-forum.de/viewtopic.php?t=1432
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1045 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fb766413d1
*) Changes on httpc dns caching
...
- Bugfix: old dns cache did not handle case insensitive hostnames correctly.
- adding a possibility to set domain name patterns defining hostnames that should not be cached by the httpc dns cache
e.g. borg-300.dyndns.org
This can be done by setting the new httpc.nameCacheNoCachingPatterns property
- using httpc.dnsResolve wherever possible within the sourcecode
[httpd.java,plasmaCrawlStacker.java]
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1044 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bc420c62f6
fixed htcache path generation (never change a running system)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1041 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
795f488222
new urlNormalform version
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1040 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c86d801b0f
removed dyndns domains from dns caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1039 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6dc42a2392
detecting of loops in kelondroTree during last/first-Node search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1038 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
17d2830394
see: http://www.yacy-forum.de/viewtopic.php?t=1416
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1037 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
dd24f0252f
*) Searchword highlighting for info page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1036 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
f9fb284fb7
*) Better handling of robots.txt files with incorrect keywords
...
See: http://www.yacy-forum.de/viewtopic.php?p=12292#12292
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1035 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
a1406f4617
urlNormalform: no logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1034 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
72cde1d894
getCachePath: no logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1033 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
1fbd72f9e0
rename "index.html" to "ndx"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1032 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
cd1107d85e
added support for URLs with '?&'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1030 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
5fb2b017cb
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1029 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
60e869f236
bugfix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1028 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
544e4ea90e
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1027 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
00ab4d8723
cleaned, small change, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1026 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b8ceb1ffde
*) Adding better https support for crawler
...
- solving problems with unkown certificates by implementing a dummy trust Manager
- adding https support to robots-parser
- Seed File can now be downloaded from https resources
- adapting plasmaHTCache.java to support https URLs properly
*) URL Normalization
- sub URLs are now normalized properly during indexing
- pointing urlNormalForm function of plasmaParser to htmlFilterContentScraper function
- normalizing URLs which were received by a crawlOrder request
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1024 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
d2507c6081
rename setJunior()... to orJunior()...,
...
added javadoc,
added getPeerType(), setIP(), setPort(String port)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1023 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
e3179a6394
added getOwnSeedFile()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1022 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
a803a509ae
bugfix: port handling in HTCache
...
grogram flow, cleared up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1021 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3d0dfd4df4
*) Using StringBuffer instead of String concatenation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1018 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
452db479cd
*) bugfix: "21" was displayed as "21" in yacyWiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1017 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
cb69047b91
*)cleanup access static methods and fields
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1016 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
56b9f34411
*)removed unused imports
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1015 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5f68b6886b
introduced new url-hashes for better ranking computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1013 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
aadace1285
fixed network image in search performance monitor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1012 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bb369c98de
fixed search result ordering by date
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1011 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
295aff52a3
*)added offline-browsing-support (onlineMode=0)
...
*)online-mode now can be changed in Status.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1010 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4d1e56e4d9
fixed intermission-bug (removed 'break for intermission' of httpd-thread)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1009 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b058ecf0bc
refactoring of image-generation; added experimental PNG encoder (not active now)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1008 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7a9fc653e9
fixed 100% cpu bug in wikiCode
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1007 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1aa707d357
fix for http://www.yacy-forum.de/viewtopic.php?p=11866#11866
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1005 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d42531e1b2
added auto-reset for NURL-DBs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1004 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
5526d2ff73
Basic WatchCrawler.html, which loades the Queues using AJAX.
...
ajax.js: generic Ajax-Functions
xml/queues/indexing_p*: the Backend
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1002 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
92c49b406b
adminAuth with userDB and adminAuthenticated (fix for statuspage)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1001 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
27f180f24b
Update of YaWoStat to 0.2.
...
Now does not try to make 400000! operations to load a 4MB textfile :-/
Program is not finished yet.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1000 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3b5d0eb053
*) Synchronizing robots.txt downloads to avoid parallel downloads of the same file by separate threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@998 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
723e056c48
*) Bugfix for ClassCastException during SessionPool.close
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@996 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
e58e85363d
*) Bugfix for ConcurrentModificationException while operating on seed properties
...
*) Bugfix for YACY database inconsistency (no more elements available in db '...seed.new.db'), re-set of db.
See: http://www.yacy-forum.de/viewtopic.php?p=11836#11836
http://www.yacy-forum.de/viewtopic.php?p=11814#11814
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@995 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d656e2b433
added a memory-profile chart generation to database performance testing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@993 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
ec3af327f7
*) Bugfix for Proxy-Authentication against remote proxy
...
See: http://www.yacy-forum.de/viewtopic.php?p=11804#11804
*) Adding first version of db test for mysql
NOTES:
- db user + db + db table must be created before starting the test
- db table must be empty. Entries can not be updated at the moment
- db connection properties must be changed in the sourcecode at the moment
TODOs:
- accepting connection properties via command line
- implementing update + remove + read operations
- 'maybe' adding code to create db + table if it doesn't exists
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@991 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5b0911d7ea
added new performance menu for search sequence configuration and monitoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@990 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4fa942511b
de-serialized read and write access
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@989 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1ff0ced515
integration of an interface class for abstract access of kelondro indexed structures like kelondroTree and kelondroHashtable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@987 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ada06b0674
bugfix for Networkimage from Hydrox
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@986 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
e3f87f9dab
Bugfix for not parsable Numbers (NumberFormatException when loading users)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@985 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d3ad712418
*) Bugfix for Seed file upload problem via ftpc
...
See: http://www.yacy-forum.de/viewtopic.php?p=11662#11662
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@984 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9a2afe88d4
*) Deactivating unlimited timeout for persistent connections because this
...
could cause problems with clients which do not shutdown persistent
connections properly.
- Setting timeout for idle persistent connections to 30 minutes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@983 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1aa4ba8b62
added post-search filtering of redundant urls (longer than existing cited)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@982 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
8e7b214e41
*) 5 inverted commas are needed instead of only 4 to strongly emphasise a text (closer to Wikipedia's syntax now)
...
*) Tried to fix nested <pre> and [= issues, but did not succeed due to sleepyness. Will continue work when I'm awake and have some time.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@981 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8d827cdb30
tried to fix problems with order of network list by last-seen (which could also improve the network picture)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@980 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
1e6cf9fd41
*) indented texts between escape symbols behave like indented preformatted texts now
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@979 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
446e7e8bef
*) Bugfix for Seed-Upload - Permission denied problem
...
See: http://www.yacy-forum.de/viewtopic.php?p=11648#11648
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@978 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
097009d910
experimental visualization of DHT access during global search (temporary)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@977 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4dcbc26ef1
introduction of search profiles; very experimental
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@976 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6c48c3ce39
*) Bugfix for ArithmeticException during IndexTransfer
...
See: http://www.yacy-forum.de/viewtopic.php?t=1362
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@974 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
fd1b08303e
fixed a bug theli pointed out (indented <pre>-blocks)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@973 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
525c8dcbd4
*) Adding Traffic Statistic for Crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@972 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9a5ab62928
*) Adding yacy specific X-YACY-Index-Control header which can be used by clients
...
to disallow yacy to index the response that belongs to the request where
X-YACY-Index-Contro is set to "no-index"
*) Bugfix for Seed-List download via Remote Proxy.
Now the pragma and cache-control http headers of the request are properly set to "no-cache"
See: http://www.yacy-forum.de/viewtopic.php?p=11639#11639
*) Bugfix for http-Proxy
yacy has ignored "no-cache"- pragma and cache-control http headers that were send in requests.
Now, these request headers are evaluated properly
TODO: Missing evaluation of "no-store" request headers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@971 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
539f9e57aa
*) Disabling RemoteProxy if remoteProxyName is null or empty
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@970 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
5a25ad9109
*) Bugfix for useRemoteProxy4YACY and useRemoteProxy4SSL check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@969 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
02d9af1a70
*) Restructuring and extending of Remote Proxy Support
...
- remote proxy configuration can now be "really" changed on the fly and takes effect immediately
- adding possibility to disable remote proxy usage for yacy->yacy communication
- adding possibility to disable remote proxy usage for ssl
- restructuring proxy configuration so that it is stored in a single place now
*) Adding possibility to import a foreign word DB (or even more of them in parallel)
at runtime into the peers DB
- this can be done by calling IndexImport_p.html
- ATTENTION: please not that at the moment this thread must be aborted via gui
before a normal server shutdown is done.
- TODO: integrating IndexImport Thread into normal server shutdown
- TODO: Adding posibility to import crawl-queues, etc. from foreign peers
- TODO: removing old import function from yacy.java and calling the new routines instead
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@968 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
222607ef0f
*) added <pre>-tag for preformatted text to yacyWiki (as requested by theli)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@967 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
117a424d00
bugfix for sharing png/gif files in WWW/SHARE
...
http://www.yacy-forum.de/viewtopic.php?p=11565
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@966 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
af3060938b
*) Bugfix for manual peer ping functionality
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@965 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6e3201b74d
*) Bugfix in httpc.java
...
- Requestheader was not passed to the underlying post function properly
- Bug seems not to have caused any side-effect until yet
*) Bugfix for manual peer ping functionality
*) Bugfix for UnresolvedPattern Problem if an Exception occurred in a servlet.
See: http://www.yacy-forum.de/viewtopic.php?t=1353
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@963 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
58b670201d
now, changed HTCacheSize needs no restart
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@961 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f97c303ebd
rights for Admin and Proxy.
...
Adminrights are OR(old auth or new).
Proxyrights are AND(you need Proxyrights and a not reached Timelimit)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@960 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
97de600a68
another bugfix for share/www.
...
Now you can Use share/ and not only share/dir.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@958 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
df147d0fb0
rightsmanagement for upload/download User - preparations for proxy and admin User
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@957 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
2dfd6bf36a
fix for networkimage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@956 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ec10220d57
Fix for last Commit: .class Files in htroot, not in the dir of the localized HTML-Files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@955 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
4db2080188
Bugfix for www and share.
...
http://www.yacy-forum.de/viewtopic.php?p=11486
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@954 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
40777556c5
*) Connection Tracking
...
- adding automatic refresh
- accepts new parameter nameLookup which can be used to deactivate
yacy-peer name lookup (because we have problems with this on large seed-dbs)
*) ViewFile
New page that can be used to view
- original content
- plain text content
- parsed content
- parsed sentences
of a webpage specified by there url hash
Mainly for debugging purpose at the moment
*) Robots.txt
Bugfix for if-modified-since usage
TODO: synchronization of downloads to avoid loading the same robots-file
multiple times in parallel by different threads
*) Shutdown
Better abortion of transferRWI and transferURL sessions on server shutdown
*) Status Page
Adding icon to start/stop crawling via status page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@950 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
bcb0d6d5ff
changed setLastSeen(long rd) to setLastSeen();
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@949 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
cdbaf637fb
added
...
- getIP(), getJunior(), getSenior(), getPrincipal();
- setIP(), setJunior(), setSenior(), setPrincipal(), setLastSeen(long rd);
- isPeerOK(), isOnline(String type);
next try to remove hello.class java.util.ConcurrentModificationException:null ;)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@948 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
e642a5d8b7
more constants
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@947 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
7b510b558a
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@946 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
ee7d847869
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@945 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
d77b982083
small fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@944 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
b00cd5640b
bugfix for 'hello.class java.util.ConcurrentModificationException:null'
...
finals
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@943 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
a98bafb939
Changes to german language file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@941 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
43a127ff3a
allow httpsTunnels to other Ports than 443. (if secureHttps=false)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@940 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
93c1cb9226
ipAuth works. thx theli.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@939 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6430fa520e
bugfix for broken HTDOCS
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@938 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
4320425a17
ipAuth (this does not work yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@937 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
95abdeb685
*) Bugfix for nextElement function of URL Enumerator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@936 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6260942590
changed search process: received indexes are now buffered and written to wordIndex after search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@934 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b7e21ec107
*) Adding DB import function which allows to import an foreign yacy DB (from directory PLASMADB)
...
into the DB of an other peer.
ATTENTION: not tested very well. please use this with care and always make a db backup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@932 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
7ee03acce0
new function cutUrlText added to shortens the URLs on IndexMonitor.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@931 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
b88a9584f8
New Errorpage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@928 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b177a80bb7
*) Bugfix for sendRespondError StackOverFlowException problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@927 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
219acc1e8f
*) Bugfix for wrong http version in response to http/1.0 requests
...
See: http://www.yacy-forum.de/viewtopic.php?t=1312
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@926 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bc56a88cc8
further refactoring of search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@925 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
0f2f783e46
no no-cache for mediaExts
...
see http://www.yacy-forum.de/viewtopic.php?p=11210#11210
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@924 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
7ca60f97bf
localization Support for Includes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@923 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d29dfb0a12
refactoring of search / preparation for better search methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@921 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
9ff2bd2f0e
allow surfing, if no timelimit set.
...
thx, theli
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@920 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
0ae166c522
*) Small changes to Index Transfer.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@919 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
461374e175
*) Restricting amount of files that yacy is allowed to open during index transfer/distribution
...
This option is configurable via config file and is set per default to 800
See: http://www.yacy-forum.de/viewtopic.php?p=11137#11137
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@918 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c8a35a0130
*) Adding new connection tracking page (currently only for incoming connections)
...
*) Displaying statistic for incoming connections on status page
*) Bugfix for Loop-Access Bug when trying to access the yacy page while yacy is configured as proxy
See: http://www.yacy-forum.de/viewtopic.php?p=6826
*) Bugfix for Referer Bug
See: http://www.yacy-forum.de/viewtopic.php?p=11098#11098
*) Adding reverse Name lookup for yacy-domain names (used by the connection tracking page)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@916 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
141417821e
limitless Accounts (set timelimit = 0)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@914 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b80b2fbdcc
crawling peers now produce waves in network graphic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@912 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b45ffecd39
log to fix http://www.yacy-forum.de/viewtopic.php?p=11111#11111
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@911 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
10d3627c90
changed word cache flush scheduling and removed possible locks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@910 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dced5c761e
added 'corona' for crawling peers in network graphic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@909 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4cf0d1eb6d
less memory occupation in ImagePainter
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@908 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6bd9348bf1
small change
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@907 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
aae5af9686
imagePainter may now paint arbitrary large circles; small update to network graphics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@906 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
be8e95d86a
debug printlns removed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@905 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f1ff33177d
reset Timelimits on Daychange
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@904 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
54a97a7355
*) IfesL: Suppressing "Broken pipe" stacktrace in log file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@903 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
a9e25c26e1
*) adding new sendRespondError method to httpd which accepts a template include file
...
for individual error messages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@902 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
5605cc8018
TimeLimits
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@901 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6a72f06c40
resizable network picture + greater on click
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@900 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
839db8869c
added high/low priority for index adding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@899 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
3966b38360
*) Bugfix for userDB time limit calculation
...
See: http://www.yacy-forum.de/viewtopic.php?p=10949
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@898 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
1688be8590
*) plasmaSwitchboard.java
...
adding more verbose logging output for db initialization
*) httpdFileHandler.java
adding cache for servlet response methods
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@897 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
e3a586d7bd
*) Using serverByteBuffer instead of ByteArrayOutputStream
...
to speedup httpdFileHandler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@896 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e9eb5e4b56
refactoring of index-entity join methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@895 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
258fd9eb8e
adding missing file for websearch refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@894 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
77ae30063d
refactoring of websearch process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@893 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
579b22d8ff
small update to network drawing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@892 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
2b5829c3da
small fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@891 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e85989510a
update to network image; added disconneced peers by disconnection time and changed colors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@890 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4c7918f5b5
added shotdown to crawl stacker (moved from 882)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@889 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
2851658c2a
re-integrated Martins last change to crawl stacker from svn 882 that I had deleted accidently
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@888 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c83594528c
integrated crawl stacker into thread control
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@887 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
16a49c1c9d
fix for graphics generation bug, see http://www.yacy-forum.de/viewtopic.php?p=10987#10987
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@886 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d666b61b83
fix for news-deletion, see also http://www.yacy-forum.de/viewtopic.php?p=11000#11000
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@885 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
4180c422e8
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@884 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
959eefbc4f
*) Robots.txt parser/ppt
...
cutting of comments at the line end
*) Adding Threadpool for stackCrawl Thread to speedup robots.txt download
and double url checks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@882 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c1c94111b0
added new network picture at Network menu using the new image-servlet method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@880 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d98ed41f31
update to image painter and test on http://localhost:8080/imagetest.png
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@878 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f65c939a60
userDB Auth
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@874 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5153ec0f3e
update to image painter
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@873 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1b2db0b52a
fix for file-share access; damaged some commits before by me :-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@870 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1a5d98cd6d
better imagePainter example and fix for typo http://www.yacy-forum.de/viewtopic.php?p=10920#10920
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@868 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
afc5ef2819
cleaned, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@867 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
60ab32c313
removed obsolete htmlPlotter
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@865 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
06f4ac494a
image-painter test; try with http://localhost:8080/imagetest.png
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@864 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
fb27428674
added restart to Status.html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@863 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
0dcfc25d98
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@862 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
4191b21e73
cleaned, finals, Properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@858 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f6cf3967de
fix for compile-bug in svn 583 (Martin guck mal ob das richtig ist: fifo oder filo-stack?)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@854 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
a2fa75e688
*) Asynchronous queuing of crawl job URLs (stackCrawl)
...
various checks like the blacklist check or the robots.txt disallow check are now
done by a separate thread to unburden the indexer thread(s)
TODO: maybe we have to introduce a threadpool here if it turn out that this single
thread is a bottleneck because of the time consuming robots.txt downloads
*) improved index transfer
The index selection and transmission is done in parallel now to improve index
transfer performance.
TODO: maybe we could speed up performance by unsing multiple transmission threads in
parallel instead of only a single one.
*) gzip encoded post requests
it is now configureable if a gzip encoded post request should be send on
intex transfer/distribution
*) storage Peer (very experimentell and not optimized yet)
Now it's possible to send the result of the yacy indexer thread to a remote peer
istead of storing the indexed words locally.
This could be done by setting the property "storagePeerHash" in the yacy config file
- Please note that if the index transfer fails, the index ist stored locally.
- TODO: currently this index transfer is done by the indexer thread.
To seedup the indexer
a) this transmission should be done in parallel and
b) multiple chunks should be bundled and transfered together
*) general performance improvements
- better memory cleanup after http request processing has finished
- replacing some string concatenations with stringBuffers
- replacing BufferedInputStreams with serverByteBuffer
- replacing vectors with arraylists wherever possible
- replacing hashtables with hashmaps wherever possible
This was done because function calls to verctor or hashtable functions
take 3 time longer than calls to functions of arraylists or hashmaps.
TODO: we should take a look on the class serverObject which is inherited from hashmap
Do we realy need a synchronization for this class?
TODO: replace arraylists with linkedLists if random access to the list elements is not needed
*) Robots Parser supports if-modified-since downloads now
If the downloaded robots.txt file is older than 7 days the robots parser tries to
download the robots.txt with the if-modified-since header to avoid unnecessary downloads
if the file was not changed. Additionally the ETag header is used to detect changes.
*) Crawler: better handling of unsupported mimeTypes + FileExtension
*) Bugfix: plasmaWordIndexEntity was not closed correctly in
- query.java
- plasmaswitchboard.java
*) function minimizeUrlDB added to yacy.java
this function tests the current urlHashDB for unused urls
ATTENTION: please don't use this function at the moment because
it causes the wordIndexDB to flush all words into the
word directory!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@853 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
01db66dc69
implemented image-servlets. the imagetest will stay there only for a limited time. Now images can be generated on-the-fly from servlets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@852 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f6a0e0f162
small bugfix to readFully
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@851 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6d5d0ac801
bugfix for startup problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@850 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5dc0d41900
bugfix in kelondroRA (hint by Martin)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@847 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0c3a20d44f
more + changed log for better understanding of outOfMemory bug and others
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@846 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e5f8163203
fixed a bug with news; news moving could lead to shurtcut loop / 100% CPU; appeared when clicked on a 'Profile' news in Network menu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@845 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
fbb5e36b80
documentation update
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@843 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
cd77078aa0
old Version restored before Release
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@842 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
a4b747fe97
ProxyAccounts based on userDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@841 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
0fd9aa6c6e
*) Bugfix: supportedFileExt Function didn't detect the file extension correctly because of missing conversion to lower case
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@837 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8a33c9b309
*) Bugfix: supportedFileExt Function didn't detect the file extension correctly if there was a dot
...
in one of the parent directories of the file.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@836 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
28c5687ff9
*) Bugfix for "download of non supported file content" via crawler
...
See: http://www.yacy-forum.de/viewtopic.php?p=10724#10724
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@835 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago