hermens
5b3e01bd3c
avoid division by zero when importing very small indexes (<100 entries)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1238 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
b7f9adc2c9
new filters added
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1231 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
79667a172e
*) Bugfix for additional parser problem
...
See: http://www.yacy-forum.de/viewtopic.php?p=14146#14146
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1221 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8c594841a8
*) Bugfix for incorrectly indexing of URLs that were requested with Cookies in the
...
Request header
See: http://www.yacy-forum.de/viewtopic.php?p=14077
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1214 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b5d02d649a
fixed bug caused strange search result behaviour
...
(results from remote peers had not been saved propery after search)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1213 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4500506735
fixed some bugs concerning url entry retrieval and intexControl interface
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1212 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
83a34b838d
* added Object allocation monitor on performanceMemory page
...
* added some final statements
* changed shutdown sequence order
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1211 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4ff3d219e8
increased delay for cacheScan start and slowed down scan process
...
to provide more time to other tasks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1210 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3031903d50
re-design of RAM cache flush into assortment cluster
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1209 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0c762daf4b
better startup failure handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1205 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f27f9ecf15
* activated write buffer for databases.
...
This should increase IO performance and reduce HD activity
* bugfixes for new exception-on-failure policy
* bugfixes for new IOChunks
* new Object pool for database write-buffer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1204 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c59d1b2f5e
- Tests with write buffer (new class kelondroBufferedIOChunks, not yet active)
...
- minor bugfixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1203 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bb79fb5d91
- changed handling of error cases retrieving urls from database
...
(no more NULL values are returned, instead, an IOException is thrown)
- removed ugly damagedURLS implementation from plasmaCrawlLURL.java
(this inserted a static value into the Object which is not really a good style)
- re-coded damagedURLS collection in yacy.java by catching an exception and evaluating the exception message
to do:
- the urldbcleanup feature must be re-tested
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1200 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
e7d16ef831
*) Corrections in jMimeMagic MagicRule-file to detect some special rss feeds
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1196 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
386d9e45d8
*) Bugfix for code cleanup
...
- Code must be in finally block, otherwise it does not work if an error occurs!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1193 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
5a1d45715d
*) Bugfix for parser configuration bug
...
- it was not possible to disable all parsers
See: http://www.yacy-forum.de/viewtopic.php?t=1579
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1191 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
a1061495d4
Fixed some spelling mistakes and added some text which (should) make it easier to understand the options.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1187 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0cdc58aaea
fixed indexing of local domains.
...
see http://www.yacy-forum.de/viewtopic.php?p=13680#13680
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1186 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
e1c2d8ec5f
*) Speedup "removed from queue"
...
See: http://www.yacy-forum.de/viewtopic.php?p=13442#12188
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1183 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
96930f0d2b
*)added function to removed malformed URLs from urlHash.db
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1182 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8862b6ba4b
*) Corrections for code cleanup 1175
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1179 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
13fdebc50d
added authentication for link deletion in search result
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1177 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
37f88b4017
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1176 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ec2b39c1ce
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1175 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8f1f2daa5e
implemented interactive link deletion of search results.
...
next steps: attach voting and restrict to administrator
to see the deletion button, move the mouse pointer to the left of a search result
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1172 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6d0f7e6988
*) Adding missing file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1171 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
44fa94ac52
*) Modifications for dbImport functionality
...
- dbImporter threads are now shutdown by the switchboard on server shutdown
- adding possibility to pause a importer thread via GUI
- Bugfix for abort function
See: http://www.yacy-forum.de/viewtopic.php?p=13363#13363
*) Modification of content parser configuration
- now it's possible to configure which parsers should be enabled for the proxy,
crawler, icap, etc. separately
-
*) htmlFilterContentScraper.java
- adding regular expression to normalize URLs containing /../ and /./ parts
*) httpc.java
- adding functionality to unzip gzipped content
- requested by roland: should be used later to allow gzipped seed lists
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1170 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dc778659fb
fixed problem with time-out during result joint which caused OR behavior instead of AND beahvior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1167 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3d8a5ae652
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1166 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
64478b1f02
*) Adding possibility to delete crawler queue entries using regular expressions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1160 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a04930f025
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1158 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
low012
90b0eb144e
just a typo...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1155 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
129b15f3e1
*) Correcting logging output of db importer thread
...
See: http://www.yacy-forum.de/viewtopic.php?t=1555
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1154 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
420d56ce79
extended db-testing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1152 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ecf765ec33
temporary fix to make jrpm extension compilable with my netbeans environment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1151 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8ed0aaae8d
*) Adding content Parser for RPM Files
...
- at the moment only the metadata is extracted
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1147 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
818d37ce44
*) Removing getSimpleName
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1143 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b35c5a48bf
*) First version of urlRedirector.pl script
...
- with this script it's possible to pass URLs from squid
to yacy via the squid redirector interface
- this URLs are then used by YaCy to feed the crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1141 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
bdf30117c1
*) Redesign of parser configuration
...
- restructuring of mimeTypes based on the parsers
- displaying parser usage count
- displaying human readably parser names
- displaying parser version information
*) httpdFileHandler.java
- adding possibility to support "streaming" servlets
which are special servlets that can communicate with
the client via the connection streams autonomous
- the name of these new servlet types must end with the
file extension .stream
- this feature will be needed by the yacy ScreenSaver
class to fetch statistic data from the peer without the
need to reconnect to the server all the time
*) Adding human readable names and version information for
all supported parsers
*) plasmaParser.java
- adding new structure to store parser statistic data
*) Adding openDocument parser
- can be used to parse odt files
*) jmimemagic
- adding rules to detect openDocument formats properly
*) serverLog.java
- adding functions that can be used to query if a given
logging level is enabled or not.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1140 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d4ac3e25b1
*) Bugfix for file system link bug during detection of invalid URLs
...
See: http://www.yacy-forum.de/viewtopic.php?p=13301
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1134 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
adf75bc9fa
better logging for invalid file path detection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1133 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
40621a5663
anhancements in ranking preparation and fixed problem with parser/mime recognition
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1132 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c650b112ea
*) Bugfix for relative URL Bug in Crawler
...
See: http://www.yacy-forum.de/viewtopic.php?p=13266#13266
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1130 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
4e73035aef
*) Bugfix for "too many open files" during index distribution
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1128 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f57e2d67f5
shortened network overview (less columns fit easier on page)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1124 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
85282b1d98
enhanced YBR recognition and search result heuristics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1121 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b9cc9029e3
added ybr selection for remote search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1119 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0e25020f51
added first generation and usage of YBR index-files. Enhanced overall ranking of search results.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1118 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
90d6c6223b
*) Adding color codes to network graphic legend
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1114 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bfe51c7228
added generation of domain-list
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1112 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago