orbiter
c60aed4435
no caching in browser of dynamic web pages sent by YaCy http
...
this may prevent unnecessary IO caused by cache storage of the browser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7207 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e63896f2a8
added an intranet scanner and a servlet which shows all intranet addresses and an option to start a site-crawl for all these addresses at once.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7203 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e54cb7fb0c
more bugfixes (also for latest commit)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7202 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
be6b48311c
misc bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7201 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d2fd93135c
- moved yacybot user agent string definition to MultiProtocolURI since there are basic access mechanisms where the bot string is needed
...
- migrated the 'yacy' user agent to 'yacybot' in many client methods since the 'yacy' user agent is only used for the proxy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7199 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
afa708d552
*) added <s>...</s> tag to WikiCode -> works just as the HTML equivalent
...
*) code changes (PMD) without functional changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7193 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a83186ac7d
fix for bug in cytrails
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7192 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
48c0d508ac
fixes for crawling of smb links (file length not always available)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7190 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0bc6284e27
- added bugfix for access tracker in case of concurrency conflicts
...
- added missing entry for new icu4j path in Mac App
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7188 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
f1ori
e670e1ef8e
add charset auto-detection for htmlParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7186 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
f1ori
ddcd5ae78c
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2989
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7185 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
f1ori
8fe1102452
fix http://forum.yacy-websuche.de/viewtopic.php?p=20889#p18426
...
reuse code from htmlParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7184 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
10a9cb1971
simplified snippet computation process and separated the algorithm into two classes
...
also enhances selection criteria for best snippet line computation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7182 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
4450c240b7
npe fix http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2982
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7181 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
84a023cbc8
fixed several search bugs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7180 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
09c208a3ab
patch for corrupted database files (just work on and forget key)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7177 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
97ee278931
enhanced search speed:
...
- better control of number of running search threads
- no time-out waiting time when no ranking feeding takes place
- local search queries by a remote peer may be faster up to 300 milliseconds
- a local search may even be faster
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7176 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ee3820c9cc
more logging for strange "java.lang.NoClassDefFoundError: de/anomic/http/server/RequestHeader" error
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7175 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
f1ori
b392ca5024
* add option to show YaCy version, usage:
...
java -cp lib/yacycore.jar net.yacy.yacy -version
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7174 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ac73072924
added a demonstration class: integrate the YaCy search results in own applications
...
This class requests a YaCy peer remotely and produces search result objects.
The class was implemented in such a way that it is as short as possible. To get a
better integration of search results, use the cora package.
This class is fully stand-alone, it does not need any other external library other than already contained in JRE.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7173 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
377f001e0d
sorting of crawl profile names in crawl profile editor, see
...
http://forum.yacy-websuche.de/viewtopic.php?p=20851#p20851
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7172 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3552476fbe
terminated migration from apache httpclient-3.1 to 4.1:
...
- remove the library
- added two classes from the httpclient-3.1 library as source code to YaCy because these classes were used by the YaCy HTTP Server
- modified the added classes ChunkedInputStream and ContentLengthInputStream in such a way that:
* there are no more dependencies to httpclient-3.1
* these classes had been simplified to serve only the purpose for the YaCy httpd
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7171 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8da4eb5de6
addition to patch in SVN 7111
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7170 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a2f9974745
some redesign in the access tracker to realize sixcoolers question about "smartes way for deleting the first Object":
...
- not so much abstraction for a collection, makes use of remove() (no operands) possible
- different way to delete elements in track (destructive, not constructive (less copies of elements in new queue))
- more abstraction for class api since no static class must be used any more
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7169 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
03f0414025
some minor correction of my last commit
...
sorry for the noise
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7168 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
42fa0eadb1
fix endless loop:
...
Collection does not support remove(int)
(isn't there a smartes way for deleting the first Object?)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7167 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
5a9ea0308f
*) further simplification of wiki code parser (less redundancy in code, less magic numbers), still not done with it...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7166 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
37baa8bae3
- fixes for concurrency exceptions and failed database integrity verification
...
- added link to yacystats peer when peer is more than one day old
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7164 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
29fe401f93
- some layout and text enhancement for site crawl start
...
- Quix0rs patch from http://forum.yacy-websuche.de/viewtopic.php?p=20839#p20839 (parts)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7163 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
461a2a6ec7
enhanced remote crawling:
...
- 300 ppm is default now (but this is switched off by default; if you switch it on you may want more traffic?)
- better timing for busy queue
- better amount of remote url retrieval
- better time-out values
- better tracking of availability of remote crawl urls
- more logging for result of receipt sending
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7159 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
670ba4d52b
- removed the remote crawl option from the network configuration submenu and
...
- added a remote crawl menu item to the index create menu. This menu also shows a list of peers that provide remote crawl urls
- set remote crawl option by default to off. This option may be important but it also confuses first-time users
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7158 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
89c2d8b81e
better initial hash computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7157 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
34e2f7f487
enhanced snippet fetch strategy: concurrent snippet fetch even for offline-snippet searches. This improves speed since it is now possible to fetch snippets offline and parsing of source files from the htcache can be enhanced using concurrency. This improves local and remote search.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7156 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0cf006865e
refactoring and enhanced concurrency
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7155 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
83ac07874f
- corrected return value of put() methods (not used anywhere, so it did not harm before)
...
- added use of LookAheadIterator which should prevent mistakes when coding iterators with embedded iterators
- added a fail-safe reaction in case of database corruption using iterators over database elements (no interruption then)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7154 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5702419194
fixed a bug in HTTPClient: keep-alive must be set to false, otherwise servers hold connections 2 seconds open until response.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7151 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5870b13f3a
- code cleanup / added debug line for further investigation in HTTPDemon.parseMultipart
...
- changed data structure for sorting in search which performs better in that specific case (too many updates)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7150 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ac1c08924e
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7149 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
14c843d364
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7148 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
39f409a7bb
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7147 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7ebef56add
- redesign of a part of the remote search client to make it possible to have a test environment for remote search performance tests
...
- added a remote search test main methods in yacyClient
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7146 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3c0e07ba72
removed all delays in shutdown process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7143 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
906c572621
- enhanced index create menu structure
...
- clear search log caches each time a search is done
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7142 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
64860dc1bb
enhanced search event logging (to be used for further improvements)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7140 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7dbc357593
patch to identify corrupted database files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7139 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
17eebd4ef8
counting crawler traffic again:
...
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2808
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7138 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
d2a3d08c44
avoid div. by zero
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7136 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2c7edea35e
- better shutdown behavior for the GUI (waits until data is written if GUI is killed)
...
- release 0.97
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7135 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
34a25856a5
- added navigation to next/prev search page using arrow keys (left/right)
...
- better information text for YaCy GUI application
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7134 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
32f73d1aaa
added copy for Info.plist for Mac application release updates (this file contains class paths and start parameters)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7133 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago