orbiter
8c1da27347
- added more comments for user in site crawl servlet
...
- added a disable/enable function in case that 'sitemap' is selected for functions that do (not) apply
- better naming of menu items
- limit default crawl depth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7162 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3f958ccc7e
fixed localization for renamed expert crawl start servlet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7161 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
58b7417a59
- added a new 'easy' crawl start menu which can be used for the special case of loading a complete domain
...
- the previous crawl start servet was renamed to CrawlStartExpert_p
- easy crawl start is now default
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7160 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
461a2a6ec7
enhanced remote crawling:
...
- 300 ppm is default now (but this is switched off by default; if you switch it on you may want more traffic?)
- better timing for busy queue
- better amount of remote url retrieval
- better time-out values
- better tracking of availability of remote crawl urls
- more logging for result of receipt sending
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7159 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
670ba4d52b
- removed the remote crawl option from the network configuration submenu and
...
- added a remote crawl menu item to the index create menu. This menu also shows a list of peers that provide remote crawl urls
- set remote crawl option by default to off. This option may be important but it also confuses first-time users
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7158 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
89c2d8b81e
better initial hash computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7157 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
34e2f7f487
enhanced snippet fetch strategy: concurrent snippet fetch even for offline-snippet searches. This improves speed since it is now possible to fetch snippets offline and parsing of source files from the htcache can be enhanced using concurrency. This improves local and remote search.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7156 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0cf006865e
refactoring and enhanced concurrency
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7155 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
83ac07874f
- corrected return value of put() methods (not used anywhere, so it did not harm before)
...
- added use of LookAheadIterator which should prevent mistakes when coding iterators with embedded iterators
- added a fail-safe reaction in case of database corruption using iterators over database elements (no interruption then)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7154 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f9a27a05e5
migrated to log4j 1.2.16
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7153 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5c67e6ca49
migrated to latest apache commons fileupload 1.2.2
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7152 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5702419194
fixed a bug in HTTPClient: keep-alive must be set to false, otherwise servers hold connections 2 seconds open until response.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7151 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5870b13f3a
- code cleanup / added debug line for further investigation in HTTPDemon.parseMultipart
...
- changed data structure for sorting in search which performs better in that specific case (too many updates)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7150 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ac1c08924e
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7149 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
14c843d364
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7148 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
39f409a7bb
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7147 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7ebef56add
- redesign of a part of the remote search client to make it possible to have a test environment for remote search performance tests
...
- added a remote search test main methods in yacyClient
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7146 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
2e75879504
fix for latest commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7145 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6e4653cf50
remove DoS protection in remote search for intranet hosts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7144 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3c0e07ba72
removed all delays in shutdown process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7143 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
906c572621
- enhanced index create menu structure
...
- clear search log caches each time a search is done
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7142 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
fc924f024e
import of oai sources from a list using a command line interface:
...
if you have a list of oai servers you can import them all using the linux command:
bin/importOAIList.sh <name-of-oai-list-file>
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7141 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
64860dc1bb
enhanced search event logging (to be used for further improvements)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7140 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7dbc357593
patch to identify corrupted database files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7139 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
17eebd4ef8
counting crawler traffic again:
...
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2808
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7138 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
547d5226ae
fix banner reload parameters (were no html errors)
...
adapted default colours
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7137 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
d2a3d08c44
avoid div. by zero
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7136 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
2c7edea35e
- better shutdown behavior for the GUI (waits until data is written if GUI is killed)
...
- release 0.97
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7135 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
34a25856a5
- added navigation to next/prev search page using arrow keys (left/right)
...
- better information text for YaCy GUI application
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7134 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
32f73d1aaa
added copy for Info.plist for Mac application release updates (this file contains class paths and start parameters)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7133 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
5ce679a053
focus search field on load, no click necessary anymore
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7132 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
013926f01c
added 'francais' as language option for default configuration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7131 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4c21d8dc9d
- changed default values for online caution (the pausing may not be necessary any more)
...
- fixed bug in WeakPriorityBlockingQueue
- show favicon faster using pre-loading (same technique as used for fast image search)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7130 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
570ca577c6
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7129 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
348dece62f
redesign of the SortStack and SortStore classes:
...
created a WeakPriorityBlockingQueue as special implementation
of a PriorityBlockingQueue with a weak object binding.
- better abstraction of ordering technique
- fixed some bugs according to result numbering (distinguish different counters in Queue)
- fixed a ordering bug in post-ranking (ordering was decreased instead of increased)
- reversed ordering numbering using a reversed ordering. The higher the ranking number the better (now).
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7128 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
hermens
03eb021568
Fix for byte[] Objects as keys
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7127 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
114bdd8ba7
fixed old sitemap importer which was not able to parse urls containing post elements
...
- removed old parser
- removed old importer framework (was only used by removed old parser)
- added a new sitemap parser in parser framework
- linked new parser with parser access in old sitemap processing routines
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7126 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
b73ea6581d
fix json in case of query includes "
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7125 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c0b08ac59b
slighlty changed way of pdf parser integration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7124 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6d83c7cb62
removed unnecessary Override statements (produces errors in strict validation)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7123 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
6a09f1f7e5
fix dedicated upnp testing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7122 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
848ef6937e
licenses for pdfbox
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7121 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5fe828fa06
- replaced pdfbox and fontbox version 1.1.0 with 1.2.1
...
- added some clear statements that shall clear static cache size within the pdfbox library
- the pdfbox library contains a memory leak; it is unsafe to run a peer with pdf parser permanently on.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7120 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c757a4aa9f
- corrected lifetime computation for search events
...
- made search event cache cleanup concurrent because cleanup may cause index modifications
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7119 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
ec8f1c0446
adapted _debug.bat to regular starter script
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7118 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
5dff8f62c4
fix tray information display for non-windows
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7117 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
24502fe3de
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7116 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
a6c2e9ef0c
add file.encoding=UTF8 to debian init script
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7115 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ffaa9a1c51
avoiding double-loading of the same resource from the web in case that a seond attempt to load the resource is started while the first attempt is still loading the content from the web. This will delay the second attempt to the time when the first attempt has finished with the possible result that the second attempt reads only from the web cache, not from the web.
...
This will also enhance the process of image result display from SVN 7105
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7114 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
fb828f3767
- performance enhancements in search response time using faster query ID computation and an ID cache
...
- code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7113 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago