orbiter
ef82cced01
removed default line 'P2P WEB SEARCH' if no line is given
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5553 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
78b7361937
fixed problem with logging
...
YOU MUST DELETE DATA/LOG TO MAKE THIS WORK! (sorry..)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5552 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f136ddcfd4
*) this change is supposed to prevent the creation of temporary files by Apache Commons Fileupload library in cases where it is not necessary (as proposed by thq in http://forum.yacy-websuche.de/viewtopic.php?f=8&t=1806 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5546 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
94110df85a
moved logging partially to kelondro
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5545 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
024da2916b
refactoring of logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5544 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
83ce65707a
(almost) completed partition of classes in kelondro
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5543 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7ee494fde5
more refactoring of kelondro:
...
- seperated BLOB from table classes
- renamed 'coding' package to 'order'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5542 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
d4281b78da
dynamic memory scale
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5541 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bf93767ec6
refactoring of kelondro database classes
...
(to be continued)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5540 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc27bf8c4c
refactoring of kelondro classes:
...
kelondro shall become independent from other packages.
moved bytebuffer, date and memory to kelondro
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5539 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fe77fc3d62
- added new property setting 'repositoryPath'
...
which can be used to map any path to http://localhost:8080/repository/
This can be used to do an intranet-indexing without the setting of
symbolic links - which does not work in Windows environment.
Now also Windows users can index their file system easily
using the intranet use case.
- fixed some problems with the identification of the alternative
path in DATA/HTDOCS in the httpd file server
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5538 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6cbca1e508
extended last fix, preventing more sorts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5533 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f9672d3f97
applied fix for inefficient put method as recommended by celle, see
...
http://forum.yacy-websuche.de/viewtopic.php?p=12424#p12424
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5532 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
3484e55be4
- small fix for bookmarksDB
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5527 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
6dd52422ea
- added two dialogs to manage bookmark tags in YaCy-UI
...
- fixed renameTag() in bookmarksDB
- added /api/bookmarks/tags/addTag.xml
- added /api/bookmarks/tags/editTag.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5525 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3154926311
some better memory protection and OOM prevention in EcoFS
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5523 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
aaafe05c02
* revert debug change
...
* contains instead of startsWith, because there might me localizied strings
* decode punycode for every domainpart seperately (see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1749 )
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5516 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
5570fa817b
* remove & from openBrowser command (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1728&hilit=#p12321 )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5515 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
419469ac27
added more methods to control the vertical DHT (not yet active .. )
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5514 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
4ef6b15eb8
limit -Xmx setting to 1999m on win32. bigger values would never work.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5513 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dedfc7df7f
removed distinction between DHT-in and DHT-out. This is necessary to make room for the new cell data structure, which cannot use this this distinction in the first place, but will enable the same meaning with different mechanisms (segments, later)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5511 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b74159feb8
preparations to integrate the new 'cell' index data structure
...
(this commit is just to move development files to my other computer, no functionality change so far)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5509 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d399444e49
added debug information to class loader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5508 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5080fc33bf
fix for http://forum.yacy-websuche.de/viewtopic.php?p=12247#p12247
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5506 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
335d6ce8fc
fix for class loading problem
...
see also http://forum.yacy-websuche.de/viewtopic.php?p=12153#p12153
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5505 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
78778df464
*) this should adjust the Dev/Main detection of the updater to the new version numbers (0.7x is Dev, if x != 0)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5504 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b423d0a036
moved all servlets from htroot/xml to htroot/api
...
the file server contains a patch that temporary matches all xml paths to api,
that means all interfaces still work. Please adopt all your interfaces to the new path.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5497 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
91af105373
last changes before release
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5493 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
7eade3f181
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1728
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5489 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d1bace5e4d
enhanced cleanup function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5488 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
cb76d9e0e4
more synchronized in BLOBHeap (will not fix problem with Runtime-Error as reported in forum)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5487 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ff41da613e
removed exception printout during load of snippets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5484 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
814a28775f
removed thread dump writing in case of invocation target exception in httpd (looked bad, not serious)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5483 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bed38a5f8c
fix for uncaught exception in RSSReader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5482 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
05c235de32
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5481 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
7608944081
*) bugfix for REMOTE_HOST environment variable in CGI code (shows hostname of client instead of hostname of YaCy peer now)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5480 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
a6b29cf72c
reverted change of search event processing in SVN 5460. The new code did not work properly,
...
it gave remote search requests too less time
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5479 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9ef77d57f5
added an access control to the search interface using white/blacklists:
...
in the network configuration, you can configure a whiteliste and a blacklist
- blacklistet clients cannot search
- whitelistet client get never any search restrictions
- for all other clients: apply DoS search restrictions
Please see the example configuriation in yacy.network.freeworld.unit
by default, all clients from localhosts get whitlistet.
If you have your own YaCy network, please put all the IPs of your peers into the whitelist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5475 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
efe801173c
better dht-in cache flush. see also:
...
http://forum.yacy-websuche.de/viewtopic.php?p=11936#p11936
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5472 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
941ab78d9b
better termination for blocking threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5471 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
3dc208fad0
bugfix: bookmarks can now handle folder names like /news and /newspaper without getting confused...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5470 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e948df68ac
longer timeout for queues during shutdown
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5469 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2b32248079
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1516&p=10545#p10545
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5468 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
c1330f5743
*) added environment variable DOCUMENT_ROOT
...
*) caught exception
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5466 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f26b8fcb1b
*) comment mode is 'moderated' instead of 'activated' by default now (to avoid spam being visible)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5465 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b2a8c653ee
small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5464 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f675d47f86
better protection against database failures
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5463 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4f45605f04
small update for timing in search result processing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5460 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9d119c6b61
migration of auto-update rules to new release strategy:
...
next stable will be 0.7, development releases are 0.*x, experimental will be if x = 1, 2, 3
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5458 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4d5b401f00
try to fix some performance problems with the internal index management:
...
- ensuring that ordered indexes stay ordered during remove
- no unnecessary ordering checks
- better test logic in crawl stacker
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5457 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
a0605325bb
fixed a NullPointer Exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5452 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b2b7edae18
fixed interactive search
...
- added dummy servlet class, because otherwise the template engine is not triggered.
thats so because the yacy httpd works much faster as normal file server without a scan
of the served pages. Therefore each page with templates must now have a class file associated to it.
- fixed json output format of yacysearch
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5449 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
2be119f0df
adjusted big peer to 28M links
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5448 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c6880ce28b
removed the permanent cache flush and replaced it with a periodic cache flush
...
The cache is now flushed only for one second every ten seconds. During a crawl the cache
fills up completely, and is only flushed if space is needed for more documents.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5446 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ef7fe537c5
fixed a cache-bug in cachedFileRA
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5445 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6c7e83909b
- refactoring of data access methods to be prepared for new cell data structure
...
- removed a memory overhead in collections which prevent OOM Exception in low memory configurations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5443 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
c8451614f3
fix for overflow
...
http://forum.yacy-websuche.de/viewtopic.php?p=11696#p11696
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5440 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c4c4c223b9
fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6072831235
no cr transmission for robinson peers
...
see also: http://forum.yacy-websuche.de/viewtopic.php?p=10290#p10290
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5436 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
afe98bc11c
*) added changes as proposed by Halborinda in http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1674
...
*) changed indention
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5434 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
07fc115e90
removed active profiling in kelondroRowSet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5433 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
be4c458951
refactoring (implemented Iterable in kelondroRowCollection)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5432 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
bb5c2cd12e
*) ISINDEX parameters will not be put on commandline anymore to prevent possible security hazards (better safe than sorry). Parmeters will have to be read from QUERY_STRING in ISINDEX case too which does not seem to be uncommon behaviour for web servers: http://vms.pdv-systeme.de/users/martinv/cgi_basics/cgi_basics.html#Datenuebergabe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5431 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b6bba18c37
replaced the storing procedure for the index ram cache with a method that generates BLOBHeap-compatible dumps
...
this is a migration step to support a new method to store the web index, which will also based on the same data structure. made also a lot of refactoring for a better structuring of the BLOBHeap class.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5430 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
db1cfae3e7
*) cleaning up after myself
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5429 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f547f9a78c
*) added CGI capabilities (run Perl scripts and other software via HTTP GET and POST)
...
*) set cgi.allow to true in yacy.conf to enable CGI (CGI is disabled by default)
*) edit cgi.suffixes in yacy.conf if necessary to use additional script types
ATTENTION: This is a rather experimental feature, not all environment variables are set yet.
Only enable CGI if you know what you are doing. Poorly implemented CGI scripts can put a system's integrity at risk!
Implementation of more environment variables and documentation due for the next days.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5428 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
bdc380cd84
* add lastModified to templateCache
...
-> no outdated files from cache anymore...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5427 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
025094675f
* remove empty directory
...
* add necessary dependency for pdfParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5424 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
c5691180cb
* skip style-tags in HTML-files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5423 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3567c58b18
added another filed information for BLOBHeap dumps: the gaps
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5420 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
abdd4aa414
added a index dump for blob heaps:
...
this will increase the shutdown time for at most some seconds, but will speed up the start-up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5419 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8c3205b62e
fix for OOB Exception
...
see http://forum.yacy-websuche.de/viewtopic.php?p=11598#p11598
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5417 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
78c568331e
added test channel to /xml/feed.rss
...
can be obtained with
http://localhost:8080/xml/feed.rss?set=TEST
returns always a single feed entry with a fresh date
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e004da48d3
- added fast fingerprint computation for files (any). Will be used in new index dump method
...
- refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2d2ce24011
* remove all encoding-stuff from proxy
...
encoding is handled by parsers or browser, proxy only passes through
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
73c8a0839c
* abort download, when proxy connection is closed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5409 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bb935fdbb0
less organization overhead for DNS caching and prefetching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
4907697cfa
* make fileuploads through proxy bigger than 65500 bytes possible
...
* remove gzip-encoding for files from cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc8189f3fb
better self-healing of corrupted databases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
963da8c3f9
* updated tm-extractors to new version 1.0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e34ac22fbd
- added new monitoring servlet at
...
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
449e697436
fix for null-seed in seedfile
...
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d376d81fc4
replaced busy thread control of crawl stacker by blocking threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f29b48d9ff
patch for IndexOutOfBoundsException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
0881190b19
* Robots.txt: don't interpret Crawl-Delays for other robots
...
fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
243e73f53b
removed unnecessary usage of kelondroBLOBTree
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8cb7170b75
- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated
...
- removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7535fd7447
- refactoring of CrawlEntry and CrawlStacker
...
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
18513e2ee2
npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2802138787
- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)
...
- fix of shallBeOwnWord target computation heuristic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
db6b3bf5a3
speed enhancement for integrated http server:
...
- tuning hacks in template engine
- bypassing the template engine if no servlet present
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7cd08bd5fb
fix for NPE in BLOBCompressor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5b94498643
fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1779c3c507
- added a read cache to the RAFile interface to RandomAccessFile
...
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e1acdb952c
fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4a2dac659e
more speed hacks:
...
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
47292e696a
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
759cef23dd
fix for bug in kelondroAbstractRA.readFully
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5378 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d39d420b39
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0b4808ba3d
added new interactive search feature:
...
- during the user types search queries, the local database is searched
- results are presented interactively
This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago