orbiter
91af105373
last changes before release
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5493 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
7eade3f181
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1728
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5489 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d1bace5e4d
enhanced cleanup function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5488 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
cb76d9e0e4
more synchronized in BLOBHeap (will not fix problem with Runtime-Error as reported in forum)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5487 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ff41da613e
removed exception printout during load of snippets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5484 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
814a28775f
removed thread dump writing in case of invocation target exception in httpd (looked bad, not serious)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5483 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bed38a5f8c
fix for uncaught exception in RSSReader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5482 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
05c235de32
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5481 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
7608944081
*) bugfix for REMOTE_HOST environment variable in CGI code (shows hostname of client instead of hostname of YaCy peer now)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5480 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
a6b29cf72c
reverted change of search event processing in SVN 5460. The new code did not work properly,
...
it gave remote search requests too less time
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5479 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9ef77d57f5
added an access control to the search interface using white/blacklists:
...
in the network configuration, you can configure a whiteliste and a blacklist
- blacklistet clients cannot search
- whitelistet client get never any search restrictions
- for all other clients: apply DoS search restrictions
Please see the example configuriation in yacy.network.freeworld.unit
by default, all clients from localhosts get whitlistet.
If you have your own YaCy network, please put all the IPs of your peers into the whitelist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5475 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
efe801173c
better dht-in cache flush. see also:
...
http://forum.yacy-websuche.de/viewtopic.php?p=11936#p11936
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5472 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
941ab78d9b
better termination for blocking threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5471 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
apfelmaennchen
3dc208fad0
bugfix: bookmarks can now handle folder names like /news and /newspaper without getting confused...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5470 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e948df68ac
longer timeout for queues during shutdown
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5469 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2b32248079
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1516&p=10545#p10545
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5468 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
c1330f5743
*) added environment variable DOCUMENT_ROOT
...
*) caught exception
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5466 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f26b8fcb1b
*) comment mode is 'moderated' instead of 'activated' by default now (to avoid spam being visible)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5465 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b2a8c653ee
small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5464 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f675d47f86
better protection against database failures
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5463 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4f45605f04
small update for timing in search result processing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5460 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9d119c6b61
migration of auto-update rules to new release strategy:
...
next stable will be 0.7, development releases are 0.*x, experimental will be if x = 1, 2, 3
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5458 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4d5b401f00
try to fix some performance problems with the internal index management:
...
- ensuring that ordered indexes stay ordered during remove
- no unnecessary ordering checks
- better test logic in crawl stacker
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5457 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
borg-0300
a0605325bb
fixed a NullPointer Exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5452 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b2b7edae18
fixed interactive search
...
- added dummy servlet class, because otherwise the template engine is not triggered.
thats so because the yacy httpd works much faster as normal file server without a scan
of the served pages. Therefore each page with templates must now have a class file associated to it.
- fixed json output format of yacysearch
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5449 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
2be119f0df
adjusted big peer to 28M links
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5448 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c6880ce28b
removed the permanent cache flush and replaced it with a periodic cache flush
...
The cache is now flushed only for one second every ten seconds. During a crawl the cache
fills up completely, and is only flushed if space is needed for more documents.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5446 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ef7fe537c5
fixed a cache-bug in cachedFileRA
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5445 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6c7e83909b
- refactoring of data access methods to be prepared for new cell data structure
...
- removed a memory overhead in collections which prevent OOM Exception in low memory configurations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5443 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
c8451614f3
fix for overflow
...
http://forum.yacy-websuche.de/viewtopic.php?p=11696#p11696
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5440 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c4c4c223b9
fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
6072831235
no cr transmission for robinson peers
...
see also: http://forum.yacy-websuche.de/viewtopic.php?p=10290#p10290
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5436 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
afe98bc11c
*) added changes as proposed by Halborinda in http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1674
...
*) changed indention
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5434 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
07fc115e90
removed active profiling in kelondroRowSet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5433 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
be4c458951
refactoring (implemented Iterable in kelondroRowCollection)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5432 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
bb5c2cd12e
*) ISINDEX parameters will not be put on commandline anymore to prevent possible security hazards (better safe than sorry). Parmeters will have to be read from QUERY_STRING in ISINDEX case too which does not seem to be uncommon behaviour for web servers: http://vms.pdv-systeme.de/users/martinv/cgi_basics/cgi_basics.html#Datenuebergabe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5431 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b6bba18c37
replaced the storing procedure for the index ram cache with a method that generates BLOBHeap-compatible dumps
...
this is a migration step to support a new method to store the web index, which will also based on the same data structure. made also a lot of refactoring for a better structuring of the BLOBHeap class.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5430 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
db1cfae3e7
*) cleaning up after myself
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5429 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f547f9a78c
*) added CGI capabilities (run Perl scripts and other software via HTTP GET and POST)
...
*) set cgi.allow to true in yacy.conf to enable CGI (CGI is disabled by default)
*) edit cgi.suffixes in yacy.conf if necessary to use additional script types
ATTENTION: This is a rather experimental feature, not all environment variables are set yet.
Only enable CGI if you know what you are doing. Poorly implemented CGI scripts can put a system's integrity at risk!
Implementation of more environment variables and documentation due for the next days.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5428 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
bdc380cd84
* add lastModified to templateCache
...
-> no outdated files from cache anymore...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5427 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
025094675f
* remove empty directory
...
* add necessary dependency for pdfParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5424 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
c5691180cb
* skip style-tags in HTML-files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5423 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3567c58b18
added another filed information for BLOBHeap dumps: the gaps
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5420 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
abdd4aa414
added a index dump for blob heaps:
...
this will increase the shutdown time for at most some seconds, but will speed up the start-up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5419 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8c3205b62e
fix for OOB Exception
...
see http://forum.yacy-websuche.de/viewtopic.php?p=11598#p11598
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5417 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
78c568331e
added test channel to /xml/feed.rss
...
can be obtained with
http://localhost:8080/xml/feed.rss?set=TEST
returns always a single feed entry with a fresh date
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e004da48d3
- added fast fingerprint computation for files (any). Will be used in new index dump method
...
- refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2d2ce24011
* remove all encoding-stuff from proxy
...
encoding is handled by parsers or browser, proxy only passes through
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
73c8a0839c
* abort download, when proxy connection is closed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5409 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bb935fdbb0
less organization overhead for DNS caching and prefetching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
4907697cfa
* make fileuploads through proxy bigger than 65500 bytes possible
...
* remove gzip-encoding for files from cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc8189f3fb
better self-healing of corrupted databases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
963da8c3f9
* updated tm-extractors to new version 1.0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e34ac22fbd
- added new monitoring servlet at
...
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
449e697436
fix for null-seed in seedfile
...
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d376d81fc4
replaced busy thread control of crawl stacker by blocking threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f29b48d9ff
patch for IndexOutOfBoundsException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
0881190b19
* Robots.txt: don't interpret Crawl-Delays for other robots
...
fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
243e73f53b
removed unnecessary usage of kelondroBLOBTree
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8cb7170b75
- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated
...
- removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7535fd7447
- refactoring of CrawlEntry and CrawlStacker
...
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
18513e2ee2
npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2802138787
- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)
...
- fix of shallBeOwnWord target computation heuristic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
db6b3bf5a3
speed enhancement for integrated http server:
...
- tuning hacks in template engine
- bypassing the template engine if no servlet present
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7cd08bd5fb
fix for NPE in BLOBCompressor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5b94498643
fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1779c3c507
- added a read cache to the RAFile interface to RandomAccessFile
...
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e1acdb952c
fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
4a2dac659e
more speed hacks:
...
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
47292e696a
more performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
759cef23dd
fix for bug in kelondroAbstractRA.readFully
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5378 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d39d420b39
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0b4808ba3d
added new interactive search feature:
...
- during the user types search queries, the local database is searched
- results are presented interactively
This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
74a3d86114
fixed a error response that might present classified information
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5372 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
c6525ab75f
fix for NPE in seed handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
1951d30a62
addendum to last commit
...
handle words with length < 3 correctly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
325ba7bfb8
only query words with length > 2
...
this is not complete, yet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
e423fa9846
*) added method to only get file names in directory listing which match a filter
...
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
513179f404
changed interface to colletctionIndex and adopted all implementing classes:
...
do not return a result of a double-check when adding entries with addUnique
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5363 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
9d64693cfb
reverting again the changes to new concurrent chunkIterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5362 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
45ad1c3dd5
- re-activated concurrent iterator for EcoFiles
...
- added javadoc for new concurrent intialization in kelondroBytesLongMap
- switched default value for commons storage to false
- version step
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5361 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2e2120046f
speed enhancement for BLOBHeap opening process
...
using concurrency of FileIO and content processing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5360 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fa26a8f25a
fix for deadlock-like behavior in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5358 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1918a0173e
added more exception handling during crawling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5357 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
10f5ec1040
reverted last commit (more testing needed)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5356 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
5af8923f37
* distribute forgotten jar-file in parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5355 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b0f2003792
fast database initialization and fast start.up of yacy:
...
- applied knowledge about concurrent files stream reading and index processing from the wikimedia reader
to the EcoTable initialization process: the file reader is now concurrent to the index generation
- changed also some initialization processes to avoid some pauses during initialization
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5354 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
0ca4bc7b79
- added reader and visualization for mediawiki-export files:
...
files exported from mediawiki using the xml schema according to
http://www.mediawiki.org/xml/export-0.3/
can be processed to be viewed in a YaCy servlet.
To acces such a file, place it into
DATA/HTCACHE/mediawiki/
i.e. the export from german wikipedia would be:
DATA/HTCACHE/mediawiki/wikipedia.de.xml
This file can then be accessed using the URL
http://localhost:8080/mediawiki_p.html?dump=wikipedia.de.xml&title=YaCy
if this is done the first time, an index file is created
(for this case: more than 4 million lines must be written, this takes about 15 minutes)
Then try the same url again.
- enhanced also the md5 computation speed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5352 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
2e63f03ca5
copy&paste vergessen :/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5351 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
danielr
cd8082b4e3
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111#p11166
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5350 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
4f996a7651
fix for logparser pattern
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5349 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d18c18971e
* dirlisting in UTF-8 encoding
...
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1550&hilit=#p11108
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5348 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
867d0f2f56
removed some unnecessary pause delays
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5346 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
d49ffcd818
* files distributed by yacy are utf-8, files from repository use the system default charset
...
* fixes http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1564#p11092
and http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1550
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5345 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8c96bc2ac1
do not use proxy caching rules for crawling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5344 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
dba7ef5144
extended crawling constraints:
...
- removed never-used secondary crawl depth
- added a must-not-match filter that can be used to exclude urls from a crawl
- added stub for crawl tags which will be used to identify search results that had been produced from specific crawls
please update the yacybar: replace property name 'crawlFilter' with 'mustmatch'.
Additionally, a new parameter named 'mustnotmatch' can be used, which should be by default the empty sring (match-never)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5342 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
96174b2b56
more debugging / better result status logging for parser/caching errors
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5341 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
90e78b2cf6
* improve encoding detection of http service
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5337 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
ef66438662
- more space in error db to store larger error messages
...
- added hash to HTCACHE storage files which will make it possible to join separate caches by just copying files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5329 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
674ad2d55b
different handling of error cases that occur during loading files with http or ftp:
...
methods throw exception instead of returning an error string
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5328 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago