theli
241b881560
*) Redesign of YaCy SOAP handler
...
- should be more fail-safe now
- better handling of compressed request bodies
- better handling of persistent connections
- better handling of AxisFaults
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2659 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
009a33170b
*) Content-Location header added
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2658 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1aa07a52cd
*) Bugfix for UnsupportedEncodingException if the media type contains multiple parameters
...
See: http://www.yacy-forum.de/viewtopic.php?p=25832#25826
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2654 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
625c2ce6b1
*) bugfix for snippet fetching problem if content but not http header is available in cache
...
See: http://www.yacy-forum.de/viewtopic.php?p=25748
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2651 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
813a8a8179
*) migration of mimeTypeParser to jmimemagic 0.1
...
- better mimetype detection for rss feeds
- better mimetype detection for odt documents (less memory consuming)
- two new detector classes implementing MagicDetector interface of jmimemagic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2650 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hermens
3f5a4153a0
Make Peers more receptible to transferred indexes
...
- Set MaxWordCount for dhtInCache to indexDistribution.dhtReceiptLimit
so that the inCache gets flushed when the limit is passed
- Modify flushCacheSome to flush enough words to get below MaxWordCount immediately
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2649 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
57415b6889
*) Bugfix for surftipps UTF-8 problem
...
See: http://www.yacy-forum.de/viewtopic.php?t=2864
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2647 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
b0a4fcce8c
fix from theli
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2642 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b6c7b91582
*) Parser now throws an ParserException instead of returning null on parsing errors (e.g. needed by snippet fetcher)
...
*) better logging of parser failures
*) simplified usage of plasmaparser through switchboard
*) restructuring of crawler
- crawler now returns an error message if it is used in sync mode (e.g. by snippet fetcher)
*) snippet-fetcher: more verbose error messages
*) serverByteBuffer.java: adding new function append(String,encoding)
*) serverFileUtils.java: adding functions to copy only a given number of bytes between streams
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2641 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
64b2ef5aae
*) Trying to bugfix shutdown problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2639 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
e03427871e
enhanced surftipps:
...
- added switchh to show or hide surftipps
- more news contribute to surftipps
- added voting system for surftipps
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2638 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1dc12d6659
*) Bugfix for shutdown problem caused by cacheScan thread
...
See: http://www.yacy-forum.de/viewtopic.php?p=25729
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2636 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
42173462f5
rename cutUrlText to shortenURLString;
...
other little things;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2635 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
af1d89e381
check url == null added;
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2634 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
cc667b0aa5
*) htmlFilterContentScraper.java: adding support for link tag
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2633 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
26dfbb7499
*) Bugfix for UTF-8: url names are now stored properly in stackcrawl, crawler, indexing queue and should be displayed correct on the gui
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2630 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
cf6acff2c2
*) Bugfix. htmlFilterInputStream document analysis did not work properly for documents smaller than the
...
default InputStream Buffer size.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2629 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
f18304ddd3
unused/not needed imports removes;
...
properties added;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2628 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ec031eb993
first version of surftipps
...
see http://localhost:8080/index.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2627 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
borg-0300
b174fbd0ca
"import ...*" removed;
...
properties added;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2626 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
807756150e
patch for strange bug reported by email
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2625 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
5c6251bced
*) some improvements for extended html document charset support
...
- new class htmlFilterInputStream.java which allows to pre-analyze the html header to extract
the charset meta data. This is only enabled for the crawler at the moment. Integration into
proxy needs more testing.
- adding eventlisterner interfaces to the htmlscraper to allow other classes to get informed
about detected tags (used by the htmlFilterInputStream.java)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2624 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
33f0f703c0
*) reinserting type cast again
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2623 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8c11a543dc
fixed line ending coding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2622 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b690597275
*) adding casts to avoid compatibility problems between java 1.4 and java 1.5 writer class usage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2621 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
5afb0cbce8
*) setting default charset (for unkown documents) to iso-8859-1
...
*)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2620 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f453c14b5d
removed unreacheable catch blocks and unused imports
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2619 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ad7f600f25
*) Bugfix. re-enabling inheritance of serverCharBuffer from writer class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2618 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
97d2a08ef1
*) restructuring needed to support parsing of documents using various charsets
...
- serverFileUtils.java:
-- adding methods to copy from stream to writer and readers to writers
-- moving httpc writeX methods into serverFileUtils class
- serverCharBuffer.java: removing inheritance from Writer class
- replacing htmlFilterOutputStream by htmlFilterWriter class which handles
content as char stream
- htmlFilterContentTransformer.java: deactivating getText mode
(still needs to be migrated to use char streams instead of byte streams)
- changes in several classes to use htmlFilterWriter instead of htmlFilterOutputStream
- changes in Scraper and Transformer classes to operate on chars instead of bytes
- httpdProxyHandler.java: bugfix. clientTimeout setting was missing in config file
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2617 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
fc594e8eda
*) adding httpContentLengthInputStream.java class to allow reading of http response bodies
...
until EOF even if a persistent connection is used
*) httpdByteCountInputStream.java: adding skip method
*) httpHeader.java: adding getCharacterEncoding function
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2616 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
cd636eb00e
*) Fix for the fix...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2615 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
f9a5b55a9e
*) Fixed bug described in http://www.yacy-forum.de/viewtopic.php?p=25448#25448
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2614 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
3aac5b26da
- added automatic tag generation when a web page from the search results is added
...
- added new image 'B' in front of search results for bookmark generation
- added news generation when a public bookmark is added
- the '+' in front of search results has new meaning: positive rating for that result
- added news generation when a '+' is hit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2613 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
8a30c5343d
*) Fixed bug where exclamation marks could get lost between [=...=] and <pre>...</pre>
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2612 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
low012
d8f4b17e31
*) Hopefully fixed bug described in http://www.yacy-forum.de/viewtopic.php?t=2825 .
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2611 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
0e84a969d6
*) Bugfix for serverCharBuffer read from file operation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2607 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
90ef19d778
*) first version of a serverCharBuffer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2606 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d374ef2bbe
bugfix for tryRemoveURLs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2605 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
f644a1c3a7
better evaluation of index abstracts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2604 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1b48473bc5
bugfix to utf8 recognition
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2603 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
90f7241b59
serverByteBuffer.trim() can now recognize utf-8 characters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2602 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
allo
2fd610b556
http://www.yacy-forum.de/viewtopic.php?p=25611#25611
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2601 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e34d9b3fec
*) charset aware headlines (after the serverByteBuffer.trim problem is solved)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2599 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
8115ac47b5
*) charset aware metadata parsing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2598 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
3ac30bdf22
*) some todo markers added for additional charset support
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2597 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
06fa891152
*) htmlFilterContentScraper.java: using proper charset for document title
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2595 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
74c3e7cf29
*) storing document charset into plasmaParserDocument object (is needed later by the condenser)
...
*) htmlFilterContentScraper.java: using proper charset for document title
*) serverByteBuffer.java: adding new toString which allows to specify the charset for byte encoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2593 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
c5d3020941
*) better errorhandling for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2592 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
d0a5a53789
*) changes needed for multi-language support
...
- parsers may need to know the charset of the byte stream
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2591 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d82875c72b
removed removal of 'funny symbols' that may have caused utf-8 problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2589 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
26ab1fa885
fixed null pointer exception
...
See http://www.yacy-forum.de/viewtopic.php?p=25598#25598
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2588 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b0e8ff6eda
*) some TODO makers for UTF-8 problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2586 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
41e27b85b7
fix for crawler condition
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2583 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0ee7e45413
bugfix for merge method (caused by bad refactoring)
...
see http://www.yacy-forum.de/viewtopic.php?p=25529#25529
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2581 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
40965e183e
bugfix for minimizeurldb and urldbcleanup
...
see http://www.yacy-forum.de/viewtopic.php?p=25539#25539
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2580 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
5c2f30eaca
adjustments to dhtInCache write
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2579 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
9ecf7f0da2
*) some TODO makers for UTF-8 problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2578 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e2f8339827
*) some bugfixes for UTF-8 related problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2577 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c89d8142bb
replaced old 'kCache' by a full-controlled cache
...
there are now two full-controlled caches for incoming indexes:
- dhtIn
- dhtOut
during indexing, all indexes that shall not be transported to remote peers
because they belong to the own peer are stored to dhtIn. It is furthermore
ensured that received indexes are not again transmitted to other peers
directly. They may, however be transmitted later if the network grows.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2574 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6e2907135a
bugfixes for remote search server part
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2573 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
cf9884e22b
first attempt to implement a secondary search
...
this is a set of search processes that shall enrich search results
with specialized requests to realize a combination of search results
from different peers.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2571 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
2a06ce5538
*) next bugfix for UTF-8
...
- Sending UFT-8 messages to other peers did not work
- httpd.java: minor corrections for UTF-8
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2570 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
bdc51591ae
*) UTF-8 Bug solved (hopefully)
...
See: http://www.yacy-forum.de/viewtopic.php?p=25522
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2569 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ef751b9d33
*) removing all string operations from the template engine
...
- engine should fully operate on bytes now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2567 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
7ef80c1026
more debugging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2566 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b251076e64
avoid ConcurrentModificationException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2563 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
75b198bc02
- updated references to indexContainer
...
- more bugfixes and debugging for indexAbstract processing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2555 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
0bed3b9ac3
removed superfluous interface
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2554 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b7e7808ea6
wordmigration now works also for new index database
...
if the new database is switched on, no 'too big' messages appear,
all the WORDS files can be completely migrated
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2553 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
a0ddf2ec11
*) AbstractCrawlWorker.java: delete already downloaded data on crawling error
...
*) plasmaSwitchboard.java: log unexpected errors while parsing/indexing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2552 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4f9e42d5ed
more changes towards better join-search
...
- fixed problems with index-abstract generation
- added analysis output for index abstract receive
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2551 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
005400a137
*) reverted last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2546 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a7281a9b4d
fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2545 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
82a6054275
- fixed bug with new indexAbstract generation
...
- added partly evaluation of indexAbstracts during remote searches
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2544 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
fded1f4a5d
*) better handling of maximum file size limit in crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2543 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
416b4e5c6b
ups
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2542 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
309accb983
memory control for ymage generation:
...
the ymageMatrix initializer throws an RuntimeException if there is not
enough memory available to generate a new ymage of wanted size
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2541 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
74d1dea30b
changes towards better join-search
...
- added generation of a compressed index within remote peers during global search
- added selection of specific urls within remote peers during secondary global search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2539 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
045ffebbd8
*) added debugline to versionstring-processing to find a possible bug in versiongeneration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2537 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ae4e8ce03e
- cut for 'probably last html-interface version': version number update
...
- small enhancement to ranking
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2536 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
64bed59ee8
enhancements to ranking
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2535 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
63893003be
*) Adding settings page for the crawler which allows to specify a file size limit and the timeout to use.
...
*) adding first version of maximum filesize check for the crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2534 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
06b1365066
*) fixed existing protection against divbyzero and removed the new one
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2530 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
94d7ced900
fix for last ranking commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2529 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
cc97a3e9c6
fixed possibly bug with indexOutOfBoundsException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2528 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
03835c2ee8
enhanced search result computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2527 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
809960ddc6
avoid division by zero
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2526 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
ac3419b65f
better debugging for indexOutOfBoundException bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2525 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
75b03a4580
fix for new ArrayIndexOutOfBoundException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2524 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a8bc768206
enhancements to ranking evaluation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2523 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
a82e926c5d
*) fix for wrong totalPPM-calculation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2522 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
33898ae7e9
*) ResourceInfoFactory.java: Bugfix for classNotFoundException
...
See: http://www.yacy-forum.de/viewtopic.php?t=2797
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2521 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
406e170e25
*) more verbose error message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2519 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b298474e22
*) Bugfix needed because of changed plasmaCrawlLURL.load behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2518 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c2e6cc8c6b
small part of Bosts patch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2517 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
96c6e4e322
- enhancements to detailed search page
...
- enhancements to search ranking computation process
- removed bugs in postranking
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2516 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
9340dbb501
fixed all possible problems with nullpointer exception for LURLs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2513 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
a5ed86105b
*) bugfix for handling of ResourceInfo object in proxy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2512 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hermens
ff4362b02d
some more fixes for new plasmaCrawlLURL.load behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2511 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hermens
7aeadbe7cc
another NullPointerException in http.ResourceInfo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2510 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
141f9e5bb4
fix for new plasmaCrawlLURL.load behavior
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2509 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
1e7fd48afd
added size method to ftpc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2508 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hermens
087f7511f8
prevent NullPointerException in http.ResourceInfo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2507 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
a2525072f2
bugfix for kelondroRow - property generation
...
this bug affected ranking parameters :-(
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2506 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hydrox
59a5511dbb
*) added missing static Strings as requested by theli
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2505 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
6578564c9a
*) Ignore more hop by hop http headers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2504 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b44514242a
*) crawler/ftp/CrawlWorker.java: better errorhandling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2503 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
7d7f30139c
*) crawler/ftp/CrawlWorker.java: delete old cache file
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2502 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
4ae0f122f8
*) ResourceInfo.java: License header added
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2501 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
043edfa4d8
*) ftp/ResourceInfo.java ResourceInfo object for ftp resources added
...
*) ftp/CrawlWorker.java better errorhandling for ftp crawler
*) plasmaCrawlEURL.java: some errorcodes added
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2499 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
4866868c0e
added write cache for LURLs
...
This was necessary to speed up the index receive process during global search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2498 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
8a0e35618b
enhancements to search result preparation
...
- added detailed count on remote search results
- enhanced search sequence during remote searches (doing local search in sequence)
- strict adherence to timout limits
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2497 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
5c1bb53d2a
Missing description for last commit
...
*) next step of restructuring for new crawlers
> HTCaching should now work protocol independent
-- introduction of new ResourceInfo objects containing protocolspecific metadata
of a resource.
-- the ResourceInfo objects now implement old functions like shallIndexCacheForXXX,
shallStoreCacheForXXX in a protocol dependent manner
> Indexing should also work protocol independent now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2496 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
dae763d8e3
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2495 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
4825bfaaf3
*) Bugfix for PrintWriter Problem
...
See: http://www.yacy-forum.de/viewtopic.php?t=2792
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2494 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d4c5e2af01
html-dirlist can now also be generated from existing connections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2493 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
7930839594
*) URL.java: userinfo was not taken over when generating a new url from a base url and a rel. path
...
*) CrawlWorker.java: using new dirhtml function of ftpc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2492 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
17ba468165
added html dirlisting generation in ftpc.java:
...
ftpc.dirhtml() generates a StringBuffer with a complete web page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2491 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
7a35b8e237
*) direct access to responseheaders of sbQueue.Entry removed to make it more http independent
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2487 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ffbf416e76
*) direct access to requestheader of htCache.Entry removed to make it more http independent
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2486 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
3870d615e3
*) setting htCache.Entry fields to private
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2485 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
393a7d10be
*) setting htCache.Entry fields to private
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2484 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
ab5a9bee66
*) adding some copyright headers
...
*) next step of restructuring for new crawlers
- adding first testversion of ftp crawler class
-- does not create a htCache entry yet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2483 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
5847492537
*) next step of restructuring for new crawlers
...
- IndexCreate_p.java: correcting problems with ftp urls
- URL.java does not cutout the userinfo anymore
(needed to transport authentication info in ftp urls, e.g. ftp://username:pwd@ftp.irgendwas.de)
- plasmaCrawlLoader.java:
-- hack to re enable https urls
-- adding function getSupportedProtocols
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2482 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
6cce47e217
test of ftp-urls in URL class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2481 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
fce9e7741b
*) next step of restructuring for new crawlers
...
- renaming of http specific crawler settings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2480 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
e3f0136606
*) next step of restructuring for new crawlers
...
- adding function isSupportedProcotol to plasmaCrawlLoader.java
- disabling robots.txt check for protocols other than http(s)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2479 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
9ded4e8d5a
*) Bugfix for name resolution in proxy mode
...
See: http://www.yacy-forum.de/viewtopic.php?p=25241
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2478 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1c8300fcec
*) Bugfix for name resolution in proxy mode
...
See: http://www.yacy-forum.de/viewtopic.php?p=25241
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2477 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
4e2a950ac9
*) next step of restructuring for new crawlers
...
- avoid using the http crawler class directly. Using the interface class instead
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2476 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
09b106eb04
*) next step of restructuring for new crawlers
...
- adding interface class (plasma/crawler/plasmaCrawlWorker.java) for protocol specific crawl-worker threads
- moving reusable code into abstract crawl-worker class AbstractCrawlWorker.java
- the load method of the worker threads should not be called directly anymore (e.g. by the snippet fetcher)
to crawl a page and wait for the result use function plasmaCrawlLoader.loadSync([...])
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2474 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
eb9b138986
*) next step of restructuring for new crawlers
...
- conversion of the crawler pool into a keyed object pool
- crawlers are now loaded based on the url protocol (of course works only for http now)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2473 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
1395aae742
*) starting restructuring which is needed to add crawlers for additional protocols
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2472 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
b4acbdaa97
*) better handling of server shutdown
...
See: e.g. http://www.yacy-forum.de/viewtopic.php?p=25234
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2470 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
f3ac4dbbb9
*) better handling of server shutdown
...
See: e.g. http://www.yacy-forum.de/viewtopic.php?t=2584
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2468 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
959b779aba
*) avoid performance loss if log level is greater than 'fine'
...
See: http://www.yacy-forum.de/viewtopic.php?p=25180
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2467 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
b515d49f87
*) fix for new combinedVersionString2PrettyString by bost
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2466 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
24316ba937
*) improved implementation of combinedVersionString2PrettyString by bost
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2465 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
57dda1a92c
*)again fixing for wrong version display, now totally working with double instead of float
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2464 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
479b74e1dd
*) fix for stupid mistake in new ppm-calc which caused decimal digits beeing written to seedinfo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2463 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
5e558fbaae
*) hopefully fixed the wrong display of yacy-version
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2462 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
auron_x
348258a557
*) changed PPM-calculation to be much more accurate
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2461 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
18b6876860
new cache flush configuration settings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2460 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hermens
f0278b4092
Bugfix for / by zero when the AssortmentCluster is empty
...
See: http://www.yacy-forum.de/viewtopic.php?t=2746
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2459 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
14e0bb0dcf
allow more references per word for new db
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2458 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
985dcbde7f
changed some parameters that may cause better memory usage and more indexing speed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2457 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
b7f4a1521b
added options to switch on or off the kelondroFlexTable for NURL, EURL and PreNURL
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2456 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
c26da4893b
turned back NURL usage of kelondroTree, kelondroFlexTable has still problems with deleted entries
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2454 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
db1eae0227
* simplified initialization of database objects
...
* replaced kelondroTree for NURLs by kelondroFlex
* replaced kelondroTree for EURLs by kelondroFlex
take care, may be very buggy
please finish crawls before updating. crawls will be lost.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2452 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
hermens
0b73f2b132
Repair DNS prefetch during cacheScan
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2451 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
27a159b401
* documentation update
...
* removed doc from release
* release information in doc/News.html
* release 0.46
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2442 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
theli
f80f776b89
*) Trying to solve NullpointerException problem in function addURLtoErrorDB
...
See: http://www.yacy-forum.de/viewtopic.php?t=2705
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2441 6c8d7289-2bf4-0310-a012-ef5d649a1542
18 years ago
orbiter
d78b824e85
fixed problem with default path after first start-up
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2440 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
1c99b5a484
*)fixed logging for urldbcleanup
...
*)changed exception handling in urldbcleanup so that it shows NullPointerException correctly
*)added more Blacklisting to urlcleaner
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2436 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
135e019883
removed one superfluous line from last commit
...
(hasnot is included in remove)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2435 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1591a55963
added object cache miss-cache use for remove method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2434 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8f3f4ab0eb
enhanced synchronisation in plasmaWordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2433 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f933f00f09
another patch to URL protocol handling for 'news', 'nntp' etc:
...
reject it! (the java.net.URL class rejects them too)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2432 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4c6e00d80a
more bugfixes for URL class, see:
...
http://www.yacy-forum.de/viewtopic.php?p=24844#24844
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2431 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
23dd972608
fixed memory calculation in performanceMemory web page
...
fixed also maximum cache size computation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2429 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b7dc251948
fixed bugs in url class:
...
- correct backpath ('..') handling
- correct absolute path handling
- included https
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2428 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1ce3c22761
better memory control:
...
- added memory monitor for preNURL-db in performanceMemory
- changed default memory assignments
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2427 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
39b4c26bdc
more memory control:
...
- catchup of OutOfMemoryError in server threads
- automatic adoption of word cache size after a Short Mem Cycle
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2426 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3e9d509c39
some small fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2425 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
276225d79e
fix for URL class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2423 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
eb633c0a4f
server threads must now supply a method that can be called in case
...
of short memory. This has been realized for the indexing thread.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2421 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f5720cb2fa
removed most synchronization in wordIndex (for testing)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2420 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0187c60010
because of a bug in the JRE 1.4.2 there was no memory protection
...
see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4686462
this commit fixes the bug by using a memory-computation patch.
All uses of Runtime.maxMemory had been replaced by serverMemory.max
The bug is not present any more in Java 1.5
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2419 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
auron_x
4eca0f8830
*) fixed PPM calculation for multiple indexer-threads
...
*) fixed totalPPM calculation and added total PPM to Network.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2418 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cfb51fdef1
less synchronization in plasmaWordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2416 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d6a928c2da
quickfix for http://www.yacy-forum.de/viewtopic.php?t=2705
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2415 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6ad471ef96
* applied many compiler warning recommendations
...
* cleaned up code
* added unit test code
* migrated ranking RCI computation to kelondroFlex and kelondroCollectionIndex
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2414 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
cf1186597b
utf fix from theli
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2412 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
9da3aa74d3
silly me, fix for the fix as advised by theli
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2408 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
bb3d9a5582
*) e.getMessage().indexOf() can only be used if there is actually an ExceptionMessage.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2407 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
7a54010a9c
*) Iterators can't be casted to IndexContainer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2406 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
5e0b6f8f83
*) sorting peer name list on Blacklist_p.html
...
*) restructuring of sharedBlacklist_p.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2405 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cd5f7e137c
fixed problem with NURL-generation upon first startup
...
(a new kelondroFlexTable was generated, which should not)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2402 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8418af141a
added several consistency checks and small changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2400 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9d13aeca13
*) removing class. does not work so far
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2399 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
95a84ae469
*) adding missing classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2398 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
eee44be602
*) adding an interface for customized blacklist classes
...
- now it's possible to use a customized blacklist engine
instead of the default one
- this can be done by configuring the property BlackLists.class
See: http://www.yacy-forum.de/viewtopic.php?t=2108
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2397 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6d2f15971a
there is a very strange error that causes that the kelondroRecords structure
...
is corrupted. The cause is, that the deleted-records-chain has wrong entries,
and one of the pointers in that chain points to a place behind the file end.
This causes an IndexOutOfBoundsException within an IO operation.
I currently don't know the reason that the deleted-records-chain is
corrupted, but the error can be catched. If this now happens with the
assortment database, the database is deleted.
See also:
http://www.yacy-forum.de/viewtopic.php?p=24586#24586
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2396 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d2e8e76218
*) now it's possible to configure the yacy blacklist separately for dht, search, proxy, crawler
...
See: http://www.yacy-forum.de/viewtopic.php?t=2541
http://www.yacy-forum.de/viewtopic.php?p=24516
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2389 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9ae9062bd3
* disabled new kelondroFlex table for NURLs
...
* added new RAM index Class
* fixed possible synchronization problem in kelondroRecords
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2388 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
689bbcf9cd
replaced kelondroTree db for NURLs by new kelondroFlexTable
...
The new database is only created if the old is deleted or does not exist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2387 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7fbba41962
synchronization fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2386 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
328f9859a5
more synchronization in plasmaWordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2385 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f43c90fa98
fixed handling of null referer in crawlOrder
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2384 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
130e6d4719
generalized index object for eurl, nurl and lurl to prepare move
...
of these tables to new kelondroFlexTable Object
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2382 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
acdf24877f
more synchronization against outOfMemoryError in wordIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2381 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
95160d7f2c
fixed size computation of index elements from the collection index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2380 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
26116cabde
added missing rowdef assignment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2379 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cfbacbbf08
reverted change in robotsParser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2378 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
abf22f6e60
removed url normalform computation from htmlFilterContentScraper.
...
This method was implemented in de.anomic.net.URL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2377 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
740d49751d
* strict type and size check in kelondroRow handling
...
* adopted all code to use the declaration form of kelondroRow
* fixed a bug in kelondroRow which caused wrong parsing of encoding type
* the bug caused bad database behaviour in new indexCollection data structure.
because of this bug, all test databases are now already void. A new database is created
* the kelondroFlexTable and indexCollection data structures now store a declaration of the row definition
into a properties file along the database files.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2375 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
314021453f
* more logging
...
* option in yacy.init to set useCollectionIndex usage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2374 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
a52f36787f
better templatedebugging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2371 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
3480d36417
added some debug code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2369 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
61b151b083
* added another auto-fix for collection index inconsitency check
...
* fixed words size computation for collection index
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2368 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0bbbd129ef
small fix for exception message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2367 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
718fbc2dae
enhancements in kelondroCollectionIndex:
...
* synchronized array and index objects
* auto-fix function for slightly corrupted index entries
* generalized internal access methods
also extended kelondroIndex interface to support ordering access
which is used in kelondroCollectionIndex for string comparisments
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2366 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f58283def2
better control of index flush
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2364 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4be21a3cab
ups
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2363 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
80b6c90d54
enhancements to prevent blocking during dht transfer receive
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2362 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
9f298083cd
*) adding more urls to the error url
...
- old error strings where replaced with there corresponding constants
See: http://www.yacy-forum.de/viewtopic.php?t=2638
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2360 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
d56f06401e
- Cache known URLs during indexReceive to avoid getting blocked during loadedURL.exists() whenever possible
...
- Small logging updates
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2359 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c09f734d06
*) offer router configuration on ConfigBasic.html
...
- checkbox to allow router configuration is shown if
- a) the UPnP forwarder is installed
- b) a UPnP enabled router was found
- c) no other forwarder was configured
See: http://www.yacy-forum.de/viewtopic.php?p=24264
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2358 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
dcbb4d0a6b
Display the size of HashBlacklistedCache on PerformanceMemory page.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2357 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d799622da1
better flush limit for index collections
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2354 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d468d665c9
some changes that may help to prevent deadlocks that cause an OutOfMemoryError
...
as described in
http://www.yacy-forum.de/viewtopic.php?p=24359
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2353 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
d54767f634
*) last step of removing embedded html from dir class
...
- migration finished
*) dir list now sorts the dirlist entries.
- directories are listed before files
- files are sorted alphabetically, case insensitive
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2351 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
279b1d969d
Integrated new indexing data structure 'collections' into the main class
...
for indexing, the plasmaWordIndex.
The new data structure is ready-to-use, but currently disabled.
It can be activated by setting the static
plasmaWordIndex.useCollectionIndex
to true. This shall be done for testing purpose.
The new index is stored to
DATA/INDEX/PUBLIC/TEXT
The directory PLASMA shall be used only for crawler in the future.
Attention: during testing the data structure in INDEX may change,
and created indexes with the new data structure may get useless.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2348 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4ff742e42d
implemented indexCollectionRI
...
this is the new database structure that is supposed to replace the
plasmaAssortmentCluster AND the plasmaWordIndexFileCluster
The new structure is not yet active and needs to be integrated into
plasmaWordIndex. This has some migration constraints that are not yet
completely solved.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2347 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
01f95eccd3
re-write of kelondroCollectionIndex. This is the data structure that
...
shall replace the current assortment files.
* used the kelondroFlexTable to hold the index of collections
* used kelondroRow definitions to declare all data structures
* fixed several bugs that appeared in kelondroRowSet and kelondroRowCollection during testing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2344 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ebc2233092
* implemented (finished) class indexRowSetContainer
...
* replaced indexTreeMapContainer by indexRowSetContainer
* deleted indexTreeMapContainer and abstract class
This is another step to the new database structure
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2343 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9183d21f25
renamed new index class to old name
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2342 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c4e922885a
replaced indexURLEntry by new class that uses a kelondroRow.Entry object
...
to store the index entry. This is another step to move to the new database structure.
A side effect of this change is, that index storage uses much less RAM space,
which affects the index RAM cache.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2341 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0b7112f8b2
fix for missing topLevelClone in indexRAMCacheRI.wordContainerIterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2340 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e357599f92
* fixed problem with indexContainer iteration from RAM:
...
indexContainers from RAM must be cloned explicitely to prevent
side-effects on stored indexContainer objects in Cache
* changed behaviour of urlReference deletion from indexContainers:
deletion does not user retrieval of all Elements from the assortments
* added textual configuration of kelondroRow and kelondroColumn definition
* update of kelondroRow usage in yacyNews
* modified kelondroAttrSeq to use modified kelondroColumn parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2339 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
57fe5cc671
*) code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2338 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
4e9f02c8ec
integration of Michaels string-extraction.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2337 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8b77afd72c
some fixes to new container merger
...
and some code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2336 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
830167596a
bugfix for
...
http://www.yacy-forum.de/viewtopic.php?p=24127#24127
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2333 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
839806a775
*) serverPortForwardingUpnp.java: code cleanup, license header added
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2332 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
03230cd887
*) removing old port forwarding classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2330 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
6e676224d0
*) adding support for upnp
...
A new port forwarding method for upnp was added.
If this method is enabled, yacy automatically determines an UPnP
capable internet gateway and configures the gateway port forwarding
settings properly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2328 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
417ed5102e
redesign of database iterators:
...
an iteration of key elements in kelondroTree databases is no longer supported.
this is now replaced by an iteration of kelondroRow.Entry objects from the database
Iteration of keys from the database was mostly followed by retrieval of the row
from the database, whcih caused unnecessary database load.
The index selection was also redesigned to use the new row iteration methods.
This affects many funktions, most important is the DHT selection routine which is now much faster.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2327 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
0db237467f
*) bugfix for URL generation from file
...
see: http://www.yacy-forum.de/viewtopic.php?p=24116
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2326 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ad692fc6c7
implemented option to extract nurls from the database
...
(plus some iteration enhancements for nurls)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2325 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7fd90ca7c8
* strict handling of NURL entry element generation, storage and stacking
...
* more space for EURL reason strings (you must delete the EURL db to use this)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2324 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5f72be2a95
some redesign of EURL storage
...
* store() is now called explicitely
* more urls are written to the EURL table
* the EURL stack does not store the complete entry any more, now only the URL hash
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2323 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1ed3e2daef
added option to extract domains and/or urls from the eurl database
...
when extracting from eurl, the html output format is recommended, since
this format adds also the fail reason to the domain/url.
The complete syntax for domain extraction is now
java -Xmx<megabytes>m -classpath classes yacy -domlist [ -source { lurl | eurl } ] [ -format { text | zip | gzip | html } ] [ <path to DATA folder> ]
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2322 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7e0a130fb5
new indexURLEntry class 'indexURLEntryNew', to replace old class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2321 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
58df8b7bbf
a large collection of different changes
...
* mainly for the transition to the new indexing database structure
* a bugfix for an endless loop inside kelondroTree iteration
* a bugfix for bulk read inside a kelondroTree iteration; the bug caused that some elements had been iterated twice
* very strong speed enhancement for url/domain extraction
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2320 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
493b1cd2bf
better logging for domain extraction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2319 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e20ff77c10
another bugfix in new url class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2318 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
685430a1b5
bugfix in new URL class, better loggin for domain extraction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2317 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c57b78722b
added some more logging to domain extraction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2316 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
79af283f6c
better debugging in new URL class for wrong port numbers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2315 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cc2be7fb43
fix for genurllist in case of bad urls
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2314 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
1b2ea58ee9
wrong substring invocation.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2313 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e4f1820b58
protection against too long authentication strings in switchboard
...
see also: http://www.yacy-forum.de/viewtopic.php?p=23943#23943
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2312 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b3f7e62e03
better handling of whitespace
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2311 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4149939c02
better handling of whitespace for gettext quotation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2310 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
97fa6788a1
added gettext support:
...
automatic replacement of string appearances in html files by
gettext quotes.
see also: http://www.yacy-forum.de/viewtopic.php?p=23901#23901
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2309 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b3c569f706
*) renaming of function getTransferedEntitySpeed to getTransferedEntrySpeed to avoid confusion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2308 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ff3f174a2d
case insentive commandline options
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2306 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
67edd80884
removed tabs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2305 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
67c486a023
some example Code, how supertemplates can be used.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2304 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5214f571cd
simplified method call in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2303 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
7b0e2521bb
Support for a supertemplate, which can do all thing, a normal template can do.
...
Its a layer under the servlets, this means, #[page]# will be replaced by serverletcode, the rest can be set by you.
(TODO: if we use this for layout, we need to read "TITLE" from the servlet's tp, to set it outside of the servlet.)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2302 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4bd626572b
added hashCode and compareTo to new URL class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2301 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
abb5264929
fix for
...
http://www.yacy-forum.de/viewtopic.php?p=23868#23868
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2300 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ff39a7a0d1
Overlay for welcome.*
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2299 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
a70cbd959b
*) further improvements for the anomic.net.url class
...
- relpath starting with javascript: are ignored now
- bugfix for concatenation of relpath starting with # or ?
in this case no slash should be added to the baseURL, otherwise
we get URLs of the form http://test.de/index.html/?param=value
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2298 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8a1f1d96b3
*) Bugfix for url concatenation. Relative urls with / or http:// at the beginning
...
were not handled correctly on url concatenation via new URL(URL,relPath).
See: http://www.yacy-forum.de/viewtopic.php?t=2623
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2297 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
ca33eaa442
- Some spelling
...
- Removed unused init value
- Set default upload value to "none", which avoids an warning which says, upload method '' would be unknown, on new installations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2295 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
8795875800
dirlisting for all empty directories.
...
no problem to update dir.java anymore, because its only in htroot/htdocsdefault needed.
migration to delete old dir.* files in the fileshare
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2294 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7935f27038
enhanced synchronization in balancer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2291 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3879a0ecd0
replaced java.net.URL usage by use of new class de.anomic.net.URL
...
This shall be seen as an experiment to exclude all cases where
there could be a DNS lookup during URL comparisment.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2290 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
07900366ac
deactivated cache-initialization for file-indexes (files in WORDS)
...
see also: http://www.yacy-forum.de/viewtopic.php?p=23801#23801
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2289 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
40aa735520
fixe timing problem causing too long delay during initialization of kelondroTree objects
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2288 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d2bb3f442e
fixed timing problem causing a division by zero exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2287 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6acb6a4d8f
tiny performance optimization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2285 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
2bdf1fc360
totalPPM
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2282 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
24a02cbeef
*) Bugfix for not parsable application/xhtml+xml resources if
...
an URL has no extension
See: http://www.yacy-forum.de/viewtopic.php?p=23687
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2280 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b0ca5fa784
some correction algorithm for preload time computation during assortment open
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2279 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e22cbaee97
- extended logging for preload
...
- reduced preload-time for IndexImport_p.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2278 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
671fd9a5c9
work towards new indexing database structure
...
(no effect on current functionality yet)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2277 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
92f4cb4d73
added option to configure the start-up delay time for kelondro database files.
...
the start-up delay is used to pre-load the database node cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2276 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ce9dd3e76d
some work in the index construction zone (no effect yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2275 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fe617d7e54
*) adding function to return the protocol type of a ssl connection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2274 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
018b3e0832
added pause option to server threads.
...
The pause is started by calling intermission(Long.MAX_VALUE)
and can be stopped by calling intermission(0)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2272 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e1a52bea22
added a class stub for the new database structure:
...
a reverse word index based on a a collection index,
which is an index for a set of array files containing
row collections.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2271 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3b69b35bf2
added pre-load of node cache entries to kelondroRecords
...
this gives the kelondroTree data structure a similar start-up
behaviour like the kelondroFlexTable: the cache is filled with
routing data in such a way that is more performant than
reading node records during normal operation.
The pre-load phase stops automatically after a time-out of 500 milliseconds
of if the cache is full.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2270 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
85d575e928
enhancements to kelondroRow and kelondroColumn
...
these are changes towards a better indexURLEntry implementation
which are needed for the new database structures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2268 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ab1ed053f5
another small correction
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2267 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b92561fb67
removed unused code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2266 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
eadbd56fc5
small adjustment to last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2265 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e9765ac4e6
introduced bulk read for node iterator in kelondroRecords
...
this speeds up the iterator by factor 2
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2264 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6643da3fbd
bugfix for http://www.yacy-forum.de/viewtopic.php?p=23463#23463
...
(affected URL DB Cleaner)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2263 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
866d53ed70
fix for DNS block bug
...
see http://www.yacy-forum.de/viewtopic.php?p=23458#23458
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2262 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6af70febef
- added kelondroTree index option to kelondroFlexTable
...
- automatic generation of index file when index is too large for RAM
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2261 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dd2865178a
major bugfix (searched a whole week for the bug) for
...
the kelondroRowBuffer, which has effect mostly to the
kelondroFlexTable but also to all other database functions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2260 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f9b9d085c4
just changed testing code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2259 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b594ee9a5a
*) Adding possibility to configure if the http proxy should send the
...
X-forwarded-for header (requested by TeeSee)
See: http://www.yacy-forum.de/viewtopic.php?t=2577
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2257 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ef84fc4956
added IOException to size() and row()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2256 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
7646849241
*) implementation of size() for mysql
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2255 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
84dfd76a6a
kelondroFlex bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2254 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
53077f5835
*)fixed paths to yacy.logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2252 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
8ba8e2b7d9
*) added cache for blacklists urlhashs recieved by DHT. DHT does not request URLs listed in this cache.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2251 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
53cbcc6d6e
Implement emergency break in index receive when the limit of the ramCache is exceeded by more than cacheLimit
...
See: http://www.yacy-forum.de/viewtopic.php?p=22911#22911
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2248 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e40987ecab
removed default memory reservation for testing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2247 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4cc6e6551f
bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2245 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
66964dc015
removed high/med/low from kelondroRecords cache control.
...
this was done because testing showed that cache-delete operations
slowed down record access most, even more that actual IO operations.
Cache-delete operations appeared when entries were shifted from low-priority
positions to high-priority positions. During a fill of x entries to a database,
x/2 delete situation happen which caused two or more delete operations.
removing the cache control means that these delete operations are not
necessary any more, but it is more difficult to decide which cache elements
shall be removed in case that the cache is full. There is not yet a stable
solution for this case, but the advantage of a faster cache is more important
that the flush problem.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2244 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6866bc2758
be quiet!
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2243 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
4c6083b264
network picture;
...
back to old version
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2242 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
955915385a
network picture;
...
small changes;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2241 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
027fa8ab1c
network picture;
...
bigger;
more dot steps;
small other;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2240 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
41c4641612
added some profiling to kelondro caching classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2239 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dd560e4b2f
finetuning
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2238 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a5661de880
added debug line
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2237 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5b1d77cd4b
some enhancements to caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2236 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
ad03715b64
*) added fill and read functions with benchmark output every 10000 operations to dbtest.java
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2235 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
b20496e42b
*) make DHT DoS check configurable (requested by KoH)
...
- check can be disabled via property indexDistribution.dhtReceiptLimitEnabled
- upper bound can be configured via indexDistribution.dhtReceiptLimit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2234 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
650c7e9e55
some enhancements to caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2233 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
d4645062bc
Correct usage of vhost in wget/wput requests:
...
- yacyClient: don't use own .yacyh domain in requests, instead use .yacyh domain of target peer for everything but ranking distribution
- natLib: use full hostname instead of just SLD.TLD
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2232 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bde748f68f
- integrated buffer in RowCollectionSet which speeds up the new
...
cache by factor 3. (but the new cache was 5 times slower than the
old cache...)
- removed old unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2231 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fa9712151a
*) dbtest.java: bugfix for mysql test. data was not written correctly
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2230 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
f55d3d0d79
*) better error message if jdbc driver could not be loaded
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2229 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
07aa779e6d
*) dbtest: adding read and write methods for mysql
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2228 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d083b15c4b
tests/fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2226 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ce9a98ee25
experiments with buffer for collection class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2225 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6aa0f6eb0f
better abstraction, less synchronization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2224 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1ec07fe351
better structure of the kelondro collection classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2223 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
c264cda678
support to parse gettext files and to generate updated files, where only the new strings are empty.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2222 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
761d3bba54
integrated kelondroRowCollection in kelondroCollectionIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2221 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8f0c1a240a
added kelondroRowCollection and kelondroRowSet which
...
shall replace kelondroCollection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2220 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
ed2cb040d1
*) Bugfix for http connection header validation
...
- Connection header was not handled correctly if it contains
multiple values, e.g. Connection: TE, close
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2219 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
f95c006dc2
generation of empty gettext files from yacy html-files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2218 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4fa3271754
adopted memory calculation to new memory amount used by kelondroCollectionObjectMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2215 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
0621106ef3
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2214 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
11951aed41
replaced node cache Object type in kelondroTree:
...
kelondroCollectionObjectMap replaces HashMap.
the kelondroCollectionObjectMap uses only 1/4 of the memory of HashMap,
but is slightly slower than the HashMap.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2213 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
12af69dd86
cosmetics
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2212 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
ea72a7c580
added new objectMap (can replace HashMap and should use less memory, but is slower)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2211 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
67a8c74be3
Fix for dynamic login with static password.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2210 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ef9eb50c3c
fix for adminlogin
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2209 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
6fe2fed87e
cookieauth works with static Admin.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2208 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
45b39ee1be
*) solving unpacking problems with to long filename by
...
a) renaming the parent folder in the tgz file to yacy
(can be configured via build properties file)
b) reconfiguring build file to throw an error if a file
name is too long
Please note that currently there is _no_ proplem with too long
class names because of step a.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2207 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
fb090652df
*) use a more compact for plasmaWordIndexAssortmentImporter.java because the long name
...
caused problems during untar operation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2206 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
4ca0857c0c
*) Index transfer now considers the pause time send by busy peers during
...
index transfer / index distribution
See: http://www.yacy-forum.de/viewtopic.php?p=22647#22491
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2205 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e8848a3532
bugfixing for new kelondroFlexTable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2204 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
b23703f260
using cookieAuth.
...
logout for httpauth seems to be broken :-(
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2202 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
7f51a43cba
disabled ipAuth for _p Pages (and broken Form-Login :-() for security reasons
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2201 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
bd22634c44
HTML-login, logout fixed.
...
TODO: If you login with the form, then logout with the form, and then try to login with httpauth, the first try will fail.
(should logged_out be resettet in ipAuth? but if there is ipAuth before proxyAuth, the logout would be broken. Maybe a combined method can help.)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2200 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
3f1ebc097e
Limit the size of the DNS cache to 5000 and the age of the entries to one day.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2199 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
d7a3fdb18b
no white pages, when clicking cancel on the password-dialog
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2198 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
ec36174b7f
Form-Login works now.
...
For _p Pages is httpauth still needed.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2194 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
11123416f3
Form-Login function.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2193 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
bd13bd78ad
Bugfix for broken Tags
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2192 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
75ed507d39
some debugging of new kelondroFlexTable class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2190 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
f01bd25489
*) Bugfix for OutOfMemory problem during minimizeUrlDB
...
See: http://www.yacy-forum.de/viewtopic.php?t=2498
*) out of date import functions removed (can be done via web gui)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2189 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
8a18302e84
some work on the intMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2186 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
56084e9edc
added a intMap, the new index management structure for kelondroFlexTables
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2185 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e374add8cb
added html tag 'area' to html parser
...
see http://www.yacy-forum.de/viewtopic.php?p=22540#22540
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2184 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
eda055e729
- enhanced kelondroCollection speed
...
- tested + debugged kelondroCollection
- inserted the new kelondroCollection Object as indexing class for kelondroFlexTable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2183 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5d2cd26529
enhanced kelondroCollection performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2182 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
5625937d1c
Language improvements
...
One very minor HTML fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2181 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
577a6e6487
some work on kelondroCollection, no effect on current YaCy functions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2180 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
2185a5cfb7
prevention of possible ConcurrentModificationException
...
This happend at my peer and created a deadlock.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2179 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
auron_x
71228cfc56
*)added ip to Blog.xml
...
*)added possibility to import Blog.xml-files
*)included german translation for new Blog-features
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2178 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b30645305b
some additions to kelondroCollection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2172 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
370c481fa7
bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2171 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
26b6cddf51
synchronized the DNS cache, because the non-synchronized version resulted in deadlocks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2168 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c36e9fc8d3
full integration of kelondroRow
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2167 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6e49721762
completed and tested kelondroFlexTable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2166 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b68f7dec9d
added stub of new table class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2165 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
b21824ef16
added new array interface
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2164 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
c75cacda95
added a flex-width-array: this is a table where it is
...
possible to add columns to an existing table
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2163 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4a907a570f
1st step to migrate kelondroTree to usage of kelondroRow instead of byte[][]
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2162 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
572d53506c
new kelondroRow objects now replace byte[][] objects in object cache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2161 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
09f780df27
more bugfixes for the new row/stack handling changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2160 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
89424ff122
multiple bugfixes for new stack/row features
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2159 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3c3c047d0a
integrated kelondroRow into kelondroStack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2156 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5bb565944f
integration of new kelondroRow into some parts of kelondro,
...
especially into the array storage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2155 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d580f582dc
update to row/column handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2154 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
eaa6f012f0
refactoring: better naming for classic DB (files in WORDS)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2151 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
5041d330ce
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2150 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
7b3b12888c
refactoring: integrated indexContainer abstraction layer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2149 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cb295fbbdc
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2147 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
bc94a714b2
Better explanation for the auto-dom-filter.
...
Some javadoc.
Small change to DetailedSearch.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2146 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
28ff7ec214
added missing file from last refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2145 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
196b8abb30
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2144 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
24686e50a2
- fixed a caching bug
...
- added counter for cache delete to distinguish between flush and delete
- changed some default paramenters for cache size settings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2143 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
b48327904a
Don't disconnect peers that report 'busy' during index transfer.
...
These peers are already being marked as not accepting remote index transmissions by yacyClient.transferIndex. That should by enough to prevent further transfer attempts until newer seed information is received.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2142 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
899c1a5df5
integrated cell-, and row-management classes
...
this will be used for automatic export features
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2141 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4d8f8ba384
added cache-performance analysis for node caches
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2140 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
a520ab2e8c
*) adding possibility to use an existing PKCS12 certificate for https
...
instead of creating a new one.
Notes:
This import is done automatically on startup if the following properties
are set in the config file:
pkcs12ImportFile =
pkcs12ImportPwd =
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2139 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c5c5c166f0
*) fix for javadoc generation
...
See: http://www.yacy-forum.de/viewtopic.php?p=21884
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2138 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1b4a27cd99
added monitoring for new miss-cache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2137 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
bd057b44dd
- automatic setting of peer-does-not-accept-remote-crawl
...
- increased percentage of object cache to node cache to 30%
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2136 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
912116c5a7
renew cache update time if a cache-read hit happens
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2135 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
81e79f2caf
fixed new cache behaviour changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2134 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
545f81b317
bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2133 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
cda087f43b
- integrated cache miss storage into object cache
...
- removed cache-miss handling from indexURL
todo: new Monitoring in PerformanceMemory_p
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2132 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
757ec28430
refactoring: better data capsulation for indexURL
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2131 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
61078b3885
*) adding support for delayed shutdown
...
- needed by Ismael to receive the Steering page properly on shutdown
- now the steering page should always be displayed properly in the web browser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2129 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
8b7626f8d1
*) Automatic redirection of browser if user changes port settings in ConfigBasic
...
See: http://www.yacy-forum.de/viewtopic.php?t=2415
*) If ssl is available, the browser conntects to yacy via https on yacy startup
See: http://www.yacy-forum.de/viewtopic.php?p=21649#21649
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2127 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
auron_x
e7199c6c7e
*)fixed "Umlaute"-Bug in Blog
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2125 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
af098b815c
- fix for blog/wiki error
...
- re-enabled new map writing methods
see http://www.yacy-forum.de/viewtopic.php?p=21729#21729
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2124 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
90d569d70f
refactoring of index management:
...
url storage is part of index management; moved plasmaURL to indexURL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2122 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a930be4ba3
refactoring of index management:
...
generalized the index entry
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2121 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
634543121b
added debug message
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2120 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
74a78ebd78
turned back handling of map-writing into kelondro because of possible bug for testing
...
see http://www.yacy-forum.de/viewtopic.php?p=21560#21560
please re-test
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2119 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e656a00b74
removed some unused methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2118 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
auron_x
232ab7902b
*)catched Null-Pointer-Exception
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2117 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
82b2bc6932
patch for index-transfer DoS problem
...
see http://www.yacy-forum.de/viewtopic.php?p=21627#21627
note that this function will make the index-transfer functionality void
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2114 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
df7e1d9df3
Changes to plasmaURL and subclasses:
...
- Improve performance of plasmaURL.exists() by remembering URL-hashes that are not present
- Use a more realistic estimation of memory usage by the existsIndex cache
- Routine cleanup of the existsIndex to limit its memory usage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2113 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a474669338
start with refactoring of index management
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2110 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago