sixcooler
787b588c33
reverted a part of svn6636:
...
- didn't work on blobs >2GB
- should be obsolete since svn6651
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2652&sid=7fa98fd3edfc2a03f26394d545e3e3c1&p=19172#p19172
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6655 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
11188cd7eb
resource observer now uses the Java 6 method to check for free space. thus, disk observing now needs Java 6 installed.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6652 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
089877f32c
my first commit - hopefully fix for merge problem
...
- http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2652
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6651 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
be18b5d8cd
fix for 'cannot switch back to default language'-bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6649 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d6391f2537
better handling of rewrite cases where the resulting rewrite blob entry is equal in size
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6648 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ef9473d92c
added another sixcooler suggestion: recycle corrupted records
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6647 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fe78edac32
- view API calls in correct date-order
...
- execute recorded API calls in date-order
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6646 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
74e736c903
missing file for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6645 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
308a973503
refactoring of tables data organisation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6644 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
85ca96227f
fix for re-enable parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6643 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8a76f38d26
Added a new steering servlet that can be used to repeat actions that had been made on the yacy interface. This can be used to:
...
- start again a previously started crawl
- submit settings (again). This option will be used to transmit
all settings of one peer to another peer if the remote-peer
steering function is ready
This steering framework will also be used for a 'schedule-everything'
which will also include a new scheduler for crawling.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6642 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
840527689b
more simplification of bookmark class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6639 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d77782a8d5
removed bookmark tags file, tags are now stored only in RAM
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6638 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ada0ce9de3
refactoring of bookmarks: there is a big performance problem in the bookmarks code and furthermore the bookmarks
...
will loose its leading role for the re-crawl funtion when the new api tables will work. To be prepared for a replacement
of such functions the bookmark class is re-organised.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6637 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3751ab4ae2
added sixcoolers patch and more checks/removed unnecessary code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6636 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d8d8562c59
fill key with zeros during normalization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6635 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a131ebbcb5
one more fix for NPE, see
...
http://forum.yacy-websuche.de/viewtopic.php?p=19010#p19010
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6634 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
24060885b6
- added Tables abstraction in data.Tables.java
...
fix for
http://forum.yacy-websuche.de/viewtopic.php?p=18910#p18910
http://forum.yacy-websuche.de/viewtopic.php?p=18894#p18894
http://forum.yacy-websuche.de/viewtopic.php?p=18814#p18814
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6631 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7fdf59a77f
misc NPE check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6630 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a512aef6ad
fix for http://forum.yacy-websuche.de/viewtopic.php?p=18918#p18918
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6629 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
38a3d55afd
added more possible php extensions for html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6621 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4403304957
bugfix for list()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6616 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3889438db6
fix for bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6615 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
23bcca07a3
removed directly linked servlets that had been there to test memory failures that appeared in that servlets
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6612 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
69c29acb6e
no exception thread dump if parser cannot parse becuase that mime-type/extension is in the deny-set
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6611 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0098e6e859
bugfix for heap iterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6610 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
db19a941cf
added new image index storage classes (not integrated yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6608 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c8aece34a4
update to yacy/ai (just more testing)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6607 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8ce936bcdd
added an api recording function: it shall be possible to record
...
all operations on YaCy in a database that should make it possible
1) to re-create a setting on fresh peers
2) to transmit a setting from one peer to another
3) to re-create crawl starts after a complete deletion of the index
This functionality will also support
4) scheduled re-crawls (new implementation)
To implement this, a new database structure has been crated that stores maps into blob heaps. to encode maps the b-encoding technique was used (this is the same encoding that torrent files use)
- added a b-encoder
- enhanced the b-decoder
- added a b-encoded map heap data structure
- added a table organisation based on b-encoded heaps
- added a servlet to maintain such tables (see Tables_p.html)
- integrated the servlet into the Advanced Settings menu
- added an api recording based on the new tables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6606 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
56e0d9bd01
- testings with image parser
...
- added image size as part of parsed text in images
- avoid unnecessary error messages if parsing of documents failed but one succeeded
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6597 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e80e060ca6
- increased thread priority for server threads
...
- decreased thread priority for crawler threads
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6596 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
234f733a3d
- relocation of seed db is better for network switch than re-initialization because of the embedding of the peers object in other objects
...
- small refactoring of blacklist interface code to remove PMD warnings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6593 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
473b11033d
fixed network switch process - crawling did not work after a switch before this fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6592 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fd7b348973
some fixes for the network switch
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6591 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7d400b17d0
html parser support for .cfm files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6590 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f6731c6240
more logging etc.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6589 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
007f8297de
added php3 as extension type for html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6588 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4f1f4863c4
fix for deadlock when initializing a SplitTable with a file of size 0, see also:
...
http://forum.yacy-websuche.de/viewtopic.php?p=18594#p18594
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6587 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
cc5dcf69ff
missing change for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6585 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ca1ef9a079
fix for http://forum.yacy-websuche.de/viewtopic.php?p=18584#p18584
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6584 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d9169cc6c3
increased proxy load time-out from 30000 to 60000 milliseconds
...
according to http://forum.yacy-websuche.de/viewtopic.php?p=17782#p17782
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6583 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
938e806182
tried to fix date problem that may have prevented that foreign peers stay in the network
...
- removed unused code
- removed possibly wrong utc difference correction
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6581 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bd05e57d3b
fix for http://forum.yacy-websuche.de/viewtopic.php?p=18563#p18563
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6580 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5df628a2a4
- added BEncoder class
...
- added BEncodedHeap class that encodes B data structures and stores that to a heap
- refactoring of MapView, this is now named MapHeap to fit into the naming scheme of the BEncodedHeap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6579 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
82f57f79e5
more PMD enhancements
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6576 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5d930c96f0
more fixes to search result page navigation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6575 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8c520f128d
reverted a change in ranking process committed this afternoon
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6573 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a06f7ddb33
more PMD recommendations
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6572 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
eb79ceb3ff
update to kelondro data structures
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6571 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
18172451a0
better search computation:
...
- increased sort limit, now 3000 entries, before: 1000
this should cause that more results can be shown in case
of strong limitating constraints, like domain navigation
- enhanced the sort process
- check against domain navigator bugs
- fix in sort stack
- showing now all naviagtion pages at first search (not only next page)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6569 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d126d6c1b5
renamed the servlet WatchCrawler_p to Crawler_p
...
this was done because that servlet may be used for wget/cronjob
triggered crawl starts and it appears to be confusing that the
name of the crawl start servlet looks like a pure monitoring tool.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6568 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
66c0a8e849
more PMD recommendations
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6567 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
909a4f91c7
added a logging output for crawl starts that shows the URL that can be used to start the crawl again
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6566 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bc96d74813
- clean-up of robots.txt parser
...
- added 'yacybot' as key to recognize robots.txt entries for YaCy
- removed unused method to get robots.txt from database
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6565 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2113fcd7e5
- fixed usage of isEmpty() which is not available in java 1.5
...
- increased visibility of some methods
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6564 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dd459281c8
applied code changes that are recommended by PMD
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6563 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
eac2daf2e8
* reenable DHT if yet enough memory is available
...
* reset treshold on reconfiguratoin
(thanks to sixcooler)
* display status message in web interface
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6562 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
0752634b8b
log YaCy version on startup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6561 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d77a8f3b3e
added some modifications recommended by PMD for better performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6560 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d1973bae2a
code cleanup: removed unused code and unused methods
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6559 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a3b8b7b5c5
some redesign of the main menu structure:
...
- moved all index generation servlets to it's own main menu item, including proxy indexing
- removed external index import because this operation is not recommended any more. Joining an index can simply be done by moving the index files from one peer to the other peer; they will be merged automatically
- fix to prevent endless loops when disconnecting http sessions
- fix to prevent application of bad blacklist entries that can cause a 'Dangling meta character' exception
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6558 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
ab3cf60dbe
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6557 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7f20963b41
add-on to last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6556 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
eeca2ded92
fix for http://forum.yacy-websuche.de/viewtopic.php?p=18500#p18500
...
- catch uncatched OOM
- less wasting of memory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6555 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
32972139af
added nice configuration for the resource observer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6554 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bb2e03761c
- fix for deadlock with 100% CPU during search
...
- fix for failure of ranking because of a ConcurrentModificationException
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6553 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3f771d2a16
fix for rss parser: be lazy when rss is not well-formed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6552 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dff4f95c78
some patches to get the torrent parser working
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6551 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
574f49903e
Prevent blob merge from possibly losing the last container
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6549 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
83d05e9176
added sixcoolers hack with some modifications:
...
http://forum.yacy-websuche.de/viewtopic.php?p=15004#p15004
old index blobs where deletions have been made because of DHT transmission should be melted down to new blobs. This uses sixcoolers methods from the forum thread but modifies the process in such a way that the blobs are not merged with themselves but simply rewritten to smaller files.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6548 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fbd24c2d84
integrated the torrent parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6547 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bd32f8b8cb
added a torrent metadata file parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6546 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
610e3ffffb
Added new classes for the implementation of concurrent greedy algorithms.
...
These classes can be used to produce an abstract worker process that can be used for common problems in artificial intelligence, such as game playing and problem solving. These classes will be used as abstraction layer for a new search process in YaCy. These classes had been created while searching for an abstraction of the current search process. It turned out that the abstraction of the YaCy search process is also an abstraction for problems in artificial intelligence and therefore the classes had been designed in such a way that it covers not only the YaCy-specific problem but also the more generic problems in ai. To test the classes they had been used in a ConnectFour implementation (game playing).
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6545 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d0b7bf9ca2
added a decoder class for Bencoding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6544 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
028657f019
*) adding more SVN properties
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6542 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
82d740050f
*) adding more SVN properties
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6541 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
e04cb8cef0
*) adding more SVN properties
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6540 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
dcb1096fb0
*) adding more SVN properties
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6539 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
7d610e0063
*) minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6538 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
82198acc06
*) minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6537 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
b75547fc60
*) minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6536 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
9bee0ac780
more logging for DHTrule
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6533 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
57d729e377
fix for negative numbers in network statistic
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6532 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4ac4fe952c
patch for npe in bookmarks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6530 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c14233a933
fix for a OOM in MapView that can cause unavailability of
...
- seed list
- bookmarks
during very low memory configuration
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6529 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d548bd41ad
fix for a npe during search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6528 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
37245430c3
fix for NPE during DHT RWI selection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6527 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
959b38b61b
fix for memory tracker
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6526 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a37878b7d5
url parser regex performance hack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6524 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b527d2ebfa
fix for media search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6522 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
362b7a929b
added extensive memory protection logic to avoid out of memory errors that may be caused by the RowCollection memory allocation function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6521 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8281e29963
- more configuration for profiling graph (number of events)
...
- more logging for a shutdown: print reason and accessing IP into log
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6520 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
f1ori
5f0f6b71b4
* revert last commit, something is more broken than before
...
* UTC timestamps and lastseen-properteries still needs some debugging
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6519 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
f1ori
8c8b642eba
* fix timezone problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6518 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
713cb26a27
update for memory observer algorithm
...
disable dht if memory is less than treshold
after 4 times, maximum 11 minutes between each detection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6517 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4782d2c438
fix for search bug that appeared when looking at page 3 of results or further
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6515 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
29fde9ed49
better control of ranking order in sort stack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6514 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
93caa38d55
fix for bug in SortStack (did not appear to shrink according to required size) - caused bad and unsufficient search results
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6513 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
66923ebc6c
- modified method in RequestHeader that delivers the host name of requester: no more reverse domain lookup (may have killed interface performance in some cases)
...
- added logging output for shutdown servlet: show ip of requester of the shutdown
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6512 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e34e63a039
preset of proper HashMap dimensions: should prevent re-hashing and increase performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6511 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4a5100789f
replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f4946eaf27
- better thread dump
...
- suppressed one server exception
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6509 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9743b70d1c
disabled keep-alive of server, not really needed for speed but a cause for much trouble and memory occupancy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6508 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
491ba6a1ba
- some refactoring in workflow
...
- some refactoring in search process
- fixed image search for json and rss output
- search navigation on bottom of search result page in cases where there are more than 6 results on page
- fixes for number of displayed documents
- disabled pseudostemming
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
969123385b
added json and rss output for image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6503 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d183f8d980
refactoring (moved code from ContentTransformer to TemplateEngine)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6498 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
23aef43786
- better synchronization in SortStack
...
- better ThreadGroup organization
- less worker threads for media search (64 was too much...)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6497 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7b1f5b0430
- better media search ranking
...
- better concurrency with enhanced synchronization in sort stack
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6496 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4df88a4e7a
- fixes for missing or bad hashCode computation
...
- fixes for bad equals() methods that had not been used by hash maps and therefore some classes did not work as objects in hash maps.
- this may also affect some cases where double-checks should have been, but did not work.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6495 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dbdf2570ba
added comparator and more fixes for SortStack/SortStore
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6494 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d2938c44a1
- added bmp parser to the document parsers
...
- image parser that implement the document parser interface return itself in the list of images of the document which should cause that the parsed images contribute to the image search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6493 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1dff620181
Better implementation of SortStack and SortStore and adoptions in all using classes to implement the necessary Comparable interface and hash code computation.
...
The better SortStack performance affects crawling and image search speed and quality.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6492 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fe41a84330
some enhancements in web caching: avoid double loading of response metadata and/or content
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6491 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
06d0dcde20
more enhancements to image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6490 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4c6312d103
enhanced image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6489 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2d8f3ee301
some performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6488 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
94b2a664f3
- use a static DiskFileItemFactory (one instantiation is enough)
...
- use more memory for the DiskFileItemFactory to avoid IO when POST commands come
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6485 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fd0658ce7c
avoid forced execution of InetAddress.getLocalHost() at startup, because that hangs at some strangely declared linux configurations. The Domains.localHostAddresses object is first instantiated with a more simple logic and enriched with more host addresses using a concurrent thread that will not block a startup process.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6482 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
013f337d3f
- avoid unnecessary host name lookups for localhost
...
- avoid unnecessary reverse domain name lookups for remote access
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6481 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
20c5d78a5c
fix for a ConcurrentModificationException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6478 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5afd9f7a91
fix for crlf writing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6477 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7144d2df6e
added crawlReceipt servlet as individual class to examine OOM problem as documented in
...
http://forum.yacy-websuche.de/viewtopic.php?p=18120#p18120
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6476 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2d3c98b742
less computation within synchronized blocks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6475 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1a146b0d73
added a patch to ignore bad mime-ignore patterns
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6474 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
29fe436e36
- fixed post-ranking including prefer mask
...
- enhanced a core database access method / less wasted ram
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6473 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5399d1e2bc
refactoring (reason: get more abstraction to use the blacklist class; for integration in other servlets)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6471 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a97fdb4566
catch for NPE in image parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6470 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
534182559c
removed concurrency hacks from SplitTable because it showed deadlock-like situation.
...
see thread dump at http://forum.yacy-websuche.de/viewtopic.php?p=18081#p18081
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6468 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1fa0ac26e9
better protection against NPEs during search/ranking
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6467 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4c99d4683d
possible fix for lost crawl profile handles: clean-up job did wrong measurement to see if crawl is still running.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6465 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
cd6745b292
accept rss feeds without channel descriptions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6464 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
08f1cbb125
another update to the pdf parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6463 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
54c54fb144
get a handle for grep: 'StackTrace'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6462 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
605e896d6c
more details for exception catching when parsing pdfs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6461 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
18b21eaffe
small fixes to search default values and server logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6460 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
6edc168cfe
option to disable dht by memory limit:
...
memory.acceptDHT in kbytes
not yet pre-enabled, will clear on every startup
please review since this could break dht in freeworld
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4431b9767e
added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e3025ee691
- new icon for OAI-PMH loading action
...
- added many stack trace outputs for exceptions in crawl profile handler to find the 'missing profile handle' bug
- catched one more timeout exception in httpd file loader
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6457 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f0b8db93f0
- more abstraction of serverCore thread access
...
- no more keep-alive when number of connections exceeds 1/2 of the allowed number of connection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6456 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
19f31bb043
- moved OAI-PMH source list file from SETTINGS to DICTIONARIES/harvesting
...
- added convenience method for loading of files from the web in LoaderDispatcher
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6455 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2889b9426e
missing code for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6454 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b6a8887ff5
better handling of running sessions without explicit hashtable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6453 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1dc7ea986a
added a dynamic keep-alive time-out for http server sessions:
...
if there are many concurrent server sessions, the timout is decreased.
This should avoid a situation where the clean-up thread is too
late to stop running http sessions that should be terminated
before the maximum number of server sessions is reached.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6452 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
e77c906673
*) minor changes mainly in comments
...
*) added svn:keyword settings for several files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6451 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
f1740edbf8
*) added skript to change memory settings, password and port (experimental, don't blame me if it messes up your configuration)
...
*) minor change in Digest class, added option in main method, might not be optimal yet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6450 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
11f7da06ed
- fixes to csv parser
...
- automatic OAI-PMH import by just clicking on one link from the provided resource list
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6449 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9b6762ec2e
- added a csv "comma separated values" parser to parse OAI-PMH sources from
...
http://roar.eprints.org/index.php?action=csv
- integrated the csv parser into the crawlers parser list
- added an extension to the OAI-PMH import function to download and show the roar csv file using the csv parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6448 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
176e334aa4
fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6446 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2fa6bf440b
workflow update to OAI-PMH importer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6445 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b0b7a4f9a5
- added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records
...
- added monitoring for retrieved records
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago