lulabad
c4c0d54b22
* added regex extended blacklistengine
...
* removed my own engines
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4618 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
368593e449
enhanced the concurrency handling of indexing process (better queue size control, better data concept, better shutdown behavior)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4617 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
low012
4c3f1b67ad
*) refactoring of Blacklist_p.java (moving entries might be slightly slower, but the code is more tidy now)
...
*) added edit functionality for blacklist entries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4616 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
466d49e90c
* added login-parameter to be able to force authentication
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4613 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
be58135b3e
possible fix for deadlock in search execution
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4612 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
c67350f138
* use putXML with forXML-parameter to ensure urls are valid xml (problem was & in url)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4611 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
f1ori
98a03249f3
- optimize selectall-javascript-function (in firefox more then 10 times faster now)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4610 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
0241d070bc
added concurrency to indexing process:
...
- the methods {parsing, semantic analysis (condensing), structure analysis (web structure)} in the serialized indexing path had been made concurrent.
- four BlockingQueues handle concurrency and hand-over of the indexing objects, the last object in the queue is stored into a blockingQueue of maximum size 1 to serialize the process for storage (which uses IO and therefore here should not be deserialized)
- a concurrency of (CPUs + 1) is default. Single-CPU users will profil from the change because large files cannot block the indexing process any more.
- removed the secondary indexing thread, which is superfluous now. Concurrency is default for all users.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4609 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
9fb5d661f2
added my Blacklistengines
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4608 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
bca87f1e38
- refactoring of serverThreads: renaming to distinguish busy-threads and blocking-threads
...
- added blockingThreads which are threads that are not driven by pause times but by BlockingQueue lookup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4606 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
968c775025
- preparation of parsing/indexing queue for concurrent execution
...
- remote crawl receipts are now transmitted concurrently in separate threads (makes remove crawls much faster!)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4605 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
9b0e20fb06
next refactoring step in document indexing to prepare concurrency environment for document parsing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4604 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7f9f639d20
- refactoring and abstraction of index reference (urls) handling: blacklisting is part of reference filtering
...
- refactoring of word/phrase handling: word abstraction from condenser becomes part of index element handling
- removed unused code parts from condenser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4603 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d6050b9ffb
- separated the LURL data storage and Crawl result stack for process supervision.
...
this is another step to enable multiple, concurrent fulltext-indexes
- another try to make the yacy-httpc more stable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4602 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
low012
8e889de50b
*) Added Lotus' patch ( http://forum.yacy-websuche.de/viewtopic.php?t=979 ), user will be taken back to last opened page after making changes in Advanced Settings.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4601 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
8d6a13bc07
refactoring of parsing-condensing-indexing process:
...
- separated parts
- removed storagePeer function
next step will be parallelization of processes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4600 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
d3b06913ec
protection against seed-db failure during enumeration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4598 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
5aa96dbc36
fix for shutdown configuration
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4596 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
93633abed8
- removed some debugging code from search process - should speed up now
...
- added some profiling code to search event - more time details in PerformanceSearch_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4594 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
fba46c51d7
fixed non-termination bug in qsort
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4593 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
541b817502
refactoring of switchboard queueing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4591 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
fc94fbe224
another improvement to the collection sorting
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4589 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
11270d450e
better quicksort-pivot computation: 30% faster (measured with test program)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4588 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
3e44293f07
- fixed a problem with thread pools in row collection
...
- added a line-viewing feature in threaddump
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4587 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
e43051b125
- fixed Threaddump output (html-escaped ie. <init>)
...
- in EcoFS converted comments to javadoc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4586 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
433ff855f7
- fixed another concurrency problem in collection sorting
...
- fixed a typing problem that was introduced in svn 4579 and caused the crawl monitor to fail
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4585 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
19286fa2d1
tried to fix seed2.old.db-problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4584 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
5fde618337
changed display of y-marks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4583 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
f3996e63b8
tried to fix more deadlocks:
...
- changed connection modes in ftpc
- replaced sort tread pool in row collections by new one using util.concurrent. the old pool had caused blockings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4582 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
daburna
26155cbb5e
update
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4581 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
daburna
1618bd5505
added submenu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4580 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
7008a218b3
avoid ConcurrentModificationException in plasmaCrawlerQueues
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4579 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
9d0af17c5b
removed debug from startYACY.sh *ups*
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4578 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
bf3e5807b0
fixed typo and remove redundant code from startYACY.sh
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4577 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
ea59dbf4c2
fixed build problem see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=956&hilit=
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4576 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7150b463ff
changed handling of default values and database paths:
...
- the default files yacy.init and for the network definition is now moved to the path defaults
- the httpProxy.conf is renamed to yacy.conf
- the DATA/INDEX/PUBLIC is renamed to the actual network nickname, which should be freeworld or sciencenet
more menu entries
- added apfelmaennchens alternative search page to the menu
- added the new thread dump page to the server log menu point as submenu
modifications
- modified the thread dump page: sorting by thread type
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4575 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
54cb097ea4
added .trigger("update") after paging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4574 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
82f17ccee2
just an example sidebar
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4573 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
apfelmaennchen
3c710f22cd
added server side driven pagination for search tabs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4572 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
25f5035f23
typo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4571 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
7fd094fcbe
small bug in ftpc: did cot compile in Java 1.5
...
Please set compiler to Java 1.5-compliance
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4570 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
0a593a4e4a
small changes to Threaddump Servlet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4569 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
995b5a2b27
added Threaddump Servlet, see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=862
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4568 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
f51bad8ae5
FTP:
...
- report connection status (to break if no connection possible)
- fixed isFolder()
- additional error output
- fixed paths with encoded symbols (ie. a%20file.txt)
- refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4567 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
820641938e
ftpc: fixed date parsing, some refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4566 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
4c584dff87
disabled soLinger to prevent that too many connections stay open (it's a TEST!)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4565 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
d1b9f50b94
added "delete Release" to Update Page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4564 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
lulabad
3d122c125c
added "current installed release" to Update Page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4563 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
orbiter
9c989fe5f7
fixed deadlock
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4562 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago
danielr
c565906050
FTP:
...
- added maxFileSize-check
- added timeout for download
- fixed dirlist (when all filenames have spaces, change to absolute links)
- enhanced isFolder()
- make sure data connection is closed, so a new can be opened
- refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4561 6c8d7289-2bf4-0310-a012-ef5d649a1542
17 years ago