orbiter
2751c52617
layout
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7415 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
387db84087
maybe found bug in non-working index dumper
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7414 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4bce2f9c20
added a (temporary) description for 27c3 search set-up that everyone can see if they use YaCy here.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7413 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
89ae6101b9
fix for NPE and added comment in search result
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7412 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0769f4caa6
added search suggestions for interactive search: is only shown if there are no search results
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7411 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a4c9d27287
- moved some variables from Stwitchboard to new class AccessTracker
...
- added a limitation in access tracking to delete queries which are older than 10 minutes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7410 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
0c79789517
* labels for index delete checkboxes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7409 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
e4aabaa1c3
* fix negative filelength for files >2G
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7408 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
cdfe8afe3f
fix for really bad table iteration implementation: reduction of IO
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7407 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
9eae33f886
*) Ooops...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7406 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
a001e8075c
*) minor enhancements
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7405 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ecfacabad0
no db lookup during search / avoid IO
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7404 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
11ea966f9e
*) added SID file (Commodore 64) sound file parser
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7403 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
ee3cef91e8
* fix filesize in ftp crawls
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7402 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b2ed4cfaf8
more small bugfixes and light refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7401 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
1b6702146f
remove '*' from query string (people believe that this is a wild card)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7400 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3ca06d6290
patch for http://forum.yacy-websuche.de/viewtopic.php?p=21460#p21460
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7399 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
903c824c2c
- allow only scanned resourced with granted status
...
- increased time-out when scanning an ip range
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7398 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
936e976c23
*) added FreeMind ( http://freemind.sourceforge.net/ ) mindmap parser
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7397 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
3d95981f7d
*) cleaning up the code a little bit
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7396 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
2a6499364d
*) minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7395 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
c0274bd123
*) minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7394 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e38217fe88
small changes to scanner
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7393 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
fe46536f6e
enhanced network scanner (less name resolving during scanning and no name resolving during search)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7392 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a083cf531e
added skin for 27c3
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7391 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e753027c43
fix for http://forum.yacy-websuche.de/viewtopic.php?p=21439#p21439
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7390 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
bf4ef1513e
- fix for map view
...
- remove some UNRESOLVED PATTERN
- maybe a fix for non-flushing cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7389 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6b70393d1d
- new java version 1.6
...
- replaced old gif animator by java 1.6 gif animator
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7388 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e88c428008
fix to ftp loader
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7387 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
59b70a5a92
another fix to the ftp crawler: now correct directory listings according to rfc2640 (path with spaces) and better title names for such files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7386 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
9b25a33fd9
- fixed numerous bugs
...
- better document names
- fixed problem with ftp crawling
- added automatic removal of search results from services that are not online according to the latest network scan: this does not delete the index but just does not show them. after the next network scan when the server is available again, the results are again showed.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7385 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7bdb13bf7f
more fixes to smb crawling: better file names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7384 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
94c48500cc
several fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7383 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0ac7311a62
fix for token parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7382 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
58b59f9bc8
- a collection of bug fixes and some redesign of the Scanner class
...
- fixed smb crawling
- added smbget to download script generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7381 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
4d5bb4c4ca
*) Urks...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7380 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c54170421a
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7379 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c288fcf634
redesigned CrawlStartScanner user interface and added more features:
...
- multiple hosts for environment scans can be given (comma-separated)
- each service (ftp, smb, http, https) for the scan can be selected
- the scan result can be accumulated or refreshed each time a network scan is made
- a scheduler was added to repeat a scan and add all found urls to the indexer automatically
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7378 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
6f4f957e50
*) cleaning up the code a little bit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7377 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
2521677a45
* deny adminForLocalhost and intranet network setup also on bootup and not only on network switch
...
* require authentication for yacybot what ever adminForLocalhost is set to
(after this patch, is the rule from above really nesseccary,
the crawler also checks the robots.txt)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7376 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
9d2159582f
* fix system update if urls are in blacklist (for example for very general blacklists like *.de)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7375 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
56264dcc17
- added CamelCase parser to MultiProtocolURI: generate better to-be-indexed words from urls
...
- integrated new parser into loader processes: enrich document parser
- fixed a concurrent modification exception in kelondro iterator
- hand-over of document size from crawler to indexer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7374 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
358feeeb39
enhanced speed and usability of network scanner servlet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7373 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
99a7fe87f9
- removed old intranet scanner (the generic scanner now completely subsumes the old one)
...
- added information about granted access
- enhanced servlet design
- added submit-feedback (because it is a long-running task)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7372 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
acab6801d9
added new network scanner
...
- you can scan any ip or host in the internet for services
- this replaces the intranet scanner
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7371 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
586cbee2bb
*) Fixed links to CrawlStart-page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7370 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
14e4fae8e9
fixes to ftp client
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7369 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a563b05b60
enhanced crawler:
...
- added a new queue 'noload' which can be filled with urls where it is already known that the content cannot be loaded. This may be because there is no parser available or the file is too big
- the noload queue is emptied with the parser process which indexes the file names only
- the 'start from file' functionality now also reads from ftp crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7368 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c36da90261
added a very fast ftp file list generator to site crawler:
...
- when a site-crawl for ftp sites is now started, then a special directory-tree harvester gets the complete directory structure of a ftp server at once
- the harvester runs concurrently and feeds into the normal crawl queue
also in this:
- fixed the 'start from file' crawl function
- added a link detector for the html parser. The html parser can now also extract links that are not included in <a> tags.
- this causes that a crawl start is now also possible from clear text link files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7367 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4565b2f2c0
removed the display option from index.html, yacysearch.html and yacyinteractive.html
...
instead, a setting at ConfigPortal.html can be made to define if the topmenu shall be shown at these pages or if there is no naviagtion at all.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7366 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago