Commit Graph

374 Commits (01cc32217fb2dccaf20284261579addca0d42474)

Author SHA1 Message Date
Michael Peter Christen 8b32dd5f9e special strategy for balancer: do not remove targets with zero wait time
11 years ago
Michael Peter Christen 9c6228d948 fix for deadlocks in crawler
11 years ago
Michael Peter Christen 10cf8215bd added crawl depth for failed documents
11 years ago
Michael Peter Christen 06afb568e2 new Strategies in Balancer:
11 years ago
Michael Peter Christen da86f150ab - added a new Crawler Balancer: HostBalancer and HostQueues:
11 years ago
Michael Peter Christen 075b6f9278 refactoring of the crawl balancer: the balancer is turned into an
11 years ago
Michael Peter Christen 6bd8c6f195 fix for wrong status codes of error pages
11 years ago
Michael Peter Christen 9e503b3376 also delete the robots.txt file from the cache when a new crawl is
11 years ago
Michael Peter Christen 1c21b3256d fix for robots.txt handling: delete old entry before starting a new
11 years ago
Michael Peter Christen 926d28dd3f fixed a bug which prevented crawl starts after a network switch
11 years ago
Michael Peter Christen d4b5c457e4 NPE fix
11 years ago
Michael Peter Christen 8b44fcf0f4 added missing @Override annotation
11 years ago
Michael Peter Christen 85a427ec54 support for multiple sitemaps in robots.txt
11 years ago
Michael Peter Christen b08375da33 fix for bad/missing values of size_i
11 years ago
reger dd5bf0b71b cleanup old reference to HTTPDemon.setAlternativeResolver
11 years ago
Michael Peter Christen e485fbd0ce - let crawl loader jobs die after 10 seconds without new jobs
11 years ago
Michael Peter Christen bcd9dd9e1d enhanced concurrent loading by using a fixed set of concurrent loader
11 years ago
Michael Peter Christen 6ed9c0164e attaching names to all Threads to get a better view in profiling tools
11 years ago
Michael Peter Christen fdaeac374a - enhanced postprocessing speed and memory footprint (by using HashMaps
11 years ago
orbiter da5d4128bf prevent npe
11 years ago
orbiter a878c7982c prevent npe
11 years ago
orbiter ced1a96f9c fixed error cache
11 years ago
Michael Peter Christen 69391e5d9e changed strategy to test existence of documents in Solr: using the
11 years ago
Michael Peter Christen 8b14e92ba4 added button in host browser to re-load 404/failed documents
11 years ago
Michael Peter Christen 6ada0daae9 making latency_factor and maximum number of same hosts in loader queue
11 years ago
Michael Peter Christen 0168f80c28 new crawling factors can now be changed during runtime
11 years ago
Michael Peter Christen 77531850b5 reverted crawling strategy from latest commit.
11 years ago
Michael Peter Christen c0da966dfa enhanced crawler speed
11 years ago
Michael Peter Christen 0d235a565b cleanup crawl loader jobs
11 years ago
Michael Peter Christen 1ea17bd9f3 - removed old metadata database and all migration code
11 years ago
Michael Peter Christen 022c6d3ce1 do YaCy p2p connections using a timeout-request which covers the http
11 years ago
reger 28eae57e8b spend CrawlQueues a fremem routine
11 years ago
reger 6932aa4d7a use configured admin-username for api calls
11 years ago
orbiter 3cb6c7861f fixed shutdown authenticaton problem
11 years ago
orbiter f3ac923a7e ftp client shall be able to open non-anonymous ftp servers if login
11 years ago
Michael Peter Christen 82c0525e71 wrong logger fix
11 years ago
Michael Peter Christen 552ef9f18e fix for bad ErrorCache.exists test (bug from latest commit)
11 years ago
Michael Peter Christen 303f5694ba avoid usage of existsByQuery. If a document can be loaded by the ID
11 years ago
Michael Peter Christen 0db8e34625 enhanced webgraph processing
11 years ago
Michael Peter Christen 1a4a69c226 set more logger to 'final static'
11 years ago
Michael Peter Christen 87a956e881 calculating and showing the number of files and the average size of a
11 years ago
Michael Peter Christen 234a974955 load image only if their parser flag is activated
11 years ago
Michael Peter Christen 030d0776ff Enhanced crawl start for very, very large crawl lists (i.e. > 5000)
11 years ago
Michael Peter Christen 4948c39e48 added concurrency for mass crawl check
11 years ago
Michael Peter Christen 1b4fa2947d - fixed a problem which ocurred when a document was not recognized with
11 years ago
orbiter 20bbde8665 fix for mustmatch regex computation: result had correct semantic, but
11 years ago
Michael Peter Christen 74d0256e93 enhanced postprocessing: fixed bugs, enable proper postprocessing also
11 years ago
Michael Peter Christen 101a6e6e14 Patch the citation index for links with canonical tags.
11 years ago
reger fd119deb00 fix NPE on modified since check ( Response.requestHeader allowed to be null)
11 years ago
Michael Peter Christen b28d43decc added two more fields source_cr_host_norm_i,target_cr_host_norm_i in
11 years ago
Michael Peter Christen 3bf0104199 fix for crawl domain counter limitation (limit was reached too early)
11 years ago
Michael Peter Christen 82bfd9e00a - crawl profiles shall be deleted from active and passive stacks if they
11 years ago
Michael Peter Christen 91a875dff5 self-healing of mistakenly deactivated crawl profiles. This fixes a bug
11 years ago
Michael Peter Christen 4f83d5f18c added the new field harvestkey_s to the collection index and the
11 years ago
orbiter 14442efa6d when profiles are cleaned, there shall be first a callback showing which
11 years ago
orbiter 0013d0d0bb removed superfluous class
11 years ago
orbiter f90d5296cb Added new data structure to be used by the balancer (not used yet).
11 years ago
orbiter 0e8d752462 refactoring
11 years ago
Michael Peter Christen e40671ddb7 better and consistent deletions for error urls
11 years ago
Michael Peter Christen 2602be8d1e - removed ZURL data structure; removed also the ZURL data file
11 years ago
Michael Peter Christen 61c5e40687 - replaced the properties object in AnchorURL with distinct variables
11 years ago
Michael Peter Christen 5e31bad711 - the webgraph shall store all links which appear on a web page and not
11 years ago
Michael Peter Christen 1a8c64117f decreased the responseHeaderDB database which is now flushed more
11 years ago
Michael Peter Christen dbef8ccfcb forced deletion of ZURL entries for a specific host for each host that
11 years ago
Michael Peter Christen e137ff4171 refactoring (im preparation for new removeHost method)
11 years ago
orbiter 26366596d9 fix for a problem which ocurres when a site is crawled where the start
11 years ago
Michael Peter Christen 69f85265e1 added an option to put image links to the crawl queue and handle these
11 years ago
Michael Peter Christen a88a62f7aa added a feature to set a collection for a crawl result based on a
11 years ago
Michael Peter Christen e4cbe9232d fixed a crawler bug where a double-occurring url was not re-crawled
11 years ago
Michael Peter Christen 765943a4b7 Redesign of crawler identification and robots steering. A non-p2p user
11 years ago
Michael Peter Christen 47b1c81d08 - refactoring
11 years ago
Michael Peter Christen dbfa865700 added a stub of a class for crawler redesign
11 years ago
Michael Peter Christen 58fe986cca Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
Michael Peter Christen cf12835f20 replaced the single-text description solr field with a multi-value
11 years ago
orbiter 268a36aaff emergency fix for crawler: this will otherwise cause loss of complete
11 years ago
reger 2b7a38640a extend content type detection on file extension for .tif .tiff .htm
11 years ago
Michael Peter Christen 735a66eff3 enhancements to crawler
11 years ago
Roland Haeder 841a28ae76 Added 'final' for all exception blocks as this helps the Java compiler
11 years ago
Michael Peter Christen 89c0aa0e74 added collection_sxt to error documents
11 years ago
Michael Peter Christen c6a6f159e8 fix for crawl stack domain counter
11 years ago
Michael Peter Christen bcc623a843 refactoring of load_delay: this is a matter of client identification
12 years ago
orbiter 3978c5ca5d fix for http://bugs.yacy.net/view.php?id=255
12 years ago
orbiter dac88561ae minimum access time has a tight connection to ClientIdentification,
12 years ago
Michael Peter Christen 5c6946dd5f replaced usage of log4j by ConcurrentLog where possible
12 years ago
Michael Peter Christen 5878c1d599 - refactoring of log to ConcurrentLog:
12 years ago
Michael Peter Christen a34e137e27 fix for citation index generation in case that entry.referrerhash() is
12 years ago
sixcooler 9551720d5c re-enable saved setting for proxy-crawl-profile
12 years ago
Michael Peter Christen 57ffdfad4c added a crawl option to obey html-meta-robots-noindex. This is on by
12 years ago
Michael Peter Christen fa08bd9d5a hack to prevent long waiting times in crawler
12 years ago
Michael Peter Christen f1c5338210 prepartion for greedy crawl profiles and refactoring
12 years ago
Michael Peter Christen 203921006a redesign of citation index storage
12 years ago
Michael Peter Christen 16d1d744fa added url_file_name_s in default collection schema for the file name
12 years ago
Michael Peter Christen 374d2e2a52 removed warning message during crawling
12 years ago
Michael Peter Christen 2fd7bbb450 reduced load on solr; no seed update in Status and no exists-check in
12 years ago
Michael Peter Christen 2648b42b27 added fixed clear method as public method
12 years ago
Michael Peter Christen e20450e798 patch in HTCache and CitationIndex loading in case that a file is
12 years ago
reger 7480e87386 - fix stopword handling for RWI see example http://bugs.yacy.net/view.php?id=247
12 years ago
Michael Peter Christen 8f2d3ce2f9 reduced locking situation in crawler: shifted synchronized location and
12 years ago
Michael Peter Christen 06d3063dc9 - no downcase when using collection modifier
12 years ago
Michael Peter Christen 8dbc80da70 redesign of index.exist-test: this shall now not be done using a single
12 years ago
Michael Peter Christen 44e363f37f refactoring of WorkflowProcessor, added process counter, update of
12 years ago
Michael Peter Christen 77faeada4d small memory leak patch
12 years ago
Michael Peter Christen 038f956821 fix for sitemap detection: the sitemap url was not visible if it
12 years ago
Michael Peter Christen bb4bf3d8fd infinity timeout bug protection patch
12 years ago
Michael Peter Christen 3a0fcfbeda Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
12 years ago
Michael Peter Christen 25499eead5 - added a new field for the regular expression in crawl start
12 years ago
orbiter e1bfe9d07a - reduction of the concurrently running processes to make YaCy more
12 years ago
Michael Peter Christen c091000165 added collection attribute also to the rss feed reader
12 years ago
Michael Peter Christen 252bb51f98 fix for wrong mime type in noload crawler
12 years ago
orbiter 2555542f7a removed the dns prefetch because that was not soo useful
12 years ago
Michael Peter Christen 788288eb9e added the generation of 50 (!!) new solr field in the core 'webgraph'.
12 years ago
Michael Peter Christen 91a0401d59 introduced a second core named 'webgraph'. This core will hold the link
12 years ago
Michael Peter Christen b6de1f42dc Full redesign of solr connection architecture. This was done to support
12 years ago
Michael Peter Christen 0b6566a389 optimizations when starting large crawl requests with many start urls in
12 years ago
Michael Peter Christen 0fe7b6fd3b migrated the index export methods from the old metadata to solr. Now
12 years ago
Michael Peter Christen 9ccdd21d76 Merge remote-tracking branch 'aleksejs/fixtrans'
12 years ago
Michael Peter Christen af465cdca5 fix for wrong robots.txt loading for https protocol
12 years ago
Michael Peter Christen 38d3feae65 added separate delete commands for the local+remote solr index, the old
12 years ago
reger 276e63401e small sanitary fixes
12 years ago
Michael Peter Christen 8f3bd0c387 fix for smb crawl situation (lost too many urls)
12 years ago
Michael Peter Christen 7c3de8b4cd - fix for localhost detection
12 years ago
orbiter 712cc37c40 if maxFileSize < 0 then the file size limit is without limit.
12 years ago
Michael Peter Christen a3cd3852ab introduced a better place to update the lastacc time value in latency
12 years ago
Michael Peter Christen 864abcd33d removed Latency update after URL selection because that causes
12 years ago
Michael Peter Christen dd241d03bb latency fix: only set last-visit time if access was actually by the
12 years ago
Michael Peter Christen 1e002ab18e added another blacklist-cleaner into balancer
12 years ago
Michael Peter Christen 10527e28ae fix for wrong display of error urls in HostBrowser
12 years ago
Michael Peter Christen 756772fbd3 fix for waitingtime computation for intranet configuration
12 years ago
Michael Peter Christen fa27e5820f - check blacklist (again) when taking urls from the crawl stack because
12 years ago
Michael Peter Christen 72f165d58b added a Boost class which stores solr query boost values. The class can
12 years ago
Michael Peter Christen 3de784c8dd replaced more split and replaceAll missing pattern pre-compilation with
12 years ago
Michael Peter Christen d48e9788d2 enhanced search result processing behavior
12 years ago
Michael Peter Christen eca68fa197 added debug code to crawler monitor
12 years ago
orbiter 5aa5202adf fixes for filesystem indexing
12 years ago
Michael Peter Christen efd2c4622d added a new fail type attribute for the index to distinguish two
12 years ago
Michael Peter Christen 5e182a566f - added another enumeration method in kelondro data structure to get a
12 years ago
Michael Peter Christen d6b82840f8 added a feature to find similarities in documents.
12 years ago
Michael Peter Christen f5ca5cea44 - added field options to all solr queries. This can be used to restrict
12 years ago
cominch d2a94cc55e refactor package
12 years ago
Michael Peter Christen 8b1c9cba3d fixed a problem with non-terminating crawls
12 years ago
Michael Peter Christen 71ed8e5e07 bugfixes for crawler
12 years ago
Michael Peter Christen 158732af37 automatically delete entries from the crawl profile list if crawl is
12 years ago
Michael Peter Christen d481abd087 added the visualization of error-urls to host browser
12 years ago
Michael Peter Christen 791e1dcfdf when a new crawl is started, delete all entries about error-urls for
12 years ago
orbiter 354ef8000d - added 'deleteold' option to crawler which causes that documents are
12 years ago
Michael Peter Christen 75dd706e1b update to HostBrowser:
12 years ago
Michael Peter Christen 0716a24737 added more / all new crawl profile fields into crawl profile editor
12 years ago
Michael Peter Christen 4a14122ba7 in case that a crawl profile has a collection assigned, use the
12 years ago
Michael Peter Christen 0fe8be7981 enhaced data structures for balancer and latency computation which
12 years ago
Michael Peter Christen ac9540dfb6 removed options for stopwords which are not used
12 years ago
Michael Peter Christen b2ffd49817 less latency
12 years ago
Michael Peter Christen 0833937c1c better balancing and duetime-cumputation also for no-delay intranet
12 years ago
Michael Peter Christen c326aa8f67 disabled writing new entries to crawl stacks to prevent that a domain
12 years ago
Michael Peter Christen c25d7bcb80 - added concurrency for robots.txt loading
12 years ago
Michael Peter Christen a87811bc38 more auto-commit calls when a search interface is opened, but not when a
12 years ago
Michael Peter Christen 2d9e577ad0 replaced the custom robots.txt loader by the standard http loader
12 years ago
Michael Peter Christen a33e2742cb - removed unnecessary synchronized and deadlock in crawler
12 years ago
orbiter 8952153ecf update to Balancer algorithm:
12 years ago
Michael Peter Christen 85ca07b90e when a new crawl is started, an equal crawl, if still running, is
12 years ago
Michael Peter Christen ae6feb5610 showing the web structure graph as animation in the crawl monitor
12 years ago
Michael Peter Christen ccc3760a47 Refactoring and redesign of data architecture to make URIMetadataRow
12 years ago
Michael Peter Christen e5b3c172ff removed hack which translated Solr documents to virtual RWI entries
12 years ago
Michael Peter Christen 43f3345c90 - removed dependencies from URIMetadataRow and made direct access to
12 years ago
Michael Peter Christen 21fe8339b4 - enhanced generation of url objects
12 years ago
Michael Peter Christen 5f0ab25382 removed the option to prevent removal of &amp; parts inside of the
12 years ago
Michael Peter Christen 53789555b9 fix for crawl start filter
12 years ago
Michael Peter Christen a06930662c replaced some more .getBytes() with UTF8/ASCII.getBytes()
12 years ago
Michael Peter Christen 4b5e0c1500 added an url rewriter which can be used to remove session ids from urls
12 years ago
Michael Peter Christen 76d218fbef fixes to crawl profiles
12 years ago
Michael Peter Christen 1533bfd63b refactoring
12 years ago
Michael Peter Christen 872f83ebe0 refactoring
12 years ago
Michael Peter Christen 8219a445f3 refactoring
12 years ago
Michael Peter Christen f879a344e7 fix for no depth limit default value
12 years ago
Michael Peter Christen 00c1c777fa refactoring
12 years ago