Commit Graph

676 Commits (6f9ed439d3c34cc61dd4b7b2b46015018e353acf)

Author SHA1 Message Date
Michael Peter Christen b28d43decc added two more fields source_cr_host_norm_i,target_cr_host_norm_i in
11 years ago
Michael Peter Christen a52f3a597e fix for canonical-from-http-header feature
11 years ago
Michael Peter Christen 2dd7c5be44 added parsing of http-canonical tags (untested, could not find an
11 years ago
Michael Peter Christen 3bf0104199 fix for crawl domain counter limitation (limit was reached too early)
11 years ago
Michael Peter Christen 82bfd9e00a - crawl profiles shall be deleted from active and passive stacks if they
11 years ago
Michael Peter Christen 91a875dff5 self-healing of mistakenly deactivated crawl profiles. This fixes a bug
11 years ago
Michael Peter Christen 095053a9b4 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
sixcooler 0cae420d8e some dns-timing changes:
11 years ago
Michael Peter Christen 4f83d5f18c added the new field harvestkey_s to the collection index and the
11 years ago
orbiter 14442efa6d when profiles are cleaned, there shall be first a callback showing which
11 years ago
orbiter 8ac2e8c8c9 added location navigator which causes that the image to the map search
11 years ago
Michael Peter Christen 96ed0c980e - added hosthash to all documents (also fail documents which is needed
11 years ago
orbiter 828603e4f1 fix for 100%CPU problem in error cache cleaning process
11 years ago
orbiter c64b51134e hack to add all tokens from the url to text_t. This was working for the
11 years ago
orbiter f3be1930cb CPU problem when pusing to the error cache; wrong class,
11 years ago
Michael Peter Christen e40671ddb7 better and consistent deletions for error urls
11 years ago
Michael Peter Christen 2602be8d1e - removed ZURL data structure; removed also the ZURL data file
11 years ago
Michael Peter Christen 31920385f7 set anchor rel attribute of all links to "nofollow" if the html meta
11 years ago
Michael Peter Christen 61c5e40687 - replaced the properties object in AnchorURL with distinct variables
11 years ago
Michael Peter Christen 5e31bad711 - the webgraph shall store all links which appear on a web page and not
11 years ago
Michael Peter Christen 35ab2cef7b added parsing of 'date', 'dc:date', 'dc.date' and 'last-modified' in
11 years ago
Michael Peter Christen 9cc8468b30 added tools to visualize image generation (i.e. during testing)
11 years ago
Michael Peter Christen dbef8ccfcb forced deletion of ZURL entries for a specific host for each host that
11 years ago
Michael Peter Christen e137ff4171 refactoring (im preparation for new removeHost method)
11 years ago
Michael Peter Christen 7a5574cd51 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
Michael Peter Christen 85456f46b2 added two new fields, exact_signature_copycount_i and
11 years ago
orbiter 26366596d9 fix for a problem which ocurres when a site is crawled where the start
11 years ago
Michael Peter Christen a2511b5600 turned images_alt_txt back to images_alt_sxt because it is not necessary
11 years ago
Michael Peter Christen 85b1922244 activated image type navigation for image search
11 years ago
Michael Peter Christen 9e12fdff23 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
11 years ago
Michael Peter Christen ab1201fdfd fixed wrong facet count
11 years ago
Michael Peter Christen 049c3b3f2e added an option to exclude image search results from text search. This
11 years ago
Michael Peter Christen 69f85265e1 added an option to put image links to the crawl queue and handle these
11 years ago
Michael Peter Christen a8c5bfcf58 avoid to create unnecessary objects
11 years ago
Michael Peter Christen 5a0de1b77d moving image description text to image text field
11 years ago
Michael Peter Christen dc179bd61f fix for catchall query goal for image search
11 years ago
reger 392174de8c remove all_words, all_strings lists from QueryGoal
11 years ago
Michael Peter Christen 169ef8963d one more fix for image search
11 years ago
Michael Peter Christen cb85b22725 redesign of the image search process (with much better results,
11 years ago
reger 29967102a2 optimized QueryGoal (reducing mem and computation by removing all_hashes)
11 years ago
orbiter f106345eef link strings should not be tokenized
11 years ago
orbiter deadeb406e image alt tag strings should be tokenized
11 years ago
Michael Peter Christen 1a3e42eca4 index migration to lucene 4.4
11 years ago
Michael Peter Christen a88a62f7aa added a feature to set a collection for a crawl result based on a
11 years ago
Michael Peter Christen 765943a4b7 Redesign of crawler identification and robots steering. A non-p2p user
11 years ago
Michael Peter Christen 47b1c81d08 - refactoring
11 years ago
Michael Peter Christen 697613170d less logging for postprocessing (this was a debugging logging with high
11 years ago
reger a5019bc470 make Vocabulary Navigator tags a hard result entry filter
11 years ago
reger a67a4b7d86 improve tld: query modifier filter pattern (to prevent tld:net accepting www.abcinet.org)
11 years ago
reger 02fe8b43ba Field Re-Indexing: display list of fields in reindex queue
11 years ago