Crawling Depth:
This defines how often the Crawler will follow links embedded in websites.
Crawling Filter:
This is an emacs-like regular expression that must match with the URLs which are used to be crawled.
Use this i.e. to crawl a single domain. If you set this filter it makes sense to increase
the crawling depth.
Re-Crawl Option:
Use:
If you use this option, web pages that are already existent in your database are crawled and indexed again.
It depends on the age of the last crawl if this is done or not: if the last crawl is older than the given
date, the page is crawled again, otherwise it is treated as 'double' and not loaded or indexed again.
Auto-Dom-Filter:
Use:
This option will automatically create a domain-filter which limits the crawl on domains the crawler
will find on the given depth. You can use this option i.e. to crawl a page with bookmarks while
restricting the crawl on only those domains that appear on the bookmark-page. The adequate depth
for this example would be 1.
Maximum Pages per Domain:
Use:
You can limit the maxmimum number of pages that are fetched and indexed from a single domain with this option.
You can combine this limitation with the 'Auto-Dom-Filter', so that the limit is applied to all the domains within
the given depth. Domains outside the given depth are then sorted-out anyway.
Accept URLs with '?' / dynamic URLs:
A questionmark is usually a hint for a dynamic page. URLs pointing to dynamic content should usually not be crawled. However, there are sometimes web pages with static content that
is accessed with URLs containing question marks. If you are unsure, do not check this to avoid crawl loops.
Store to Proxy Cache:
This option is used by default for proxy prefetch, but is not needed for explicit crawling.
We recommend to leave this switched off unless you want to control the crawl results with the
Cache Monitor .
Do Local Indexing:
index text:
This enables indexing of the wepages the crawler will download. This should be switched on by default, unless you want to crawl only to fill the
Proxy Cache without indexing.
Do Remote Indexing:
If checked, the crawler will contact other peers and use them as remote indexers for your crawl.
If you need your crawling results locally, you should switch this off.
Only senior and principal peers can initiate or receive remote crawls.
A YaCyNews message will be created to inform all peers about a global crawl , so they can omit starting a crawl with the same start point.
Exclude static Stop-Words
This can be useful to circumvent that extremely common words are added to the database, i.e. "the", "he", "she", "it"... To exclude all words given in the file yacy.stopwords from indexing,
check this box.
Starting Point:
Existing start URLs are re-crawled.
Other already visited URLs are sorted out as "double".
A complete re-crawl will be available soon.