63f39ac7b5
added 3 new crawling steering options:
...
- re-crawl by age of page (enter in minutes)
- auto-domain-filter
- maximum number of pages per domain
NOT YET TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1949 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
1fc3b34be6
some pre-work (without function yet) to implement:
...
- re-crawl (by age of last crawl)
- auto-crawl-filter by crawl depth (to be explained..)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1948 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
bab74b0499
*) escaping special chars in the url properly
...
- was a problem for QuickCrawlLink_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1607 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
6d73c6b481
*) adding xml version for QuickCrawlLink_p.java (used by yacybar)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1465 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
3feeba3d7b
*) better handling of urls containing query parameters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1445 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
a04930f025
code cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1158 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
6f8d7d3bcd
*) Adding first version of YaCy bookmarklet
...
- this can be used to easily crawl a webpage which is currently opened in the browser
- to get the bookmarklet javascript simply call http://localhost:8000/QuickCrawlLink_p.html
and drag and drop the link shown to your Browsers Toolbar/Link-Bar.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1053 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter