65eaf30f77
redesign of crawl profiles data structure. target will be:
...
- permanent storage of auto-dom statistics in profile
- storage of profiles in WorkTable data structure
not finished yet. No functional change yet.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7088 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
3197ca42ed
preparations to move the HTCache into cora:
...
- move the header framework classes to cora
- move the ARC caching classes to cora
- refactoring of code to call these classes from cora
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7068 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
5e7081cd19
refactoring towards a unified loading mechanism for MultiProtocolURIs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7065 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
90531f78ff
refactoring of the cora package to get subpackages for http and ftp (smb to come)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7063 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
a82a93f2fc
- better url double check in crawler
...
- more logging for error urls
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7032 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
6950d8a33d
fixes to SMB crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6900 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
25aef069a6
continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6790 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
3300930fc5
- (almost) fixed FTP crawler
...
- integrated/fixed SMB crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6742 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
9623d9e6d2
added a smb loader component for the YaCy crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6737 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter