orbiter
89b4fff1c2
adopted ant script for new exif library
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6746 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
24e5faee75
added exif parsing for jpg images
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6745 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
82f76e1296
removed log line
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6744 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0f8004f9da
enhanced html parser to recognize a href tags inside header tags
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6743 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3300930fc5
- (almost) fixed FTP crawler
...
- integrated/fixed SMB crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6742 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1198b9989d
bugfixes, more sorttable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6739 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9623d9e6d2
added a smb loader component for the YaCy crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6737 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ae2f3f000f
better handling of table copy abandon .. prevent memory leak
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6734 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0769517129
added a robots.txt monitor in the crawler monitor submenu
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6733 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
72f00dee59
removed never-used server access account function
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6731 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
de01fe0e6d
fix for bug in url parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6722 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c4bdb1e7f2
added one more option in ViewFile to show an iframe like for the orginal web page content but using the cache than the direct link to the content in the web. Upgraded the very old and previously not any more used CacheResource_p servlet to a new and working version.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6719 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1bbe14d23f
SVN 6716 unfortunately contained parts of the unfinished SMB integration. To fix compile errors the remaining parts of the SMB implementation stub is added with this commit.
...
This adds the jcifs smb library.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6717 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
884b262130
- added a new Wiki Namespace Navigator
...
- some redesign of Navigator data structures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6716 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
270fb38674
- fixed some bugs in Table viewer
...
- added 'select all' feature in Tables_p
- enhanced ViewFile.html: has now an input field to load arbitrary resources from the web and analyze them (!!!)
- included the ViewFile servlet into the Index Administration menu
- show in ViewFile if ressource is in url-db and/or in Web cache
- bugfixes to BEncodedHeap and Tables management
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6713 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
727dd9b193
- fixed a bug in robots.txt parser
...
- moved storage of robots.txt entries to WorkTables, so it is now possible to browse the robots entries with the table browser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6710 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
54af9e6b49
- added parsing of robots meta-tag in html headers to detect a noindexing request
...
- added evaluation and indexing prevention in case that a noindexing is given in a html file
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6709 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
cd6de83905
next try for for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2703
...
(reverted 6692)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6694 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
bfe4693e9a
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2703
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6693 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
564927ce72
redesign of CrawlResult data structures because of OOM occurrences during URL deletion processes.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6675 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
30c8185139
fix for sid check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6673 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ef62d017e5
integrated session id filtering for crawler
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6672 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d8d9984913
added framework for session id filtering (not ready yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6671 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2bc36de336
- fix for bug in svn 6669
...
- cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6670 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d378ca4604
better handling of concurrency in seed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6669 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
6538043d89
fix for http://forum.yacy-websuche.de/viewtopic.php?p=19189#p19189
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6668 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
e071d71f19
fix for yacy-banner-network-values
...
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2521
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6659 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
787b588c33
reverted a part of svn6636:
...
- didn't work on blobs >2GB
- should be obsolete since svn6651
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2652&sid=7fa98fd3edfc2a03f26394d545e3e3c1&p=19172#p19172
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6655 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
11188cd7eb
resource observer now uses the Java 6 method to check for free space. thus, disk observing now needs Java 6 installed.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6652 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
sixcooler
089877f32c
my first commit - hopefully fix for merge problem
...
- http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2652
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6651 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d6391f2537
better handling of rewrite cases where the resulting rewrite blob entry is equal in size
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6648 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ef9473d92c
added another sixcooler suggestion: recycle corrupted records
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6647 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fe78edac32
- view API calls in correct date-order
...
- execute recorded API calls in date-order
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6646 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
308a973503
refactoring of tables data organisation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6644 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
85ca96227f
fix for re-enable parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6643 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ada0ce9de3
refactoring of bookmarks: there is a big performance problem in the bookmarks code and furthermore the bookmarks
...
will loose its leading role for the re-crawl funtion when the new api tables will work. To be prepared for a replacement
of such functions the bookmark class is re-organised.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6637 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3751ab4ae2
added sixcoolers patch and more checks/removed unnecessary code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6636 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d8d8562c59
fill key with zeros during normalization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6635 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
24060885b6
- added Tables abstraction in data.Tables.java
...
fix for
http://forum.yacy-websuche.de/viewtopic.php?p=18910#p18910
http://forum.yacy-websuche.de/viewtopic.php?p=18894#p18894
http://forum.yacy-websuche.de/viewtopic.php?p=18814#p18814
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6631 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7fdf59a77f
misc NPE check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6630 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
38a3d55afd
added more possible php extensions for html
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6621 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4403304957
bugfix for list()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6616 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
69c29acb6e
no exception thread dump if parser cannot parse becuase that mime-type/extension is in the deny-set
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6611 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0098e6e859
bugfix for heap iterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6610 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
db19a941cf
added new image index storage classes (not integrated yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6608 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c8aece34a4
update to yacy/ai (just more testing)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6607 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8ce936bcdd
added an api recording function: it shall be possible to record
...
all operations on YaCy in a database that should make it possible
1) to re-create a setting on fresh peers
2) to transmit a setting from one peer to another
3) to re-create crawl starts after a complete deletion of the index
This functionality will also support
4) scheduled re-crawls (new implementation)
To implement this, a new database structure has been crated that stores maps into blob heaps. to encode maps the b-encoding technique was used (this is the same encoding that torrent files use)
- added a b-encoder
- enhanced the b-decoder
- added a b-encoded map heap data structure
- added a table organisation based on b-encoded heaps
- added a servlet to maintain such tables (see Tables_p.html)
- integrated the servlet into the Advanced Settings menu
- added an api recording based on the new tables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6606 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
56e0d9bd01
- testings with image parser
...
- added image size as part of parsed text in images
- avoid unnecessary error messages if parsing of documents failed but one succeeded
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6597 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e80e060ca6
- increased thread priority for server threads
...
- decreased thread priority for crawler threads
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6596 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7d400b17d0
html parser support for .cfm files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6590 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago