sixcooler
29c2289b5c
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
sixcooler
605bc4c10e
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
orbiter
aa322bc6d0
fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8050 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
97d1347adb
added also a default accept field to robots.txt downloads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8049 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f183d3822c
added a default accept header in http requests since some http fraud detection functions check that this header field exist
...
see also: http://bad-behavior.ioerror.us/ in source file browser.inc.php
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8048 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
06352b8d6b
more logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8047 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a99934226e
more logging for debugging of robots.txt
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8046 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7a5841e061
fix for robot parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8045 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
458c20ff72
fix for robot parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8044 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
e7dedc56f2
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
orbiter
017a01714d
- enhanced logging in robots.txt parser for remote debugging
...
- robots.txt is now more robust against database operations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
7545822db5
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
apfelmaennchen
a8dfe787ed
- updated to jquery flexigrid 1.1
...
- YMarks.html automatically recognizes if a bookmark is a crawl start
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8040 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
0aa5e134ea
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
orbiter
eb1c7c041d
write info about robots.txt evaluation into getpageinfo_p.xml
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
apfelmaennchen
abba31f02e
- bugfix for correctly sorting ymarks
...
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
3b70ff7046
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
orbiter
3a15e58e28
- increased stability when opening the robots table
...
- increased stability when deleting tables
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8034 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
775b44017e
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8033 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
c99a4c0920
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
orbiter
e914a30099
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8032 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
b92c6bf897
Trying ImageIO instead of awt-Toolkit for parsing
14 years ago
sixcooler
db5ef90b0f
Merge branch 'master' of https://git.gitorious.org/yacy/rc1.git
14 years ago
sixcooler
3ef9f301ba
some customize on Memory-Performance-Graph
14 years ago
sixcooler
8f25070460
weekly rewrite of blobs
14 years ago
sixcooler
d6c1ab4e0f
some more unreserved characters
14 years ago
sixcooler
f522f61af0
clean offline copy of URL Tables
14 years ago
sixcooler
54193457bc
cutom keep alive strategy
14 years ago
sixcooler
249a78ff2a
G1 Memory Strategy - not used now
14 years ago
sixcooler
ccf1583188
cutom keep alive strategy
14 years ago
sixcooler
f280e339a8
no force on Memory Request for these parser
14 years ago
apfelmaennchen
5f7dbe1c42
- some refactoring (ymarks)
...
- improvement for autotagger (is now able to create/detect multi word tags e.g. 'open source')
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
78ce3b13be
typo
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8027 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
85d6bf4ac4
fixed urls to media content during indexing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8021 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0d858d48ec
replaced String with StringBuilder in suggestion process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3a807e10cf
- added a cache for active crawl profiles to the crawl switchboard
...
- moved the domain cache for domain counter from the crawl switchboard to the crawl profiles. the crawl domain counter is now therefore relative for each crawl start, not for the whole crawler.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8018 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
37e35f2741
normalization of url using urlencoding/decoding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8017 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e58438c01c
- added a new retry connector for solr (for cases where solr responses are slow)
...
- added a new exist property into the metadataRepository which includes solr entries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8016 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
d8d9735b4f
stability bugfix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8012 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c31564ef08
stability bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8011 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f121f4bb45
fix for link in Supporter and Suftipps page
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8010 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
94eab08794
- updated opensearchdescription text and icon
...
- removed automatic setting of maxitems during search (can be set now elsewhere)
- updated RSSMessage.java
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8009 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
279482a76d
fix for npe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8007 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
1b86d06d1e
fix for http://bugs.yacy.net/view.php?id=62
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8004 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
9e4875230f
performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8001 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
eb9c9edb01
enhanced table method (used by almost all yacy api interfaces)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8000 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4ad9fc2bff
new snippet strategy for search hits in metadata: show beginning of text instead of hit position
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7999 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a9838f8b99
fix for http://bugs.yacy.net/view.php?id=59
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7997 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
hermens
d3df03838a
make sure myself-target is always inserted at its appropriate position
...
this was previously omitted if the own peer should have been the first target
or the peer was the last peer before the rotation to AAAAAAAAAAAA
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7996 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
hermens
c3e7efa846
added sender side prevention of rwi flooding as mentioned in SVN 7993
...
saves memory and speeds up enqueueContainers by limiting the size of transfer.Chunk
saves network bandwidth by not transmitting RWIs that would get discarded at the target anyway
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7995 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago