borg-0300
245cc34d51
small fix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1969 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
e2853f357d
added more lowercase to url normal form generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1968 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
59d52fb4a9
fixed some problems with crawl profiles
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1967 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4a5b5515b5
added a lowercase to url normal form generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1966 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
44d72f06c4
more Caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1965 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
918445a2f4
Bugfix for last commit.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1964 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
708cc6c8d9
fixed some bugs for auto-filter and added monitor in profile list
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1959 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
c58789177f
bookmarkCache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1956 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
250864406f
...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1955 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
rramthun
42b0b10a95
-Adding Windows Media to types which are not sended compressed
...
-Renaming writeandzip to writeandgzip to avoid confusion about type of compression
-Adding new startup message to windows script
-The usual language "enhancements" ;-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1953 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
2ed4fa96b7
tagCache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1952 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
allo
330eb9c74f
bookmarkDB cleanup
...
(preparation for tagCache)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1951 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
e82899ba57
fixed missing urls map initializer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1950 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
63f39ac7b5
added 3 new crawling steering options:
...
- re-crawl by age of page (enter in minutes)
- auto-domain-filter
- maximum number of pages per domain
NOT YET TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1949 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1fc3b34be6
some pre-work (without function yet) to implement:
...
- re-crawl (by age of last crawl)
- auto-crawl-filter by crawl depth (to be explained..)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1948 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
theli
c9e6b5e391
*) check size of indexing-queue and crawler pool before processing remote triggered crawl jobs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1946 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1509314ea6
set tighter control during DHT index and peer selection
...
see http://www.yacy-forum.de/viewtopic.php?p=19329#19329
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1945 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
fcc0683200
*) undoing last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1944 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
9411961eec
*) another little fix for DHT-Transfer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1943 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
77f3237de3
adapted for isListed()
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1942 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hydrox
8b14a0c833
*) little fix for DHT-Transfer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1941 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
1f4412a146
adopted isListed to discussed new behavior as discussed (url, getFile)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1940 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
59fc55ea1e
added checks to protect peers from wrong seeds
...
see also: http://www.yacy-forum.de/viewtopic.php?p=19249#19249
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1939 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
063ef4660a
bug?
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1936 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
82358677a9
added another shiftK2W to flushCacheSome
...
this should fix the bug that the DHT cache is not flushed if there is no indexing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1935 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
128e4ab199
- in serverSystem: maxPathLength is now a variable, not a method
...
- upon startup the calculated maximum path length is shown
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1932 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
a37b09e303
implemented automatic adoption of chunk-read-ahead in kelondroTree to needed chunk size
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1931 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
30e3e3a0fd
adopted MAXPATHLENGTH to host system capabilities
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1930 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
d808765087
something Javadoc
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1929 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
85bb8e32a1
Bugfix for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1928 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
3fe402069f
try to fix
...
see: http://www.yacy-forum.de/viewtopic.php?p=19175#19175
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1927 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f16f1f15cd
bugfix for 100% CPU bug; thanks to Matthias for analysis
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1926 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
254a13efd9
MAXPATHLENGTH used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1925 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
8865948e4e
Cleanup;
...
Methode replaceRegex added;
Constant MAXPATHLENGTH added;
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1923 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
6c70f4a0cf
renamed wordHashes for a word hash set generation to wordHashSet
...
This was done because the wordHashes iterator will get another integer
parameter and then conflicts with the wordHashes set generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1921 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
d5f8f40c31
removed correcting iterator
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1920 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
526407f32e
adoptions, fixes for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1919 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
488a0ed580
replaced old keyIterator and rowIterator by buffered iterators
...
that are synchronized with database access
Main change is done in kelondroTree, other classes are only adoptions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1918 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
4e9a8f41fd
rwiDBCleaner + dbImporter: Iterate over small excerpts of
...
word hashes instead of the whole DB especially while changing
the DB in the process.
see http://www.yacy-forum.de/viewtopic.php?p=19136#19136
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1917 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
hermens
474379ae63
remove TABs from plasmaDbImporter.java
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1916 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
dba02f399f
starting of re-design of kelondroTree iterator
...
- new access to iterator
- added many IOException handling in other Classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1914 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
f02b426073
made kelondroTree.nodeIterator private
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1910 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
borg-0300
5f6fdf1786
Bugfix for getCachePath(URL url)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1909 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
303b6463a8
added debug line to URL storage for testing
...
see http://www.yacy-forum.de/viewtopic.php?p=19129#19129
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1908 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
431a4f3609
eliminated correcting iterator in kelondroTree
...
VERY EXPERIMENTAL! NOT TESTED!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1907 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
91dca2cd8d
fixed a bug in last commit: LURL entries cannot be written,
...
because a stored property was not set to false (but true)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1906 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
3286b1f498
re-organisation of lurl-creation and -stacking
...
this was necessary to prevent useless write to the database
in case of blacklist appearance of the url
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1905 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
9cca36a107
no more strict comparator checking in map exclude method if not needed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1901 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
0b903c5317
removed usage of kelondroNaturalOrder from plasmaCondenser to experimental
...
exclude cause of a 100% bug.
see http://www.yacy-forum.de/viewtopic.php?p=19076#19076
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1900 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago
orbiter
4239db0d1c
fixed new ordering for backup iterator TreeSet
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1899 6c8d7289-2bf4-0310-a012-ef5d649a1542
19 years ago