orbiter
4c99d4683d
possible fix for lost crawl profile handles: clean-up job did wrong measurement to see if crawl is still running.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6465 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
cd6745b292
accept rss feeds without channel descriptions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6464 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
08f1cbb125
another update to the pdf parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6463 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
54c54fb144
get a handle for grep: 'StackTrace'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6462 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
605e896d6c
more details for exception catching when parsing pdfs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6461 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
18b21eaffe
small fixes to search default values and server logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6460 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
6edc168cfe
option to disable dht by memory limit:
...
memory.acceptDHT in kbytes
not yet pre-enabled, will clear on every startup
please review since this could break dht in freeworld
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4431b9767e
added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e3025ee691
- new icon for OAI-PMH loading action
...
- added many stack trace outputs for exceptions in crawl profile handler to find the 'missing profile handle' bug
- catched one more timeout exception in httpd file loader
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6457 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f0b8db93f0
- more abstraction of serverCore thread access
...
- no more keep-alive when number of connections exceeds 1/2 of the allowed number of connection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6456 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
19f31bb043
- moved OAI-PMH source list file from SETTINGS to DICTIONARIES/harvesting
...
- added convenience method for loading of files from the web in LoaderDispatcher
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6455 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2889b9426e
missing code for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6454 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b6a8887ff5
better handling of running sessions without explicit hashtable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6453 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1dc7ea986a
added a dynamic keep-alive time-out for http server sessions:
...
if there are many concurrent server sessions, the timout is decreased.
This should avoid a situation where the clean-up thread is too
late to stop running http sessions that should be terminated
before the maximum number of server sessions is reached.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6452 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
e77c906673
*) minor changes mainly in comments
...
*) added svn:keyword settings for several files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6451 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
f1740edbf8
*) added skript to change memory settings, password and port (experimental, don't blame me if it messes up your configuration)
...
*) minor change in Digest class, added option in main method, might not be optimal yet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6450 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
11f7da06ed
- fixes to csv parser
...
- automatic OAI-PMH import by just clicking on one link from the provided resource list
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6449 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9b6762ec2e
- added a csv "comma separated values" parser to parse OAI-PMH sources from
...
http://roar.eprints.org/index.php?action=csv
- integrated the csv parser into the crawlers parser list
- added an extension to the OAI-PMH import function to download and show the roar csv file using the csv parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6448 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
176e334aa4
fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6446 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2fa6bf440b
workflow update to OAI-PMH importer
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6445 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b0b7a4f9a5
- added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records
...
- added monitoring for retrieved records
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
350d13e153
very first working version of oai-pmh importer: if given the right url, the importer can read and index listRecord xml files and calculate the right resumptionURL which is then given as next default start point for the importer url input.
...
no automatic harvesting by now, this will be done later
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6443 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
58616d99e4
patch for yacy disk usage detection on lvm host
...
by Michael S.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6442 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
79251e6f60
configurable disk space hardlimit for dht
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6441 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a0e891c63d
- some redesign in UI menu structure to make room for new 'Content Integration' main menu containing import servlets for Wikimedia Dumps, phpbb3 forum imports and OAI-PMH imports
...
- extended the OAI-PMH test applet and integrated it into the menu. Does still not import OAI-PMH records, but shows that it is able to read and parse this data
- some redesign in ZURL storage: refactoring of access methods, better concurrency, less synchronization
- added a limitation to the LURL metadata database table cache to 20 million entries: this cache was until now not limited and only limited by the available RAM which may have caused a memory-leak-like behavior.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6440 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4240785f20
added anti-alias function for line drawing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6438 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
30f108f97d
added stub of oai-pmh importer (not working yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
77c99e500f
added more control over memory allocation
...
should avoid some of the OOMs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6436 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
52470d0de4
- fix for xls parser
...
- fix for image parser
- temporary integration of images as document types in the crawler and indexer for testing of the image parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6435 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5e8038ac4d
- refactoring of blacklists
...
- refactoring of event origin encoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6434 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
26fafd85a5
- more refactoring
...
- fixed problem with parsers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6433 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3528b970d6
- refactoring
...
- added new experimental (not-yet-working) image parser
- added new test image
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6431 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a8ce192f63
- shifted main classes to new package net.yacy
...
- fixed some bugs in last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6427 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b79f4f062f
refactoring of yacy documents and parsers: they depend now only on the kelondro classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6426 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
0fd9540866
Configuration of HTTPDProxyHandler logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6425 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
cee7a05ff2
- de-serialized the pdf parser
...
- added fail callback for file indexer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6415 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9db928ce53
replaced fontbox 0.7.3 with fontbox 0.8.0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6414 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c2272785c7
- fix for xlsx and pptx parsing
...
- less exception logging for swf parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6413 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c864901087
- moved httpd.mime to defaults path
...
- some documentation fixes
- adopted a default setting for the search window: moves css setting to base.css
- some enhancements for the DocumentIndex class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6410 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
8829ec5f18
*) made sure that is replaced with a space and not just deleted in CharacterCoding.java
...
*) added annotations and made minor changes to serverObjects.java
*) set subversion properties for several files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6409 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
6c347a37eb
more options for DocumentIndex
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6408 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
6192205533
more final modifier
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6407 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
0f6b011e1a
fix for new index location and better way to use own classes by reflection
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6406 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7a3bbd950f
:-(
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6405 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b953f04f90
one more reflection fix
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6404 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
77d6604856
fix for npe, see http://forum.yacy-websuche.de/viewtopic.php?p=17727#p17727
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6403 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2a7fe35f92
performance tuning using more final modifiers in the kelondro core
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6402 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
cb4de9ceee
fixed a bug in table iterator (did not recognize elements in write buffer)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6401 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e7f18ba24b
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6399 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ce8dc575ca
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6398 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bea3b99aff
moved table and util classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6397 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bd876eb4b7
moved io classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6396 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c0e0e1f422
moved blob classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6395 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1e4f8b56ed
accumulated classes from different packages into the new rwi package
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6394 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
194da25a2f
moved kelondro index
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6393 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4446acc8cd
moved kelondro order
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6392 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f677d534b1
start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root
...
- moved here the logging classes as part of the new net.yacy.kelondro package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6391 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ea473e32b8
refactoring
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6390 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
735e2737e3
* added index segments
...
This is a major change in the organization of indexes.
Please consider a back-up of your data before you run this update.
All existing index files will be moved and renamed to a new position.
With this change, it will be possible to maintain different indexes for different purposes and it will be possible to have a distinction between DHT-in and DHT-out specific indexes. Tenants may also have their own index, and it may be possible to have histories and back-ups of indexes. This is just the beginning, many servlets must be adopted after this change, but all functions that had been there should still work.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6389 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
09de5da74a
once again a performance hack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6388 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2f6d88403e
uä
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6387 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d2615ea5a8
increased memory for scraper buffer to enhance parsing speed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6386 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4bbbb74ec4
removed not necessary synchronization
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6385 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
67e5464cc2
Fix for SVN6380: x[] Arrays are unsuitable Keys for Maps without using a proper Comparator.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6384 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
aeab8c7917
Prevent failed DHT attemps from overwriting newer peer info
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6382 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
9324b5b6c5
Enhancements to DHT
...
- speed up deletion of containers when selscted from whole index
- correctly eliminate all references to unavailable URLs, not just the first encountered
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6381 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
e49e2d75fe
Limit the time Transmission.Chunks stay in the transmissionCloud by using a Map that iterates entires in insertion order.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6380 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
92db7c5d07
increased timeout for index retrieval
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6379 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
386b9f35f6
activated resource observer for windows 7
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6378 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
6e0dc39a7d
- some fixes to prevent blocking situations
...
- better logging for the crawler
- better default values for the crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6377 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
51f2bbf04b
possible fix for problem in http://forum.yacy-websuche.de/viewtopic.php?p=17655#p17655
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6376 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f8371707e5
- possibly better termination for SplitTable
...
- better abstraction in DidYouMean
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6375 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
87780f2562
produce did-you-mean also for queries with more than one word
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6374 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
04a548a1e3
- temporary integrated the transferURL servlet as static class instead as a class that is called using reflection to investigate the OOM problems in that class
...
- fixes for numerous other problems
- removed dead code
- resdesign of the strings-method, which produces now less memory overhead and may help to prevent OOMs
- another fix for the deadlock problem in SplitTable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6373 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ea427df944
fixed a worst case situation of the condenser which may cause a temporary full CPU load because of a bad data structure usage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6372 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3e38035389
fix for interrupted thread during has() property check
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6370 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5bd1c1d205
just added some comments that had been produced to learn about OAI-PMH
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6369 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
6aa474f529
- better logging for web cache access and fail reasons
...
- better Exception handling for web cache access
- distinction between access of web cache for proxy and crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6367 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3671c37989
added experimental oai-pmh reader and integrated it with the existing dublin core parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6366 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
58a00205d5
re-activated the emergency close when too many server connections exist
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6364 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c57d2070e6
more logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6363 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a995b95367
tried a fix for the httpd access bug (too many unclosed sessions)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6362 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e1fba41cad
better logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6361 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2275f885a8
possible fix for concurrency problem
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6360 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
a6a3090c3d
*) blacklist cleaner supports usage of regular expressions now
...
*) refacored BlacklistCleaner_p.java for better readability
*) moved check of validity of patterns to the Balcklist implementation since patterns might be valid in one implementation, but not in another
*) added method to check validity to Blacklist interface
*) fixed some minor issues like typos or wrong whitespaces
*) set subversion properties for a whole bunch of files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6359 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5a93807781
improved web cache speed:
...
- removed one computation out of a synchronization
- removed one not necessary has() call
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6358 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2e8b2867ff
double performance of store method because it avoids one 'has'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6357 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
afda5b1adc
new join method for indexes (not yet used)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6356 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
65b66c2c18
better handling of array files of length 0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6355 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1957b5797a
fix for seed generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6354 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
432154f725
new strategy for concurrent database index key retrieval
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6353 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a11cd9f80f
- removed reverse name lookup for http access logging (grr..)
...
- removed a synchronization in seed info string generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6351 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2e6bdce086
- added more logging to balancer
...
- changed balancer logic slightly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6350 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1171a72006
fix for deadlock as seen in http://forum.yacy-websuche.de/viewtopic.php?p=17521#p17521
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6343 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
031e6eefbd
some updates to dublin core, metadata browsing, file indexing and parser stability
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6342 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
62a7341c4d
Fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2204
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6341 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
f65bfaa9af
*) Removed base tag from errror page. This has been added by myself a long time ago as a workaround for some weird behavior of my router, but as it turns out, it does more bad than good in general: If HTTPS is used for communication with YaCy, entering a wrong passwort led to an errror page with a form which would send username and password unencrypted with the user possibly being unaware of this.
...
*) changed some comments, added some annotations, added SVN properties here and there
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6340 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e4797ebcde
fix for http://forum.yacy-websuche.de/viewtopic.php?p=17509#p17509
...
corrupted files are ignored
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6339 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
efa7fb34f0
better oom-awareness of miss-cache in cache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6338 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3e9dcfc204
fix for http://forum.yacy-websuche.de/viewtopic.php?p=17504#p17504
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6337 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c3a4aee255
some redesign with a possible fix for the ReferenceContainerCache.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6336 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
aca8a78eb8
fix for shutdown of DocumentIndex objects
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6333 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
23ab6fbca4
- navigation appear at correct position when opengeodb-results are also presented after a search
...
- show an about box if about.headline and about.body is set
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6332 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4db34eea73
fix for OOM problem in kelondro Cache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6331 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8ea1d7ab59
fix for wrong assert condition in search abstract generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6330 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fbd77bd77c
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6328 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
54c7cbf1d9
- fast result for local search in case that less than 10 hits exists
...
- small change in display of RAM in profiling
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6326 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
28d4b921b6
different approach for file search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6325 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f99f86c5c5
added concurrency to file indexing class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6324 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
902d16cf6c
fixes to parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6323 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4a1c852435
fix in usage of RAM copy for Table objects and some cosmetics in asserts.
...
This bug affected Tables in case that a removeOne() was called and a RAM copy of the table was active. It may happen for peer owners with a lot of RAM assigned to YaCy. The bug appeared especially during crawling when the balancer tried to get new entries from the crawl queue.
This bug may help to solve report at
http://forum.yacy-websuche.de/viewtopic.php?p=17417#p17417
and will be tracked there
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6322 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
dce450e2e0
possible fix for "hung" doc-documents
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6320 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e627f75415
one more fix to badwords and stopwords
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6316 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
721b88efbd
- fixed a problem loading blacklists with new yacycore.jar
...
- fixed badwords and stopwords initialization
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6315 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
80d5005044
fixed seed upload methods - replaced reflection with direct instantiation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6314 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
68465c37af
added a convenience class to add files into a YaCy index
...
to make this possible, the yacyURL must be able to process file:// urls, which has also been implemented
testing of the new class resulted in some bugfixes in other classes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6313 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2e41e10ffd
- updates to yacyVersion parser (remove old targets)
...
- added javadoc target to built script (does not work yet without errors)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6312 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
27d00285aa
- added a new file reader cache that may serve as full-file-copy of blob database files. This is not yet used
...
- removed class FileWriter and replaced all usage of that class with CachedFileWriter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6309 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fd6b9cb7dc
refactoring of IO access classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6308 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d64569aa39
reuturn only recommendations of words that have a greater count than the original word
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6307 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
604c37927f
used comparator for did-you-mean that uses index sizes for comparisment, but:
...
- limit comparisment to only the first 10 elements that had been sorted before without IO
- added a size cache to index computation because the size is computed at least twice in set comparator
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6306 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a58d9cae7d
- show location name in geolocalization search result
...
- added link from location icon to openstreetmap browser with coordinates
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6305 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
573d03c7d7
added configuration to enable ram table copy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6304 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3be54e1891
fix to rule when to use a ram table copy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6302 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
700218846c
disabled or removed sleep calls
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6301 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
342c5d0fd4
fixed city name detection: finds now also substrings of city names
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6300 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
18aa0609ca
fix for caching of word hash computation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6299 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a10a6cce45
patch for http://forum.yacy-websuche.de/viewtopic.php?p=17289#p17289
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6298 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
53bbdfd19a
*) setting SVN keywords
...
*) minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6297 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
25f6145934
*) preventing null pointer exception in case empty search word or only one character is enterd or all search words are removed by filters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6296 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
low012
248f3fd9b5
*) cleaned up code for better readability
...
*) added a few copyright notices
*) removed redundancy in constructors of ListToken
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6295 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
eaddf2d464
- corrected layout of map preview
...
- added caption to maps containing latitude and longitude information
- prevented that maps occur on second search page
- added location names to did-you-mean
- some refactoring of did-you-mean
- added equal and compareTo test to Coordinates class to make that work in set
- fixed utf-8 support for library files
- fixed a bug in images search icon view caption
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6294 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
4b83875abd
Small fixes for the heapCacheIterator in ReferenceContainerCache:
...
- Start the iteration at startWordHash
- When used with rotation, let the iteration stop when the cache is empty
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6293 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fd668f531b
fixed map layout
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6292 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2740d9dd79
added integration of osm maps for search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6291 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
af3a696fc4
added a fast-fail concept in search processes. The search now has better control if all the remote searches may bring any result. If all processes are finished, then all search tasks fail fast.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6290 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ce972ff4ef
update to default ranking profile which has now some settings to deny some phpbb3 pages which are redundant in the index when crawling phpbb3.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6288 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
44579fa06d
- fixed a problem loading images through yacy's document loader,
...
this denied non-parseable documents which excluded all images
- fixed url of osm tile server
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6287 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
67eddaec4b
changed way to integrate dictionary files:
...
the must be downloaded manually by the user and placed in DATA/DICTIONARIES/source
for each externally imported dictionary file there will be a translator that converts the input file once
into a YaCy-internat data format.
Files that will be provided together with yacy releases may still be placed in <root>/dictionaries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6286 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d656a94f55
fix for bad paths in dictionary processing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6285 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3b9aaf9e9f
- inserted new library tests inside DidYouMean
...
- some redesign of DidYouMean that was necessary to follow
a special rule how a library should be used:
- the library provides words that start or end with a test
word which may be possibly also an empty set of words
- all words that the DidYouMean produced with the four
production rules are used to generate a set of
library-completed words
- if this process results in any words from the library,
only library-genrated words are taken
- if the is no library-generated word at all, take the
artifial generated word
- all words that result from these rules are tested against
the index
- the result is ordered using a lightweight comparator that
prefers short words
- a not-so-much-io test against the index is beeing prepared
next
- insered the library initialization into the switchboard
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6284 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
8c35ffe34c
fixes to the dymlib
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6283 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bfa273bcc1
added a library provider which holds libraries in static objects,
...
which can be used by any other classes to support their functions.
libraries are designed in such a way that users can create and
insert their own library files, but can also be imported from
other sources. As an example the "Korpusbasierte Wortgrundformliste
DeReWo des Institut für Deutsche Sprache" from
http://www.ids-mannheim.de has been integrated. This dictionary
is licensed to be used for all non-profit purposes. In case that
YaCy is used for commercial uses, this library must be removed.
The new library provilder reads the original source and translates
it into a simple word list to be used for the did-you-mean library
provider. More libraries may be provided in the future using
a download-servlet which puts files from the internet into the
<application-root>/dictionaries/ path.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6282 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1762a7bcd6
- moved DidYouMean to the data package
...
- added a DidYouMeanLibrary class that shall support the did you mean function with additional word lists
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6281 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
bf8ed00e9e
removed debugging code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6280 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
ead48c4b25
fix for preparation of search result pages with offset > 10:
...
- less pages are fetched in advance
- just-in-time fetch of next required pages
- fix for missing hand-over of offset to fetch threads
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6279 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
39a311d608
better care to do not loose the merge/dump thread
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6278 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
10d3e856b5
better concurrency, less blocking & performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6277 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1a9cfd8718
some performance hacks (CPU only, not IO)
...
this will cause better computation speed for single- and multi-core;
there are enhancements that will speed up old and slow machines as well
as multi-core CPUs. Indexing of surrogates has been speed up
from 4000 PPM to over 20000 PPM on a simple dual core office computer.
Since the enhancements are mostly in core routines, the hack should also
speed up search performance.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6276 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
92407009b2
cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6275 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago