orbiter
4782d2c438
fix for search bug that appeared when looking at page 3 of results or further
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6515 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
29fde9ed49
better control of ranking order in sort stack
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6514 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
66923ebc6c
- modified method in RequestHeader that delivers the host name of requester: no more reverse domain lookup (may have killed interface performance in some cases)
...
- added logging output for shutdown servlet: show ip of requester of the shutdown
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6512 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e34e63a039
preset of proper HashMap dimensions: should prevent re-hashing and increase performance
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6511 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4a5100789f
replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f4946eaf27
- better thread dump
...
- suppressed one server exception
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6509 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9743b70d1c
disabled keep-alive of server, not really needed for speed but a cause for much trouble and memory occupancy
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6508 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
491ba6a1ba
- some refactoring in workflow
...
- some refactoring in search process
- fixed image search for json and rss output
- search navigation on bottom of search result page in cases where there are more than 6 results on page
- fixes for number of displayed documents
- disabled pseudostemming
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
969123385b
added json and rss output for image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6503 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
d183f8d980
refactoring (moved code from ContentTransformer to TemplateEngine)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6498 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
23aef43786
- better synchronization in SortStack
...
- better ThreadGroup organization
- less worker threads for media search (64 was too much...)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6497 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7b1f5b0430
- better media search ranking
...
- better concurrency with enhanced synchronization in sort stack
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6496 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4df88a4e7a
- fixes for missing or bad hashCode computation
...
- fixes for bad equals() methods that had not been used by hash maps and therefore some classes did not work as objects in hash maps.
- this may also affect some cases where double-checks should have been, but did not work.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6495 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
dbdf2570ba
added comparator and more fixes for SortStack/SortStore
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6494 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1dff620181
Better implementation of SortStack and SortStore and adoptions in all using classes to implement the necessary Comparable interface and hash code computation.
...
The better SortStack performance affects crawling and image search speed and quality.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6492 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
fe41a84330
some enhancements in web caching: avoid double loading of response metadata and/or content
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6491 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
06d0dcde20
more enhancements to image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6490 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4c6312d103
enhanced image search
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6489 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2d8f3ee301
some performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6488 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
94b2a664f3
- use a static DiskFileItemFactory (one instantiation is enough)
...
- use more memory for the DiskFileItemFactory to avoid IO when POST commands come
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6485 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
013f337d3f
- avoid unnecessary host name lookups for localhost
...
- avoid unnecessary reverse domain name lookups for remote access
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6481 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
20c5d78a5c
fix for a ConcurrentModificationException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6478 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
7144d2df6e
added crawlReceipt servlet as individual class to examine OOM problem as documented in
...
http://forum.yacy-websuche.de/viewtopic.php?p=18120#p18120
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6476 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
29fe436e36
- fixed post-ranking including prefer mask
...
- enhanced a core database access method / less wasted ram
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6473 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5399d1e2bc
refactoring (reason: get more abstraction to use the blacklist class; for integration in other servlets)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6471 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1fa0ac26e9
better protection against NPEs during search/ranking
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6467 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4c99d4683d
possible fix for lost crawl profile handles: clean-up job did wrong measurement to see if crawl is still running.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6465 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
18b21eaffe
small fixes to search default values and server logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6460 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
6edc168cfe
option to disable dht by memory limit:
...
memory.acceptDHT in kbytes
not yet pre-enabled, will clear on every startup
please review since this could break dht in freeworld
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
4431b9767e
added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
e3025ee691
- new icon for OAI-PMH loading action
...
- added many stack trace outputs for exceptions in crawl profile handler to find the 'missing profile handle' bug
- catched one more timeout exception in httpd file loader
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6457 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
f0b8db93f0
- more abstraction of serverCore thread access
...
- no more keep-alive when number of connections exceeds 1/2 of the allowed number of connection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6456 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
2889b9426e
missing code for last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6454 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b6a8887ff5
better handling of running sessions without explicit hashtable
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6453 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
1dc7ea986a
added a dynamic keep-alive time-out for http server sessions:
...
if there are many concurrent server sessions, the timout is decreased.
This should avoid a situation where the clean-up thread is too
late to stop running http sessions that should be terminated
before the maximum number of server sessions is reached.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6452 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b0b7a4f9a5
- added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records
...
- added monitoring for retrieved records
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
lotus
79251e6f60
configurable disk space hardlimit for dht
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6441 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a0e891c63d
- some redesign in UI menu structure to make room for new 'Content Integration' main menu containing import servlets for Wikimedia Dumps, phpbb3 forum imports and OAI-PMH imports
...
- extended the OAI-PMH test applet and integrated it into the menu. Does still not import OAI-PMH records, but shows that it is able to read and parse this data
- some redesign in ZURL storage: refactoring of access methods, better concurrency, less synchronization
- added a limitation to the LURL metadata database table cache to 20 million entries: this cache was until now not limited and only limited by the available RAM which may have caused a memory-leak-like behavior.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6440 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
30f108f97d
added stub of oai-pmh importer (not working yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
77c99e500f
added more control over memory allocation
...
should avoid some of the OOMs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6436 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
52470d0de4
- fix for xls parser
...
- fix for image parser
- temporary integration of images as document types in the crawler and indexer for testing of the image parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6435 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
5e8038ac4d
- refactoring of blacklists
...
- refactoring of event origin encoding
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6434 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
26fafd85a5
- more refactoring
...
- fixed problem with parsers
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6433 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
3528b970d6
- refactoring
...
- added new experimental (not-yet-working) image parser
- added new test image
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6431 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
a8ce192f63
- shifted main classes to new package net.yacy
...
- fixed some bugs in last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6427 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
b79f4f062f
refactoring of yacy documents and parsers: they depend now only on the kelondro classes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6426 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
hermens
0fd9540866
Configuration of HTTPDProxyHandler logging
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6425 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
cee7a05ff2
- de-serialized the pdf parser
...
- added fail callback for file indexer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6415 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
9db928ce53
replaced fontbox 0.7.3 with fontbox 0.8.0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6414 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago
orbiter
c2272785c7
- fix for xlsx and pptx parsing
...
- less exception logging for swf parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6413 6c8d7289-2bf4-0310-a012-ef5d649a1542
15 years ago