orbiter
be4c458951
refactoring (implemented Iterable in kelondroRowCollection)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5432 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
bb5c2cd12e
*) ISINDEX parameters will not be put on commandline anymore to prevent possible security hazards (better safe than sorry). Parmeters will have to be read from QUERY_STRING in ISINDEX case too which does not seem to be uncommon behaviour for web servers: http://vms.pdv-systeme.de/users/martinv/cgi_basics/cgi_basics.html#Datenuebergabe
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5431 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
b6bba18c37
replaced the storing procedure for the index ram cache with a method that generates BLOBHeap-compatible dumps
...
this is a migration step to support a new method to store the web index, which will also based on the same data structure. made also a lot of refactoring for a better structuring of the BLOBHeap class.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5430 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
db1cfae3e7
*) cleaning up after myself
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5429 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
f547f9a78c
*) added CGI capabilities (run Perl scripts and other software via HTTP GET and POST)
...
*) set cgi.allow to true in yacy.conf to enable CGI (CGI is disabled by default)
*) edit cgi.suffixes in yacy.conf if necessary to use additional script types
ATTENTION: This is a rather experimental feature, not all environment variables are set yet.
Only enable CGI if you know what you are doing. Poorly implemented CGI scripts can put a system's integrity at risk!
Implementation of more environment variables and documentation due for the next days.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5428 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
bdc380cd84
* add lastModified to templateCache
...
-> no outdated files from cache anymore...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5427 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
6792c2a07d
* change mime type of xml documents from application/xml to text/xml
...
-> for easier Javascript requests
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5426 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
cb1e887027
* move svnRevNr classes to libbuild
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5425 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
025094675f
* remove empty directory
...
* add necessary dependency for pdfParser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5424 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
c5691180cb
* skip style-tags in HTML-files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5423 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
9d5d30f877
*) http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1672
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5422 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5448aad328
removed unused code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5421 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
3567c58b18
added another filed information for BLOBHeap dumps: the gaps
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5420 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
abdd4aa414
added a index dump for blob heaps:
...
this will increase the shutdown time for at most some seconds, but will speed up the start-up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5419 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
28d2d28573
added support for filetype search
...
(just use filetype:<type> in the search query)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5418 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8c3205b62e
fix for OOB Exception
...
see http://forum.yacy-websuche.de/viewtopic.php?p=11598#p11598
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5417 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
78c568331e
added test channel to /xml/feed.rss
...
can be obtained with
http://localhost:8080/xml/feed.rss?set=TEST
returns always a single feed entry with a fresh date
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e004da48d3
- added fast fingerprint computation for files (any). Will be used in new index dump method
...
- refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
eab72424df
*) Fixed small bug: When adding new elements to blacklist via import, the blacklist which the elements were added to was supposed to be displayed, which did not work correctly.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5414 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
0e56675596
*) cleaning up ;-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5413 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
cf69557ea2
*) blacklists can be exported as XML or plain text now
...
*) blacklist import via file upload works now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5412 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
1594a15be9
*) explicit mentioning of blacklist in blacklist cleaner
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5411 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
2d2ce24011
* remove all encoding-stuff from proxy
...
encoding is handled by parsers or browser, proxy only passes through
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
73c8a0839c
* abort download, when proxy connection is closed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5409 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
bb935fdbb0
less organization overhead for DNS caching and prefetching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
4907697cfa
* make fileuploads through proxy bigger than 65500 bytes possible
...
* remove gzip-encoding for files from cache
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
fc8189f3fb
better self-healing of corrupted databases
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
963da8c3f9
* updated tm-extractors to new version 1.0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
51f1a1927c
* remove saaj.jar and axis.jar and references to it (was for soap-stuff?)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5404 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
low012
5a89266598
*) new parameters for future use (better blacklist handling for im- and export)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5403 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e34ac22fbd
- added new monitoring servlet at
...
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
449e697436
fix for null-seed in seedfile
...
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
d376d81fc4
replaced busy thread control of crawl stacker by blocking threads
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
f29b48d9ff
patch for IndexOutOfBoundsException
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
f1ori
0881190b19
* Robots.txt: don't interpret Crawl-Delays for other robots
...
fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
243e73f53b
removed unnecessary usage of kelondroBLOBTree
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
8cb7170b75
- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated
...
- removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7535fd7447
- refactoring of CrawlEntry and CrawlStacker
...
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
6569cbbec1
npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
...
(break to avoid bad side effects)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5394 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
18513e2ee2
npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
2802138787
- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)
...
- fix of shallBeOwnWord target computation heuristic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
b1e211b258
no error-alert: http://forum.yacy-websuche.de/viewtopic.php?t=1639
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5391 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
13cb0916ee
changes to statistics and content of thread dump servlet
...
(points now more directly to performance leaks without mentioning class calls inside of sun/java calls that cannot be changed anyway)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5390 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
db6b3bf5a3
speed enhancement for integrated http server:
...
- tuning hacks in template engine
- bypassing the template engine if no servlet present
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
7cd08bd5fb
fix for NPE in BLOBCompressor
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
5b94498643
fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
1779c3c507
- added a read cache to the RAFile interface to RandomAccessFile
...
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
orbiter
e1acdb952c
fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
2c682d649b
- no stop shortcut (-> stop via tray)
...
- store registry keys on current profile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5384 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago
lotus
e918d64c23
show hand-cursor an labels
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5383 6c8d7289-2bf4-0310-a012-ef5d649a1542
16 years ago