orbiter
c17d102bd8
enhanced speed for OrderedScoreMap inc method and size comparisment in concurrent environments
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7653 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b788182954
some enhancements to scoring speed
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7652 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
01690eab86
fix for mediawiki importer and wikicode parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7651 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4c013d9088
more UTF8 getBytes() performance hacks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7649 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
cominch
9ac02caf00
different initialization of empty variables in alternative constructor. This leads to wrong interpretation of user credentials, resulting in unnecessary "@" in front of host, and different urlhash values.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7646 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
57ce1fb491
reverted synchronization from SVN 7641
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7643 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
17530ca7b5
fix for bug http://bugs.yacy.net/view.php?id=10
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7642 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7c8e764201
removed synchronization again...
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7641 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
96c32e87b0
fixes to crawler and new user-agent crawl-delay handling
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7640 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
cb6f709a16
- enhancements in surrogate reading
...
- better display of map in location search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7636 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
1ff9947f91
*) added new user right: extended search right (allows to define users who can query more results than anonymous users)
...
*) cleaned up code a little bit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7635 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
564184909a
enhanced the surrogate parser: better reading of UTF-8 characters
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7634 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
156cf02703
- added an index constraint 'has location' to the condenser
...
- added evaluation of the 'has location' constraint to search using the /location operator
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7633 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
41b8d7f655
fix for url normalization (no backpath resolving in post parameters)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7632 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0430a94eaa
the location search shows now not re-evaluated locations but only such locations that are attached as metadata to web pages
...
- added parser for in-text appearing geo-locations
- added geo-locations to rss search result
- added evaluation of metadata-attached geo-locations in yacysearch_location to show search results within a map
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7631 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
8412f8787d
fix for http://bugs.yacy.net/view.php?id=8
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7630 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
9b25d07295
- added geo information parsing to html parser
...
- extended metadata information in index with geolocalisation
- added display of location in yacydoc and ViewFile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7629 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
cbf87fe72f
write PID to yacy.running
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7627 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b1a8d0c020
enhancements to web cache and less strict caching rules
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7620 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f3baaca920
- enhancements to DNS IP caching and crawler speed
...
- bugfixes (NPEs)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7619 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
df71776929
* fix bug #7
...
* log requires poison to finish, so Base64Order main-function doesn't finish, when called from debian configure script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7616 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
78d4c45d09
enhancement during search process: fast fail of search in case that all index feeder have terminated.
...
This change should affect filtering and navigators and should cause that search navigation gets faster
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7614 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a50f28e6e7
- fixed missing save operation for peer name change
...
- fixed import of mediawiki dump files
- added script to add mediawiki dump files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7609 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
2b5f8585bf
performance hack for Balancer and ip address parsing
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7608 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b1d133b69f
another anhancement to the ThreadDump function: better multiple dumps and filtering out of not interesting dump parts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7606 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a35d513bd8
fix for not-deleted .gap and .idx files
...
see also: http://forum.yacy-websuche.de/viewtopic.php?p=22128#p22128
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7605 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a6935e7dc8
fix for active dns resolving: do not resolve in case that the dns server is not available (offline mode)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7604 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
859c99886c
fix for multiple thread dump
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7603 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
61acf55da4
avoided using a synchronized(this) for the hash computation to prevent that the lock on the object is (accidently) stolen by another thread and replaced this synchronization using the protocol object. Made also the protocol object final.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7602 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
c2a968c23f
fix for bug in formatting in ThreadDump
...
and added hint for linux/Mac users that they may use the LOCKED feature using the start option -l
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7601 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
078ecacf61
avoid synchronization in DigestURI hash requests
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7599 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
1989ebc24b
removed more warnings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7598 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0324de1467
removed debug line
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7597 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
1aba7869bf
patch for Windows: do not use the thread lock feature from previous commit if used on Windows
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7596 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0a11727374
added new feature for Thread dump:
...
"THREADS WITH STATES: LOCK FOR OTHERS"
will show only such threads that lock other threads. This is the 'opposite part' of the blocked threads.
Because that this uses a thread dump that is produced with a kill -3 on the PID of the process and such thread dumps are written by the Java core outside of System.out and Sytem.err it is necessary to read the dump from a log in the file system. Such a log is only written if YaCy is started with startYACY.sh on a linux system. That means:
this feature is only available on linux and Mac OS X if YaCy is started with ./startYACY.sh -l
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7595 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b62b79675b
removed type cast warnings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7594 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a07a1a8b1e
removed type cast warnings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7593 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
8edaccfedf
removed unused variables
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7592 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e6c3507b17
disabled some of the previous changes (did not work in openjdk)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7591 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f9e5c21083
update to thread dump logs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7590 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
8f11d3a5bb
redesigned the ScoreMap classes:
...
- new concurrent score map using atom operation from java concurrency classes
- redesigned difference beween StaticScore and Dynamic Score into ScoreMap and ReversibleScoreMap allowed that many classes can now use simple ScoreMap Objects which can be used better in concurrent environments using the ConcurrentScoreMap
- switched from DynamicScore to ConcurrentScoreMap usage wherever possible
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7586 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a564230c48
more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7585 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
dc0db3550e
avoid string conversion
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7584 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
694fa3a2a5
- replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion
...
- changed menu structure slightly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7583 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
30aed9824a
moved getBytes() to UTF8.getBytes() to use a default String encoding
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7580 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
lotus
cb6d307bba
adding extension for parser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7579 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3820525464
more memory protection: auto-flush of caches in case of memory shortage
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7575 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
7962d35425
- removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons:
...
1) if the file is changed for a re-crawl this is not reflected in the steering because it would take the previously uploaded crawl start file
2) browsers do not submit the full path of the selected file even if this path is shown in the input field because of security reasons. There is no work-around or hack to make the submission of the full path possible
- fixed deletion of crawl start point urls in crawl stack and balancer double-check
- fixed a problem with steering self-call (no resolving of localhost)
- added more logging for the crawler to supervise why crawl urls are not taken by the loader
- added a javascript onload-function to select domain restriction in all cases where a crawl is started from a file or from a url
- fixed the restrict-to-domain pattern computation, added a 'www.'-prefix and added this functionality also to a crawl start from file
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7574 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
96bb33ed9b
added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7573 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e1b6916423
always try to guess the size of a StringBuilder to prevent too many memory re-allocations
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7572 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago