orbiter
dec24244cf
added convenience class to generate UTF StringBody objects with a default UTF8 charset.
...
Reason: if this is not used in StringBody-Class initialization, a default charset name is parsed.
This is a synchronized process and all classes using default charsets synchronize at that point
Synchronization is omitted if this class is used
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7530 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
1110d16af9
performance hack: replaced generic row.getColBytes() call with row.getPrimaryKeyBytes() where the column is 0
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7529 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
19b2a50578
- enhanced date formatter cache
...
- added more instances of formatter objects to different classes to make them independent in case of lockings that may applay during synchronization of the date formatter object (date formatting is not thread-safe and must be synchronized therefore)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7528 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
48a61c39a3
speed hacks in BLOB ArrayStack:
...
- more concurrency if possible
- less threads if no concurrency necessary
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7527 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a92d80a545
performance enhancements using an alternative to a insensitive collator (a complex string compare):
...
- less synchronizations
- better speed
..at most important and commonly used classes: http headers, url parsing and html parsing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7526 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f2e8ffd768
enhancement in synchronisation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7525 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
bcea497644
next try to fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3193&start=0&sid=b98aa9a7466397602b436eb45f4a9d39
...
tested proxy, crawl, updatedownload - please do further testing!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7524 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
ad7fcb9d61
Enhanced Base64Order transformation: less overhead (transformation between StringBuilder and byte[])
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7523 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
f95e50ec3d
more explanation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7522 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
bb36bf841a
emergency commit (sorry sixcooler for not waiting) because without that automatic updating peers would not be able to do the next update.
...
Please see http://forum.yacy-websuche.de/viewtopic.php?p=22059#p22059
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7521 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
8ad4e10491
fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3193&start=0&sid=b98aa9a7466397602b436eb45f4a9d39
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7520 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0ce17d823a
- fixed bug in ordering
...
- fixed ConcurrentModificationException in set join
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7519 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
dec4f36700
- fix for missing favicons in search widgets
...
- fix for bad digest/hash computation in case of interrupts to class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7518 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
804ae2275b
- do not delete idx and gap files if the heap is not modified
...
this change may have bugs in it which may cause damage to your existing data. please use with care.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7516 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e3ef4e3021
- increased default peer ping time from 2 minutes to 1 minute
...
- filtering out too old peers when reading seed lists (limit is now 240 minutes)
- added concurrent host names resolving in front of the http client because the http client uses the java built-in DNS resolve which is not multithreading-safe (i have seen deadlocks in thread dumps showing that this bug in jdk is still there)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7515 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5e45ded8e2
- removed locks from WordReference
...
- refactoring of HeapReader/Writer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7514 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
cd19d0517e
added dns resolve to HTTPClient POST using a dns cache to prevent that that not-thread-safe built-in dns cache inside apache http client is used
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7513 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
d28f8040e0
removed unnecessary recording function that caused also a performance problem after serving too much files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7512 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
af87af0d4c
- removed synchronization in serverSwitch which should improve speed
...
- fixed wrong assert in network graph
- enhanced double check method in table class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7511 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4bd65532da
initialization of libraries concurrently (faster start-up)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7510 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
57e6728cb7
- removed usage of /etc/alternatives/www-browser because of problems with lynx, see:
...
http://forum.yacy-websuche.de/viewtopic.php?p=21959#p21959
please look if the browser that is linked with /etc/alternatives/www-browser can be detected and insert call again if
it can be made sure that this does not call lynx
- replaced severe warnings with just warnings in yacyClient
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7506 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
91eeaf2cff
fix in ftp client
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7505 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
e717bf74ba
more logging, more care about OOMs
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7503 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
d84b4a072e
healing for some OOM problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7502 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4aa406fb0f
added log output to find bug in url parser for short hosts
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7501 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
82f262f685
- enhanced circle drawing speed
...
- beautified 'moving dot' feature (using smaller and correctly positioned dots)
- added moving dots to DHT transfer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7500 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
29dc416ac6
more animations in graphics. See network and access picture.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7498 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
93b9c4fbc9
added missing file for latest commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7497 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
a80ee9a03d
THE GRID is coming to YaCy .. see new animated graphics on http://localhost:8090/AccessGrid_p.html
...
showing incoming and outgoing connections in an animated way
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7496 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
ce012e11aa
*) deleted LogStatistics since the page did not work anymore and it seemed to be obsolete, tell me if you miss it and I will add it again
...
*) a few minor changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7494 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
c5051c4020
*) fixed bug which caused entries to not be deleted when deleting by URL on IndexCreateWWWLocalQueue_p.html (I hope this did not break anything else)
...
*) cleaned up code a little bit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7493 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
d58071947a
maybe terminateOldSessions is too slow, removed sleep
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7492 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
3e380c51b6
update to browser start with linux
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7486 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6083f2f171
fix for (false) oom
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7484 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
b35fda43ea
more changes to headless mode; now non-headless mode is used when:
...
- YaCy runs on Windows
- YaCy is started with the -gui option
in all other cases YaCy runs in headless mode
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7481 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
6c52e31993
new methods to open a browser
...
- if YaCy is started with the option -gui, it is not in headless mode. Then the java 1.6 browse method is used if all other methods fail
- in linux, the path /etc/alternatives/www-browser is used if no firefox is installed
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7480 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5892fff51f
introduction of dht-burst modes: this can expand the number of target peers in some cases where a better heuristic is needed. The problematic cases are either when a muti-word search is made (still a hard case for our term-oriented DHT) or when a network operator wants that all robinson peers are asked. We therefore introduced two new network steering values that switch on more peers during the peer selection. Because the number of peers can now be very large, the number of maximum httpc connections was also increased.
...
Please see new coments in yacy.network.freeworld.unit for details of the new DHT selection methods.
The number of maximum peers is now not fixed to a specific number but may increase with
- the partition exponent
- the number of redundant peers
- the robinson burst percentage
- the multiword burst percentage
The maximum can then be the number of senior peers (all visible peers).
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7479 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
4588b5a291
- fixed document number limitation for crawls that restrict the number of documents per domain
...
- some restructuring of the document counting and logging structures was necessary
- better abstraction of CrawlProfiles
- added deletion of logs to the index deletion option (if the index is deleted using the servlets) which is necessary to reset the domain counters for the page limitation
- more refactoring to get the LibraryProvider more clean
- some refactoring of the Condenser class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7478 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
64f32e8f00
*) replaced all IPs in IP filters for proxy with the proper regular expression
...
*) some cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7477 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
sixcooler
3e8b72be50
update to httpclient-4.1 - sorry forgot some
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7474 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
low012
74b22dfa24
*) fixed bug which affected blacklist entries which consisted of domain _and_ path parts
...
*) minor cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7471 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
93732d6773
increased number of target peers
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7468 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
70ca7cec8c
fix for http://forum.yacy-websuche.de/viewtopic.php?p=21763#p21763
...
and another fix for non-working global search when search options are switched off
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7467 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
fe93caac5a
added flags and administration options to show advanced search and to show search result attributes (for each search result)
...
Administration can be done at ConfigPortal.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7466 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
431f780f41
patch for bad data in url metadata
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7464 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
5905f912c5
replaced more double types with float
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7462 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
0cdfb82963
replaced more appearance of double values by float values
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7461 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
eb12e15738
moved all Double values to Float values because of
...
http://www.exploringbinary.com/java-hangs-when-converting-2-2250738585072012e-308/
YaCy does not really need double-precision floating point computation anywhere, so this should not affect any feature
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7460 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
f1ori
982aa689ef
* fix StringIndexOutOfBoundException in WebStructureGraph
...
* add better escaping to saveMap and loadMap
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7458 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago
orbiter
88773e4daa
changed the default port from 8080 to 8090
...
see also: http://forum.yacy-websuche.de/viewtopic.php?p=21683#p21683
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7454 6c8d7289-2bf4-0310-a012-ef5d649a1542
14 years ago