Commit Graph

5951 Commits (354f0d9acdcdaba63cf499c6481a80586b4f9930)

Author SHA1 Message Date
orbiter 1f36bf4dae enhanced assortment capacity; added extended WORDS migration
20 years ago
allo e6c381a2e2 Translationfile Division for different files
20 years ago
rramthun 0f11399d16 Some corrections...
20 years ago
orbiter 9f505af7aa preparations for bulk remote crawls
20 years ago
orbiter 0f663bcebf added global ppm computation
20 years ago
orbiter 9c72b4cdec replaced index dump stack by an dump array and limited url number in assortment ram (prevents too much RAM occupation)
20 years ago
orbiter 51962d55bf added 'PPM', page-per-minute statistics
20 years ago
orbiter 159f795f65 bugfix (null pointer exception in assortments)
20 years ago
orbiter 1d2155675b changed assortment memory cache flush
20 years ago
orbiter 19dbed7cc8 code clean-up
20 years ago
orbiter 00f63ea00d fail-save patch for pattern matching
20 years ago
orbiter 0a6be961ea added pattern organization
20 years ago
orbiter 40036ba69c fixed dht transmission; added url-blacklist blocking also for remote search
20 years ago
orbiter 311e627363 blocking of blacklisted urls in indexReceive and small changes
20 years ago
orbiter 2f0d7ea8d3 removed htcache stati (superfluous now)
20 years ago
orbiter 277048501e bugfix
20 years ago
orbiter 8b89c46afe fixed problem with cache write
20 years ago
orbiter 455ae9f55f fixed htcache-store problem and due-time for remote crawls
20 years ago
theli 55d10b864c *) further improvements in shutdown behaviour
20 years ago
theli 96386d6fc6 *) further improvements in shutdown behaviour
20 years ago
theli 0b95c9c434 *) Bugfix for Thread.getID() usagage + PeerPing-Shutdown Deadlock
20 years ago
theli 13eeaa08f3 *) httpc.java:
20 years ago
theli f57b60cd60 *) Correcting Problems with htmlFilterContentScraper.java
20 years ago
orbiter b79070b471 fixed proxy/scraper
20 years ago
orbiter 419f8fb398 fixed bugs/missing code regarding new crawl stack
20 years ago
orbiter 112c5d3332 the new file-based indexing queue
20 years ago
orbiter 858cd94299 replaced indexing ram-queue by file-based stack-queue
20 years ago
allo a31029c5de == as separator
20 years ago
allo 9f0f45bcae Warning: this is untested.
20 years ago
theli 57c30f1d78 *) bugfix for usage of httpc without gzip content encoding
20 years ago
theli 0e2c33ee55 *) Network.html/Network.java:
20 years ago
orbiter 5159a090b0 fixed parser bug with lowercase force (appeared in: http://spellbound.sourceforge.net/)
20 years ago
orbiter 7f7cbc5019 fixed bug with snippets
20 years ago
orbiter 3607159a9a fixed connection of out-dated peers
20 years ago
orbiter fbef7fed81 adopted latestVersion to float handling
20 years ago
orbiter eb74fa0c82 fixed a bug with snippet-length
20 years ago
orbiter ec8a5ad060 fixed base implementation (verified with www.burningboard.net/archive/index.html)
20 years ago
orbiter 1d3cfb380a experimental implementation of base parsing in html parser
20 years ago
orbiter 86f2aa8478 fixed seed-load date bug (evaluating server date for age computation)
20 years ago
allo 8290969150 Automatic regeneration of Locales on Restart with a new (SVN) Version
20 years ago
allo 129929b396 Preparations for automatic Languagefile upgrade on new YaCy Version.
20 years ago
orbiter 664bceced5 removed debug-lines
20 years ago
orbiter 75ebdbc852 enhanced snippet-generation (case where snippet is too long)
20 years ago
allo bef3aaec38 better Translationsystem.
20 years ago
orbiter 8a4f297324 fixed/enhanced snippet error-handling; suppression of results where no snippet exists
20 years ago
orbiter 712fe9ef18 bugfixed utf-8 decoding and parser
20 years ago
allo 82d78fdab7 new Implementation and basic bbCode(Ok, opnly one Command ...)
20 years ago
allo a60f5efcd6 rename
20 years ago
theli 6e97f70549 *) httpd.java: improved errorhandling
20 years ago
theli 9d8c66fb5e *) adding possibility to forward received yacy-messages (htroot/yacy/message.java)
20 years ago
theli 08e4334c1d *) Status.java: showing amount of time since last upload of seed-file
20 years ago
theli eee6322aaf *) Adding redirection support to plasmaCrawlWorker.java
20 years ago
theli cd279907c0 *) Adding redirection support to plasmaCrawlWorker.java
20 years ago
theli 6697d5e52e *) correcting fkt. mediaExtContains
20 years ago
low012 a333715ffd This changes eliminate a way to cripple Message pages on other peers by sending special messages with HTML tags. (See forum for details.)
20 years ago
orbiter abba8fe61b fixed utf-8 decoding in htmlFilterAbstractScraper and removed httpd timing
20 years ago
orbiter 3addf58046 enhanced snippet-loading with threads
20 years ago
orbiter 4afcf10158 added kelondroHashtable (not finished yet)
20 years ago
orbiter 56d28a16f0 bugfixes
20 years ago
orbiter d6c85228a6 enhanced snippet computation
20 years ago
theli d53b2393e5 *) autoconfig.java: ip address was not reported correctly when port-forwardin is on
20 years ago
theli fafda068f9 *) allowing crawler to process resources with statuscode 203
20 years ago
theli aae9a433a6 *) correcting usage of supportedFileExt-List
20 years ago
orbiter 1e7f062350 many bugfixes, memory leak fixes, performance enhancements; new kelondroHashtable; activated snippets
20 years ago
orbiter 68dc2b0c6b added kelondroArray, the basis for upcoming kelondroHash and some bug fixes
20 years ago
orbiter 5728cdf8b5 bugfix
20 years ago
orbiter a19541e563 code-enhancements after analysis with AppPerfect
20 years ago
theli 0e48177ad8 *) Bugfix of NullPointerException when sending Error Respond
20 years ago
theli bf78e0a6a4 *) Bugfix of Socket closed Exception
20 years ago
orbiter 85075269a6 extended fail-safe memory-managament. prevents too much allocation, too often GC and should help for the 100%CPU-bug
20 years ago
orbiter e3c92818db avoiding OutOfMemoryError routines
20 years ago
orbiter 3e8ee5a46d enhanced caching in kelondroRecords and added better synchronization/finalizer
20 years ago
orbiter 3be98f194d tried to find the socket bug
20 years ago
theli dff937a9a3 *) Adding some logging statements to detect problems with seed upload functionality
20 years ago
theli 98230a67f0 *) correcting version number for remote port forwarding feature check
20 years ago
theli 9f5d239051 *) correcting version number for remote port forwarding feature check
20 years ago
theli 0405de7635 *) Avoiding NullpointerException while testing exception test
20 years ago
theli db3ed75728 *) closing stream correctly
20 years ago
theli aea355c03c *) adding test for connection status of port forwarding feature
20 years ago
theli bf66c6d6ec *) Bugfix for java.lang.NoClassDefFoundError: de/anomic/yacy/seedUpload/sshc:de/anomic/yacy/seedUpload/sshc
20 years ago
orbiter 5d06ded005 enhanced html parser speed
20 years ago
orbiter 5a490aa065 fixed html parser
20 years ago
orbiter a25b5b4986 fixed possible memory leak in htmlScraper: be aware that now links can get lost; further work necessary
20 years ago
orbiter dbda6e1e85 corrections to port forwarding
20 years ago
theli 822c9fa276 *) forgotten to add new classes for server port forwarding under version control
20 years ago
theli 9e47ba5ad6 *) adding missing calls for function close() to avoid "too many open file" bug
20 years ago
theli 9a98988c3c *) Bugfix for SSL/NIO Bug
20 years ago
orbiter a1ffc27041 preparations for image/movie/music indexing
20 years ago
orbiter f45dc29f35 maintenance
20 years ago
orbiter a5b40923b6 added word migration to assortments (start with 'java -classpath classes yacy -migratewords')
20 years ago
theli 33e992cc8b *) unsing println instead of print
20 years ago
theli 890e3f4d4a *) adding missing calls for function close() to avoid "too many open file" bug*) adding
20 years ago
orbiter beb652847e added run-target for ant (needed for NetBeans4)
20 years ago
theli c7d294d8d4 *) Bugfix for:
20 years ago
theli 6dd3ec0dc4 *) Adding debug="true" debuglevel="lines,vars,source" to ant build files
20 years ago
orbiter 4f9c30ef49 using mime-type instead of file extension for doctype
20 years ago
theli 8b89f433b1 *) Changing loglevel to debug
20 years ago
theli bbc46f037f *) avoid exception throwing on already catched exception
20 years ago
theli f0440911e8 *) removing unnecessary stacktrace printout
20 years ago
theli ee9e110366 *) removing old logging configuration properties from yacy.init
20 years ago
orbiter 878ff0ae7b corrections
20 years ago
theli 361ba50056 *) changing reference to logger
20 years ago
theli 7b4778b4b9 *) changing reference to logger
20 years ago
(no author) dedadd1067 *) remove old serverLog class
20 years ago
theli e5ee4198af *) changing reference to logger
20 years ago
theli 49a23bb89c *) adding reference to logger
20 years ago
theli 02d43a7bd4 *) adding additional methods that should be implemented by all
20 years ago
theli 652a19cf13 *) adding a method to read and zip a file (needed by the filehandler)
20 years ago
theli af9cd67334 *) migration to Java NIO
20 years ago
theli 1eff96f471 *) removing buzy waiting
20 years ago
theli c1a4e0dc28 *) changing reference to logger
20 years ago
theli d0083f845f *) changing reference to logger
20 years ago
theli 1b5ae054f8 *) changing reference to logger
20 years ago
theli 68f30811fa *) changing reference to logger
20 years ago
theli fbbea813c5 *) changing references to logger
20 years ago
theli 451ca6b818 *) changing reference to logger
20 years ago
theli 4d7540e679 *) adding new constants
20 years ago
theli cb17ff4aa2 *) adding support of proxy access logging (much similar to squids
20 years ago
theli f157181086 *) starting implementation of Content-MD5 header
20 years ago
theli d2042ec8fa *) adding constants for all needed connection properties
20 years ago
theli 06b0db2cac *) adding toString method to
20 years ago
theli 7b0b72dd23 *) adding new streams for
20 years ago
theli 318dfc4636 *) activating new logging framework
20 years ago
theli aae3e3b7ab *) reusing buffer instead of creating a new one
20 years ago
theli 829b65c1c8 *) adding additional classes needed for new logging
20 years ago
low012 8c2789b22a to catch is an irregular verb
20 years ago
orbiter 4574fa4ce7 bugfixes
20 years ago
theli 83b41ef2f7 *) Adding timeouts for shutdown
20 years ago
theli ef6851798b *) changing thread priority while parsing a pdf file to avoid 100% CPU usage.
20 years ago
orbiter 33f9315e58 implemented multithreading of indexing
20 years ago
orbiter ca3b4ccaf4 added snippet-routines (not yet finished)
20 years ago
theli 167ddd5612 *) Displaying "SVN xxx" instead of "@REPL_VERSION@" if yacy is started directly from SVN
20 years ago
orbiter ee0758fe4d bugfixes/empty-dir-deletion/snippet-test-activation
20 years ago
orbiter 594c591223 changes towards 0.38
20 years ago
orbiter d8fdc2526e added experimental snipplet-generation (to be disabled for 0.38)
20 years ago
orbiter 3771b10b89 implemented automated migration indexCache 0.37 -> indexAssortmentCluster
20 years ago
orbiter e89ded9e41 bugfixes
20 years ago
orbiter 650ca3955a added flush-thread for index cache and added language-name mapping in Language_p
20 years ago
mkossin 3dd3431c58 Fixes a Problem with mySeed.txt. If this file is accessible but has no content it will be recreated: http://www.yacy-forum.de/viewtopic.php?p=3657#3657
20 years ago
orbiter 3d8a2ff937 enhanced parallelization of local/global/remote crawling
20 years ago
orbiter a05d738ea4 enhanced caching, removed bug causing outOfMemory
20 years ago
allo 3b03d99797 Bugfix: Wrong Path of DATA/LOCALE Dir
20 years ago
allo 8796987def Copy the shipped Locales on Startup to DATA/LOCALES
20 years ago
allo f57d280afd serverLog instead of System.out.println
20 years ago
allo a8b2dccc7a New Translation System.
20 years ago
orbiter 21110dcd5e fixed bugs with open files and caching
20 years ago
orbiter f8f8dd05db fixed "Too many open files" - bug
20 years ago
theli 74eb21f62e *) adding image tag into rss template
20 years ago
orbiter 5f90daa265 implemented localization environment
20 years ago
theli 84f9d8f7f0 *) migrating ant build files to generate a single extension tar per default
20 years ago
orbiter fdd606c8c8 fixed bugs
20 years ago
theli 8bd49ba535 *) setting root dir for all tar files properly
20 years ago
orbiter 0c35171c85 assortment fine-tuning
20 years ago
orbiter 76dc892017 refined assortment
20 years ago
theli 0e1d9e9722 *) shrinking httpc linebuffer when httpc is returned to pool. This is done to free memory
20 years ago
theli 0484c41a84 *) replacing system.xxx.println with logging statements
20 years ago
theli c2378a968f *) setting thread name for search threads
20 years ago
theli 7994c485f1 *) Trying to set the document title properly
20 years ago
theli 285936d778 *) trying to set document title properly
20 years ago
theli 573a8e8047 *) setting document title properly
20 years ago
orbiter 4b01ff7548 activated assortments, removed write-queues
20 years ago
allo d63fae5bd4 SVN Templates Restored
20 years ago
rramthun e015d35b21 Updated version number and date (please don't forget in the future)
20 years ago
orbiter e26ac60c3e modified assortment data structures
20 years ago
allo ced07e3aec javadoc comment added
20 years ago
orbiter 79be6f003d enhanced Assortment class
20 years ago
theli 9ee3e69021 *) Solving "Warning: You did not close the PDF Document" problem when an OutOfMemory Exception occured ...
20 years ago
orbiter 5c6147a54c introduced assortment structure (generalization of singletons)
20 years ago
theli 05ab7c4d68 *) Correcting Problems with "transparent proxy" mode.
20 years ago
theli 73e297f30f *) adding proper default values for RealtimeParsableMimeTypes if something goes wrong with the configuration file
20 years ago
theli 0db0619204 *) Correcting wrong path
20 years ago
theli 893a662329 *) Adding missing cast statement
20 years ago
theli 361f05978d Multiple updates regarding the yacy seedUpload facility,
20 years ago
theli c57306f620 *) Correcting Problem with Template Include - Unresolved filename
20 years ago
allo 07ca4083d3 right Prefix for Includes
20 years ago
theli bcdc44c065 *) Adding more constants. Needed for next commit.
20 years ago
theli ddc5675781 *) Correcting typo
20 years ago
theli d2c4e9a55e *) Implementing yacy forum wishlist item: "Pause Crawling"
20 years ago
rramthun 2d751ba831 Fixed a spelling mistake
20 years ago
theli 511f1c9b25 *) Bugfix of http://www.yacy-forum.de/viewtopic.php?p=2508&highlight=#2508
20 years ago
(no author) 8559c6ed5f *) removing obsolete class
20 years ago
theli 4dd387aae9 *) moving constants (see last commit) to proper httpHeader class
20 years ago
theli aea3479a9c *) introducing of additional constants
20 years ago
orbiter e5e46eccf8 fixed version/date
20 years ago
orbiter 287d2e6f10 further enhanced caching (new cache flush methods)
20 years ago
orbiter 376b917c91 fixed shut-down by stopYACY.sh
20 years ago
orbiter ea478f3975 enhanced indexing-caching
20 years ago
orbiter 0cfe94bb66 fixed last commit + added missing files
20 years ago
orbiter b4030e5023 implemented serverSwitchActions - action-hooks
20 years ago
theli 5a651284a2 *) Adding server shutdown hook
20 years ago
allo 77fd0fdf3f Includes, which use a normale Template Key to find the sourcefile
20 years ago
allo abb6750090 added Include Funktion. #%include.inc%# includes other Templates
20 years ago
theli 6f4d2e5272 *) fixing replace bug.
20 years ago
theli f678cda514 *) correcting bug in transparent proxy support
20 years ago
theli a5fec449c8 *) setting threadnames for kelondroMap:writequeue and publishSeed
20 years ago
theli dae6f46812 *) Forgotten to do transparent proxy support also for http head and post
20 years ago
theli 74f12bb0f3 *) adding transparent proxy support
20 years ago
theli 92a65874d6 *) fixed proxy authentication bug:
20 years ago
orbiter 10a4a2741d fixed missing close
20 years ago
orbiter db1da3345d introduced singleton-database
20 years ago
theli cbdc499ba6 *) adding many missing (File)?(Input|Output)Stream.close() calls to avoid "Too many open files bug".
20 years ago
orbiter a9b22647dc fixed bug in indexDump.stack - generation
20 years ago
orbiter 1d7fed87dc redesign of index caching - removed indexCache.db
20 years ago
rramthun 3f85978519 Fixed one spelling mistake, limited input for ICQ numbers to 9 digits and made ICQ number in peer profiles clickable.
20 years ago
theli 1dad015b0b *) Migration of Ant build files
20 years ago
theli 2aa5fe8f50 *) Import statements reorganized
20 years ago
rramthun 978c16edd3 Deleted startYACY_hidden.bat and fixed annoying spelling mistake.
20 years ago
theli 351c86d5d9 *) Migration of optional Content Parser integration
20 years ago
orbiter d0010ff0b0 last changes for release 0.37
20 years ago
orbiter f99930c04b fixed brute-force + peer-disconnect - Bug
20 years ago
orbiter c7c6aaf06e many bug-fixes
20 years ago
orbiter 48650c082c fixed 100%-CPU-Bug in plasmaCondenser
20 years ago
orbiter 995673d795 several bugfixes
20 years ago
orbiter 2de90020ed fixed caching+synchronization+brute-force-denial
20 years ago
orbiter 9156fd53bc fixed bugs in last commit
20 years ago
orbiter e25f2354c2 removed synchronization and thread blockings
20 years ago
theli 3756e6d20f *) "Httpc object was not returned to object pool." bug fixed.
20 years ago
theli 47e426ff7e *) one possible deadlock (because of nested object locks) removed in class kelondroMap
20 years ago
theli 58a65b60bd *) synchronized keyword removed from function processLocalCrawling to avoid deadlocks.
20 years ago
theli 65fc650109 *) plasmaCrawlLoader shutdown problem fixed (hopefully)
20 years ago
orbiter ba16da72b4 fixed not-working kelondroRecords-Cache
20 years ago
orbiter d03d60f8b5 separated yacy-core from yacy-libx; fixed makerelease
20 years ago
allo c09c54c652 staticIP Property, for people with dyndns aliases ;-)
20 years ago
allo d005d7484e yacyDebugMode - allow Lan-IPs for testing
20 years ago
orbiter 7fb645b0ab enhanced crawling performance, changed memory settings, new performace options
20 years ago
theli 10078bb354 *) date string was accidentally replaced with the current value
20 years ago
theli fd584c113c *) some minor changes
20 years ago
theli f44b219e44 *) Eclipse has accidentally copied in the wrong file header into the new files (because these headers were accidentally set as default for the whole workspace instead of the project)
20 years ago
theli 081ebd5517 *) I've accidentally used Java 5.0 syntax for enumerations
20 years ago
theli 58b1a0ba40 *) adding an new package for extra content parsers
20 years ago
orbiter 8b31f9e202 enhanced shut-down behaviour & added experimental nio-wrapper for kelondroRA (not active yet)
20 years ago
orbiter 87a61a01c2 fixed bad-gzip-trailer behaviour (now cuts off trailer)
20 years ago
orbiter 00f223cfc1 fixed post-parsing (a case when the bluelist is empty)
20 years ago
theli c9c0a1f11c *) Trying to speedup local crawling
20 years ago
(no author) 942914ffd2 *) Adding additional functions to serverByteBuffer so that it
20 years ago
(no author) 432e01910b *) Bugfix: Image falsification
20 years ago
orbiter 97ec8d65e4 fixed makerelease & clean-up of dead code
20 years ago
(no author) 4a76ccc6d6 *) Some minor bugfixes
20 years ago
(no author) 1fec00bc24 *) Bugfix to avoid Nullpointer-Exceptions
20 years ago
(no author) f39812da91 *) Some performance improvements
20 years ago
orbiter b9203bdb50 bug fixes and code cleaning
20 years ago
allo c13411c198 Buildfile which inserts the Date.
20 years ago
(no author) b7d4389e4b *) support for Proxy Auto-Config File generation added.
20 years ago
orbiter c0807abd33 new crawl/proxy/cache design + fixes
20 years ago
orbiter e7d055b98e very experimental integration of the new generic parser and optional disabling of bluelist filtering in proxy. Does not yet work properly. To disable the disable-feature, the presence of a non-empty bluelist is necessary
20 years ago
orbiter 96516fc9d8 fixed bugs (search+kelondroException, dns)
20 years ago
orbiter a87a17a3c8 prepared generic text parser environment
20 years ago
orbiter e374aca2cd enhanced exception handling in kelondro
20 years ago
orbiter 072052f150 fixed bugs (dns, seedDB)
20 years ago
orbiter 89eb9a2292 fixed bug with crawl profiles
20 years ago
orbiter 248077d3f0 initial load with yacy 0.36
20 years ago