theli
d2c4e9a55e
*) Implementing yacy forum wishlist item: "Pause Crawling"
...
see: http://www.yacy-forum.de/viewtopic.php?t=48
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@118 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
2d751ba831
Fixed a spelling mistake
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@117 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
511f1c9b25
*) Bugfix of http://www.yacy-forum.de/viewtopic.php?p=2508&highlight=#2508
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@116 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
8559c6ed5f
*) removing obsolete class
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@115 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
4dd387aae9
*) moving constants (see last commit) to proper httpHeader class
...
*) migrating fileHandler + proxyHandler to use constants instead of hardcoded values
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@114 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
aea3479a9c
*) introducing of additional constants
...
to improve maintainability of the sourcecode
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@113 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e5e46eccf8
fixed version/date
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@112 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
287d2e6f10
further enhanced caching (new cache flush methods)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@111 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
376b917c91
fixed shut-down by stopYACY.sh
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@108 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
ea478f3975
enhanced indexing-caching
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@107 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
0cfe94bb66
fixed last commit + added missing files
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@106 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
b4030e5023
implemented serverSwitchActions - action-hooks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@105 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
5a651284a2
*) Adding server shutdown hook
...
... to allow a graceful shutdown even if the java runtime was interrupted using
<ctrl>+<c>
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@104 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
77fd0fdf3f
Includes, which use a normale Template Key to find the sourcefile
...
Syntax: #%[key]%#
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@103 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
abb6750090
added Include Funktion. #%include.inc%# includes other Templates
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@102 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
6f4d2e5272
*) fixing replace bug.
...
using
stringvar = stringvar.replace(xxx)
istead of
stringvar.replace()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@101 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f678cda514
*) correcting bug in transparent proxy support
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@99 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a5fec449c8
*) setting threadnames for kelondroMap:writequeue and publishSeed
...
so that a thread dump is more verbose
*) Moving code for transparent proxy support to a separate function
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@98 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
dae6f46812
*) Forgotten to do transparent proxy support also for http head and post
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@97 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
74f12bb0f3
*) adding transparent proxy support
...
Now a firewall can transparently redirect all
http traffic through yacy.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@96 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
92a65874d6
*) fixed proxy authentication bug:
...
http://www.yacy-forum.de/viewtopic.php?t=174
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@95 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
10a4a2741d
fixed missing close
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@94 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
db1da3345d
introduced singleton-database
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@92 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
cbdc499ba6
*) adding many missing (File)?(Input|Output)Stream.close() calls to avoid "Too many open files bug".
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@90 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
a9b22647dc
fixed bug in indexDump.stack - generation
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@88 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
1d7fed87dc
redesign of index caching - removed indexCache.db
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@86 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
3f85978519
Fixed one spelling mistake, limited input for ICQ numbers to 9 digits and made ICQ number in peer profiles clickable.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@85 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
1dad015b0b
*) Migration of Ant build files
...
- now the ant build file has the same functionality as the makerelease build file
- from now on the ant build files can be used instead of the makerelease build script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@84 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2aa5fe8f50
*) Import statements reorganized
...
Now it's easier to determine which class really uses which other class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@82 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
rramthun
978c16edd3
Deleted startYACY_hidden.bat and fixed annoying spelling mistake.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@81 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
351c86d5d9
*) Migration of optional Content Parser integration
...
- each additional parser must be in a subpackage
of plasma.parser
- each parser must have its own ant build file (which will
be called automatically from the main build file)
- Calling the main build file results in building a separate
zip file for each optional parser. This zip file includes:
+ sources of the Parser.java
+ compiled classes of the Parser.java
+ needed additional libs (libx)
- To install an additional parser the user simply needs to
extract the zip file listed above into his/her yacy directory.
- The configuration (enabling/disabling) of a parser can be done
via the webinterface (currently the settings dialoge) and is
done "on-the-fly". The installation can not be done "on-the-fly"
at the moment because of classpath issues.
- The classpath of the linux startup/stop scripts is generated
automatically now (including all libraries from lib and libx).
*) Bugfix: File Extension was not calculated correctly by the crawler
e.g.: file extension was accidentally: .php?param=value
Corrected.
*) Adding additional parser for parsing of rss/atom feeds
- added needed libs to do this.
TODO:
- automatic building classpath for windows startup scripts
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@78 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
d0010ff0b0
last changes for release 0.37
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@76 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
f99930c04b
fixed brute-force + peer-disconnect - Bug
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@75 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
c7c6aaf06e
many bug-fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@73 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
48650c082c
fixed 100%-CPU-Bug in plasmaCondenser
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@72 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
995673d795
several bugfixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@71 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
2de90020ed
fixed caching+synchronization+brute-force-denial
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@67 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
9156fd53bc
fixed bugs in last commit
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@65 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e25f2354c2
removed synchronization and thread blockings
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@63 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3756e6d20f
*) "Httpc object was not returned to object pool." bug fixed.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@62 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
47e426ff7e
*) one possible deadlock (because of nested object locks) removed in class kelondroMap
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@61 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
58a65b60bd
*) synchronized keyword removed from function processLocalCrawling to avoid deadlocks.
...
This synchronized keyword is not needed anymore because of the crawler jobqueue which
is responsible for the synchronization now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@60 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
65fc650109
*) plasmaCrawlLoader shutdown problem fixed (hopefully)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@59 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
ba16da72b4
fixed not-working kelondroRecords-Cache
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@56 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
d03d60f8b5
separated yacy-core from yacy-libx; fixed makerelease
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@55 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
c09c54c652
staticIP Property, for people with dyndns aliases ;-)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@54 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
d005d7484e
yacyDebugMode - allow Lan-IPs for testing
...
where was the Code from 0.25 lost?
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@53 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
7fb645b0ab
enhanced crawling performance, changed memory settings, new performace options
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@51 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
10078bb354
*) date string was accidentally replaced with the current value
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@50 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
fd584c113c
*) some minor changes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@49 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f44b219e44
*) Eclipse has accidentally copied in the wrong file header into the new files (because these headers were accidentally set as default for the whole workspace instead of the project)
...
Fixed.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@48 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
081ebd5517
*) I've accidentally used Java 5.0 syntax for enumerations
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@47 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
58b1a0ba40
*) adding an new package for extra content parsers
...
*) adding content parser for
- pdf (using the pdf-box library)
- doc (using the textmining.org library)
*) adding a Interface for content parsers
*) adding a configuration file which can be used to configure which parser is used for which mimeType
*) Sempahore class was moved and renamed to serverSemaphore
*) Changing yacy shutdown behaviour
Buzy waiting loop for shutdown was removed and replaced with a blocking call (using the semaphore class mentioned above) to the new switchboard.waitForShutdown method.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@46 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
8b31f9e202
enhanced shut-down behaviour & added experimental nio-wrapper for kelondroRA (not active yet)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@44 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
87a61a01c2
fixed bad-gzip-trailer behaviour (now cuts off trailer)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@42 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
00f223cfc1
fixed post-parsing (a case when the bluelist is empty)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@41 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
c9c0a1f11c
*) Trying to speedup local crawling
...
- introduction of a threadpool for crawling
- introduction of a job queue to avoid buzy waiting for a free crawler slot
*) New classes added
- queue for receiving of crawler jobs
- semaphore class to do reader/writer synchronization (mutual exclusion)
- message object to hold all needed data about a crawler job
*) Trying to solve session-thread shutdown problem
- session thread stopped variable is now set from outside before interrupting the
session thread.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@39 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
942914ffd2
*) Adding additional functions to serverByteBuffer so that it
...
can be used instead of a ByteArrayOutputStream
*) Using a serverByteBuffer for lineBuffering in class httpc
instead of a ByteArrayOutputStream
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@35 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
432e01910b
*) Bugfix: Image falsification
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@34 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
97ec8d65e4
fixed makerelease & clean-up of dead code
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@33 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
4a76ccc6d6
*) Some minor bugfixes
...
- httpc: wrong error-message on 404
- httpc: error message was accidentally shown when object
was released from pool
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@31 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
1fec00bc24
*) Bugfix to avoid Nullpointer-Exceptions
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@30 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
f39812da91
*) Some performance improvements
...
- many classes set to final
- implementation of a session-thread pool
- reusage of the server handler class (normally the httpd object)
within the session thread
- implementation of a httpc object pool
- introduction of a linebuffer in httpd which can be reused
- reusing the properties table in the httpc
- added to apache libs (commons-collections, commons-pool) which
are needed for the object/thread pool implementation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@26 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
b9203bdb50
bug fixes and code cleaning
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@22 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
c13411c198
Buildfile which inserts the Date.
...
The Version is set in the source; so it will be correct if you check old versions out.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@21 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
(no author)
b7d4389e4b
*) support for Proxy Auto-Config File generation added.
...
File is accessible using:
http://proxy:8080/autoconfig.pac
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@20 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
c0807abd33
new crawl/proxy/cache design + fixes
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@18 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e7d055b98e
very experimental integration of the new generic parser and optional disabling of bluelist filtering in proxy. Does not yet work properly. To disable the disable-feature, the presence of a non-empty bluelist is necessary
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@17 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
96516fc9d8
fixed bugs (search+kelondroException, dns)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@16 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
a87a17a3c8
prepared generic text parser environment
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@15 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
e374aca2cd
enhanced exception handling in kelondro
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@14 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
072052f150
fixed bugs (dns, seedDB)
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@13 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
89eb9a2292
fixed bug with crawl profiles
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@12 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
orbiter
248077d3f0
initial load with yacy 0.36
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago