theli
4e07828807
*) httpdProxyHandler.java
...
- harmonizing proxy exception handling
- adding malformed URL + blacklist check for http head method
- adding malformed URL check to http post method
- chunked encoding is now not used anymore for http post if clients
are http/0.9 or http/1.0 clients (same behaviour as already implemented for get)
- now an exception will be thrown on internal httpc errors to force an error output
to the client or a connection close. This should help to fix the "binary data in browser window" bug
*) plasmaSwitchboard.java
- fixing the following Bug
E 2005/09/03 18:02:42 PLASMA Could not index URL http://mis04.de/FAIL/snot.php : null
java.lang.NullPointerException
at de.anomic.plasma.plasmaSwitchboard.processResourceStack(plasmaSwitchboard.java:1000)
at de.anomic.plasma.plasmaSwitchboard.deQueue(plasmaSwitchboard.java:625)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at de.anomic.server.serverInstantThread.job(serverInstantThread.java:95)
at de.anomic.server.serverAbstractThread.run(serverAbstractThread.java:243)
This bug could occure if the cached responseHeader is null
- getting the mimeType now from the parsed document instead of the responseHeader because the
mimeType could have been changed during content parsing (e.g. because of the mimetypeParser)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@656 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
f90f699ab1
missing package line.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@655 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
89d7c6e45e
*) Bugfix: offset of 1 prevented correct replacement of HTML
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@654 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a7256e8f4e
*) Adding X-Forwarded-For Header
...
See: http://www.yacy-forum.de/viewtopic.php?t=1118&highlight=xforwardedfor
*) httpc.java: Bugfix for incorrect http response statuscode parsing
In some situations the statustext whas chopped
*) Adding a lot of fileheaders containing YaCy copyright and license
*) httpd.java: Adding additional debugging http header that should help du detect
the "binary data in browser window" bug.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@653 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
06a451768f
a simple robotsParser.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@652 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
81cb8feb15
back to 649 :/
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@651 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
5194511e8e
*) attempt to find bug
...
See: http://www.yacy-forum.de/viewtopic.php?t=1121
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@650 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
8f3d19b331
*) Suppress stacktrace on proxy error for "Connection reset"
...
See: http://www.yacy-forum.de/viewtopic.php?t=1107
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@646 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
6991b9e2b9
*) Suppress stacktrace on crawler error for "Connection reset"
...
See: http://www.yacy-forum.de/viewtopic.php?p=9071
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@645 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a20814291f
*) Bugfix for "Race condition zwischen httpc und switchboard"
...
See: http://www.yacy-forum.de/viewtopic.php?p=9036
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@644 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
286853fd39
*) Bugfix for "YACY hängt sich beim Beenden auf" Bug
...
See: http://www.yacy-forum.de/viewtopic.php?p=8997
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@643 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a47f9238fe
*) Blacklist is now also used by the crawler
...
See: http://www.yacy-forum.de/viewtopic.php?t=1069
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@642 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
dc0a2d4c11
*) Bugfix for Loader Queue:
...
Job count was not displayed correctly
*) IndexingQueue:
- now it's possible to delete single entries from the queue
- now it's possible to clear the whole queue
See: http://www.yacy-forum.de/viewtopic.php?t=995
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@641 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
732a107160
*) Bugfix for "-UNRESOLVED_PATTERN-" Bug on IndexCreateWWWLocalQueue_p.html and "urlEntry.url() == null" Bug
...
- Logging message for "urlEntry.url() == null" is now displayed as info
- IndexCreateWWWLocalQueue_p.html now detects null entries while looping throug the list and removes them automatically
See:
- http://www.yacy-forum.de/viewtopic.php?t=532#8781
- http://www.yacy-forum.de/viewtopic.php?t=639
- http://www.yacy-forum.de/viewtopic.php?t=1071
- http://www.yacy-forum.de/viewtopic.php?t=338
- http://www.yacy-forum.de/viewtopic.php?t=980
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@640 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
33aaffbfc6
*) Displaying content size of each entry in indexing queue
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@639 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
022c1ab179
performance fix for yacyDebugMode and useYacyReferer.
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@638 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
allo
286442fbc5
do not Use YaCy-Sites as Referer, if useYacyReferer = false
...
http://www.yacy-forum.de/viewtopic.php?p=8896#8896
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@637 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f9eb550fbc
*) Bugfix for NullpointerException in serverAbstractSwitch.setConfig
...
See: http://www.yacy-forum.de/viewtopic.php?t=692#5575
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@636 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
7626823519
BUGFIX for last 'commit'
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@635 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
971756e8dd
the delete size is smaller
...
See: http://www.yacy-forum.de/viewtopic.php?t=1084
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@634 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
0471019606
*) IndexCreateIndexingQueue_p.html now also shows indexing jobs that are currently in process
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@633 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3dc6845bef
*) Logging error message to logging output if no errormessage can be send to the user by the proxy
...
Note: This is only done if you set the logging level of PROXY to FINE
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@632 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3df5c7a6cf
*) Displaying an proxy error page instead of a white page if the server has closed
...
the connection before yacy was able to receive the http response line
See: http://www.yacy-forum.de/viewtopic.php?p=8866#8866
http://www.yacy-forum.de/viewtopic.php?t=704
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@630 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
cc493ef8c1
Added change from Hermes
...
See: http://www.yacy-forum.de/viewtopic.php?t=1050
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@629 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
4edb5b6f1e
*) Bugfix for "ProxyAccess logging" Bug
...
Loglevel was not set corretly for Proxy.access logger
See: http://www.yacy-forum.de/viewtopic.php?p=8875#8875
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@628 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
7a7254713d
*) Moving Logging directory per default to DATA/LOG
...
See: http://www.yacy-forum.de/viewtopic.php?t=940#7656
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@627 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
bead8a32aa
*) IndexCreate_p.java:
...
Crawler StartURLs will now also added to the errorURL-DB if an error occures on this url
*) kelondroStack.java, plasmaSwitchboardQueue.java
Adding method which returns a list of all entries in the queue. This list is used by IndexCreate_p.java
instead of an iterator to display the indexing-list.
Advantages: avoid concurrent modifications of the list while displaying it.
Speedup because now we have to access only one sync function instead of multiple ones
(one for each entry)
*) IndexCreateIndexingQueue_p.java
Using new list() function of plasmaSwitchboardQueue
*) httpdFileHandler.java
If a servelet returns the special value "LOCATION" the httpFileHandler does a Redirection of
the Browser to the URL specified by the servelet. This can e.g. be used when a http get request is
used insead of a post request, but a refresh should not be allowed.
*) IndexCreateWWWLocalQueue_p.html
Now it's possible to delete single entries of the local crawler queue
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@626 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
48aaf703cc
*) Adding additional logging output to detect crawling problems
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@625 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
59b8a98c7e
*) Bugfix for suppressing of stacktrace in log on crawler error "MalformedURLException"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8840
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@623 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
low012
2ee4f9f4e5
*) added escape characters [= and =] to yacyWiki
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@622 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
c1d7527929
better cache cleanup
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@621 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
2e6df95786
*) adding toString method
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@620 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
ebbd063c92
*) Making mimeTable static final
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@619 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
60e77dcc60
*) Adding additional loglevel finer + finest according to Thread http://www.yacy-forum.de/viewtopic.php?p=8750#8750
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@618 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
fa54b5f38d
cleanup spaces
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@617 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
4fd5b95b1f
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logSevere instead of logFailure and logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@615 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
6adf8a4bde
*) Renaming Logger function names to reflect the proper Java Logging API Loglevels
...
- please use logFine instead of logDebug
- please use logFailure instead of logError
See: http://www.yacy-forum.de/viewtopic.php?p=8726#8726
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@614 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
f19c09b227
*) Suppress stacktrace on crawler error for "MalformedURLException"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8733#8733
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@613 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
cc1df08069
*) Adding missing synchronized blocks
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@608 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
0dfa8b62e2
*) Changing Proxy-Useragent string according to thread http://www.yacy-forum.de/viewtopic.php?p=8183#8183
...
A typical useragent string now e.g. looks like:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.10; YaCy 0.401/00602; yacy.net) Gecko/20050716 Firefox/1.0.6
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@607 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
borg-0300
bf14e6def5
*) proxyCache, proxyCacheSize can be changed under 'Proxy Indexing'
...
- path now are absolute
*) move path check from plasmaHTCache to plasmaSwitchboard
- only one path check when starting
*) small other
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@606 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
3dfda1c9da
*) More verbose output on ftp-seed-upload failure
...
See: http://www.yacy-forum.de/viewtopic.php?p=8000#8000
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@605 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
e3aa3a2d75
*) Bugfix for ProxyAccess Logger
...
URL was accidentally logged without the parameters
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@604 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
9b818b1ce3
*) Pausing Crawlers if there is not enough space on disk
...
See: http://www.yacy-forum.de/viewtopic.php?p=8648
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@603 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
b33094e925
*) Trying to solve "Too many open files bug"
...
*) Temp.Bugfix for "Bug in Index Restore"
See: http://www.yacy-forum.de/viewtopic.php?p=8647#8647
Orbiter: Please take a look
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@602 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
b67f008eb8
*) Trying to solve "Too many open files bug"
...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@601 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
34790acf02
*) Bugfix for suppressing of stacktrace in log on crawler error "unknown host"
...
See: http://www.yacy-forum.de/viewtopic.php?p=8615#8615
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@600 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
a812fb86cc
*) Port Forwarding Feature does not detect broken connection properly.
...
Therefor a test-request was added to the isConnected function to detect broken connections
and to keep open connections alive
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@596 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
af7b8f75bd
*) Making proxyAccessLogging configureable via yacy.logging file
...
- logging can be disabled now
- logging directory / filelimit / rotation count can be configured now
See: http://www.yacy-forum.de/viewtopic.php?t=965&postdays=0&postorder=asc&start=30#8280
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@595 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago
theli
8132a44305
*) Better error handling if yacy SOAP extension is not installed
...
See: http://www.yacy-forum.de/viewtopic.php?t=1040
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@594 6c8d7289-2bf4-0310-a012-ef5d649a1542
20 years ago