Commit Graph

2 Commits (8a0e35618b7a216955fb8d8be3228d02c18106d1)

Author SHA1 Message Date
orbiter 3879a0ecd0 replaced java.net.URL usage by use of new class de.anomic.net.URL
19 years ago
allo 62664d7252 AJAX Check for robots.txt before crawling.
19 years ago