Commit Graph

3 Commits (7057e0b3e2cf8d49b5a2f9fea6cd5388b7a862e6)

Author SHA1 Message Date
Michael Peter Christen f23c4142e0 added option to configure a custom user agent within allip networks
11 years ago
Michael Peter Christen a2fba6584f use submitted default userAgent if cloning a crawl
11 years ago
orbiter d29b6db270 made crawl start pages public since they do not reveal individual
11 years ago