Commit Graph

3 Commits (2f181d0027e31c61435e321a1d709a359ca3c1cf)

Author SHA1 Message Date
orbiter 54af9e6b49 - added parsing of robots meta-tag in html headers to detect a noindexing request
15 years ago
orbiter 3528b970d6 - refactoring
15 years ago
orbiter b79f4f062f refactoring of yacy documents and parsers: they depend now only on the kelondro classes
15 years ago