From: Ian Jackson <ijackson@chiark.greenend.org.uk>
Date: Tue, 7 Sep 2010 19:03:09 +0000 (+0100)
Subject: add robots.txt to stop crawlers delving into lookup/ and doing searches etc.
X-Git-Tag: 6.7.0~15
X-Git-Url: https://www.chiark.greenend.org.uk/ucgi/~yarrgweb/git?p=ypp-sc-tools.db-test.git;a=commitdiff_plain;h=024f4a767b43a66e6ff004532f039b874e401b80

add robots.txt to stop crawlers delving into lookup/ and doing searches etc.
---

diff --git a/yarrg/web/robots.txt b/yarrg/web/robots.txt
new file mode 100755
index 0000000..38a5ff7
--- /dev/null
+++ b/yarrg/web/robots.txt
@@ -0,0 +1,11 @@
+<%flags>
+inherit => undef
+</%flags>
+<%perl>
+$r->content_type('text/plain');
+</%perl>
+User-Agent: *
+Disallow: /lookup
+Disallow: /test/code/lookup
+Disallow: /test/data/lookup
+Disallow: /test/both/lookup