chiark
/
gitweb
/
~yarrgweb
/
ypp-sc-tools.db-test.git
/ blobdiff
commit
grep
author
committer
pickaxe
?
search:
re
summary
|
shortlog
|
log
|
commit
|
commitdiff
|
tree
raw
| inline |
side by side
add robots.txt to stop crawlers delving into lookup/ and doing searches etc.
[ypp-sc-tools.db-test.git]
/
yarrg
/
web
/
robots.txt
diff --git a/yarrg/web/robots.txt
b/yarrg/web/robots.txt
new file mode 100755
(executable)
index 0000000..
38a5ff7
--- /dev/null
+++ b/
yarrg/web/robots.txt
@@ -0,0
+1,11
@@
+<%flags>
+inherit => undef
+</%flags>
+<%perl>
+$r->content_type('text/plain');
+</%perl>
+User-Agent: *
+Disallow: /lookup
+Disallow: /test/code/lookup
+Disallow: /test/data/lookup
+Disallow: /test/both/lookup