Instructing spiders/crawlers

[James Davis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Biggins wrote:

> Right now, it's increasingly advisable to use all three, though I expect
> the sitemap will eventually substantially dominate because it is by far
> the most powerful. 

XML sitemaps and robots.txt aren't equivalent. XML sitemaps tell spiders
what pages are available to index but don't provide instruction on where
it's forbidden to look for pages.

James

- --
James Davis	+44 1235 822 229	PGP: 0xC7C92EB7
JANET-CERT	0870 850 2340	(+44 1235 822 340)
		Atlas Centre, Chilton, Didcot, Oxfordshire, OX11 0QS
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGQ0o/Ile3s8fJLrcRAvEzAJ4pDpGw6O1KOrIXUmQXPNNukQyQAACgvzmH
87+HrIHl1nwYQtX/rnD/CKM=
=L+up
-----END PGP SIGNATURE-----