On Fri, Jul 30, 2010 at 07:55:17PM +0100, Peter Fairbrother wrote: > Can someone answer a question for me please - how does crawling work? > How does the crawler find different pages? By following links from other pages, generally speaking.