Some spider activities are legitimate – maintaining mirror copies of the site to improve accessibility, or search engines looking for text and keywords to add to their catalogues. Search engines also maintain caches of pages for their users so that load times can be reduced. Other spider activities may not be. A web site owner may be aware that a ‘spider’ is reading his site, but he cannot know its real intent any more than you can know why a person reads a book.