Semantic Crawler is the component that provides data to Index Generator. In the case of web harvesting or other sorts of dynamic data, it is unreasonable to expect that a sophisticated Index Generator is also, for example, a sophisticated harvester. The Semantic Crawler handles gathering data for the Index Generator and, to the extent they agree on delivery methods, can pre-process the data for the Index Generator to use.