Beginning in late February 2015 the National Library of Australia (NLA) is undertaking its tenth annual crawl and harvest of the Australia web domain. This web crawl is being conducted on behalf of the NLA by the Internet Archive based in San Francisco. This crawl has a target of harvesting 500 million web documents (unique files) from the .au domain for the NLA's archival collection of freely available websites and documents.
Previous crawls of the Australian web domain for the purpose of collecting archival content were conducted during January and February 2014, March and April 2013, March and April 2012, February and March 2011, September and October 2009, July to September 2008, August and September 2007, August and September 2006 and June and July 2005.
For the purpose of these collections, the Australian web domain includes openly accessible web content on the .au top level domain. In addition some sites identified by DNS lookup as having an IP address located in Australia may be included. Because of the scope of this project publishers and webmasters cannot be contacted in advance. However, the content harvested during the whole domain crawl will not generally be made available by the NLA without the permission of the content owners or other legal warrant. In this regard this archival project is distinct from the NLA's PANDORA Archive which is a selective web archive the contents of which are collected and made accessible with the prior permission of the content owners. Content harvested from this domain crawl may at some time be included in the Internet Archive's own Wayback Machine collection. Should authors or publishers have objections to content being available from the Internet Archive collection, it may be removed from public availability in accordance the the Internet Archive's terms of use.