Wget
GNU wget is a free software tool that permits advanced file retrieval from networks and the Internet. For example, it allows for simplified mirroringing and recursive downloading.It is written in a highly portable style of C. It is a command line program and is used mainly on Unix systems. Graphical front-ends such as gwget and builds for Microsoft Windows platforms are also available.
Wget is also non interactive, meaning that it can work in the background, while the user is not logged on. By contrast, most Web browsers require constant user's presence, which can be a great hindrance when transferring a lot of data.
When mirroring, or "recursive downloading", wget respects the robots exclusion standard. Wget can be instructed to convert the links in downloaded HTML files to the local files for offline viewing.
File name wildcard matching and recursive mirroring of directories are available when retrieving via FTP. Wget can read the time-stamp information given by both HTTP and FTP servers, and store it locally. Thus Wget can see if the remote file has changed since last retrieval, and automatically retrieve the new version if it has. This makes Wget suitable for mirroring of FTP sites, as well as home pages.
Wget has been designed for robustness over slow or unstable network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved, continuing from where it left off.
Wget supports proxy servers, which can lighten the network load, speed up retrieval and provide access behind firewalls.
External Links
Portions of this article are based on the Wget documentation.
