Cheatsheet & Examples: wget

Download a single file

Example Usage: wget http://example.com/file.txt

What it does: Downloads the file located at the specified URL and saves it in the current directory with the original filename.

Command-line Arguments Explained:

• http://example.com/file.txt: The URL of the file to be downloaded.

Save a file with a custom name

Example Usage: wget -O custom_name.txt http://example.com/actual_file.txt

What it does: Downloads the file from the specified URL and saves it under the custom name provided.

Command-line Arguments Explained:

• -O: Specifies the output filename for the downloaded file.
• custom_name.txt: The desired filename for the saved file.
• http://example.com/actual_file.txt: The URL of the file to be downloaded.

Recursively download a website

Example Usage: wget -r http://example.com

What it does: Downloads an entire website, including all linked pages, recursively, and saves them locally.

Command-line Arguments Explained:

• -r: Enables recursive downloading, which means it will traverse and download all pages linked to the target URL.
• http://example.com: The URL of the website to be downloaded.

Download multiple files from a list

Example Usage: wget -i file_list.txt

What it does: Downloads all files listed in the file_list.txt text file, one by one.

Command-line Arguments Explained:

• -i: Reads the list of URLs from the specified input file.
• file_list.txt: A text file containing the URLs of the files to be downloaded.

Check if a file exists without downloading it

Example Usage: wget --spider http://example.com/file.txt

What it does: Verifies whether the file at the URL exists and is accessible, without initiating a download.

Command-line Arguments Explained:

• --spider: Simulates checking the URL (like a "spider" bot) without downloading the file.

Download in the background

Example Usage: wget -b http://example.com/large_file.zip

What it does: Downloads the file in the background, allowing the user to continue using the terminal.

Command-line Arguments Explained:

• -b: Runs wget in the background. Output is logged to a file.

Limit download speed

Example Usage: wget --limit-rate=100k http://example.com/large_file.zip

What it does: Restricts the download speed to the specified value (e.g., 100 kilobytes per second).

Command-line Arguments Explained:

• --limit-rate=100k: Limits the download speed to 100 kilobytes per second.

Resume an interrupted download

Example Usage: wget -c http://example.com/partial_file.zip

What it does: Resumes a partially downloaded file from the last point of interruption.

Command-line Arguments Explained:

• -c: Enables resuming of an interrupted download.

Download only if the file is newer

Example Usage: wget -N http://example.com/updated_file.txt

What it does: Downloads the file only if it is newer than the local version, or if the local file does not exist.

Command-line Arguments Explained:

• -N: Enables time-stamping, checking if the remote file is newer than the local copy.

Download with authentication

Example Usage: wget --user=username --password=secret http://example.com/protected_file.txt

What it does: Downloads a file that requires HTTP authentication credentials.

Command-line Arguments Explained:

• --user: Specifies the username for authentication.
• --password: Specifies the password for authentication.

Use a specific user agent

Example Usage: wget --user-agent="Mozilla/5.0" http://example.com

What it does: Sets a custom user agent string to mimic a browser or bypass restrictions based on user agent detection.

Command-line Arguments Explained:

• --user-agent: Defines the user agent string sent in the HTTP request header.

Download a file with a timeout

Example Usage: wget --timeout=30 http://example.com/really_slow_file.zip

What it does: Sets a timeout for the connection, terminating the download if no response is received within the specified seconds.

Command-line Arguments Explained:

• --timeout=30: Specifies the connection timeout in seconds (e.g., 30 seconds).

Mirror a website (recursively and retain directory structure)

Example Usage: wget --mirror http://example.com

What it does: Mirrors the website, including all subpages and maintaining the original directory structure.

Command-line Arguments Explained:

• --mirror: A shortcut for -r --no-parent --convert-links, which recursively downloads and preserves the site's structure.

Download from a secure (HTTPS) site without certificate verification

Example Usage: wget --no-check-certificate https://example.com/secured_file.txt

What it does: Downloads a file from an HTTPS site without verifying the SSL certificate, useful for self-signed certificates.

Command-line Arguments Explained:

• --no-check-certificate: Skips SSL certificate verification.

Download multiple files with a single command

Example Usage: wget http://example.com/file1.txt http://example.com/file2.txt

What it does: Downloads multiple files by specifying their URLs in a single command.

Command-line Arguments Explained:

• http://example.com/file1.txt http://example.com/file2.txt: URLs of the files to be downloaded sequentially.

Output to a specific directory

Example Usage: wget -P /path/to/dir http://example.com/file.txt

What it does: Saves the downloaded file to the specified directory instead of the current working directory.

Command-line Arguments Explained:

• -P: Specifies the directory where the file should be saved.

Download a file with a custom header

Example Usage: wget --header="Authorization: Bearer token" http://example.com/api/data

What it does: Attaches a custom HTTP header (e.g., for authentication) to the download request.

Command-line Arguments Explained:

• --header="Authorization: Bearer token": Adds the specified HTTP header to the request.

Specify the proxy server

Example Usage: wget --proxy=on http://example.com/file.txt

What it does: Uses the specified proxy server for the download, with proxy settings read from environment variables or the default configuration.

Command-line Arguments Explained:

• --proxy=on: Enables the use of a proxy server. Proxies are typically set via environment variables like HTTP_PROXY or HTTPS_PROXY.

Download files with a retry limit

Example Usage: wget --tries=5 http://example.com/retry_file.txt

What it does: Retries the download up to the specified number of times if it fails.

Command-line Arguments Explained:

• --tries=5: Sets the maximum number of retry attempts for failed downloads.

Download files with a specific output document location

Example Usage: wget --output-document=/path/to/save/file.txt http://example.com/remote_file.txt

What it does: Saves the downloaded file to a specific location, overriding the default filename.

Command-line Arguments Explained:

• --output-document: Specifies the exact path and filename for the saved file.

Download a file using the POST method

Example Usage: wget --post-data="key=value" http://example.com/post_endpoint

What it does: Sends a POST request with the given data to the specified URL.

Command-line Arguments Explained:

• --post-data="key=value": Sends the specified data as the body of the POST request.

Skip parent directories during recursion

Example Usage: wget -r --no-parent http://example.com

What it does: Recursively downloads a website but restricts the process to the exact directory specified, not descending into parent directories.

Command-line Arguments Explained:

• -r: Enables recursive downloading.
• --no-parent: Prevents wget from accessing parent directories of the specified URL.