# Cheatsheet & Examples: wget ## Download a single file Example Usage: `wget http://example.com/file.txt` What it does: Downloads the file located at the specified URL and saves it in the current directory with the original filename. Command-line Arguments Explained: - `http://example.com/file.txt`: The URL of the file to be downloaded. ## Save a file with a custom name Example Usage: `wget -O custom_name.txt http://example.com/actual_file.txt` What it does: Downloads the file from the specified URL and saves it under the custom name provided. Command-line Arguments Explained: - `-O`: Specifies the output filename for the downloaded file. - `custom_name.txt`: The desired filename for the saved file. - `http://example.com/actual_file.txt`: The URL of the file to be downloaded. ## Recursively download a website Example Usage: `wget -r http://example.com` What it does: Downloads an entire website, including all linked pages, recursively, and saves them locally. Command-line Arguments Explained: - `-r`: Enables recursive downloading, which means it will traverse and download all pages linked to the target URL. - `http://example.com`: The URL of the website to be downloaded. ## Download multiple files from a list Example Usage: `wget -i file_list.txt` What it does: Downloads all files listed in the `file_list.txt` text file, one by one. Command-line Arguments Explained: - `-i`: Reads the list of URLs from the specified input file. - `file_list.txt`: A text file containing the URLs of the files to be downloaded. ## Check if a file exists without downloading it Example Usage: `wget --spider http://example.com/file.txt` What it does: Verifies whether the file at the URL exists and is accessible, without initiating a download. Command-line Arguments Explained: - `--spider`: Simulates checking the URL (like a "spider" bot) without downloading the file. ## Download in the background Example Usage: `wget -b http://example.com/large_file.zip` What it does: Downloads the file in the background, allowing the user to continue using the terminal. Command-line Arguments Explained: - `-b`: Runs `wget` in the background. Output is logged to a file. ## Limit download speed Example Usage: `wget --limit-rate=100k http://example.com/large_file.zip` What it does: Restricts the download speed to the specified value (e.g., 100 kilobytes per second). Command-line Arguments Explained: - `--limit-rate=100k`: Limits the download speed to 100 kilobytes per second. ## Resume an interrupted download Example Usage: `wget -c http://example.com/partial_file.zip` What it does: Resumes a partially downloaded file from the last point of interruption. Command-line Arguments Explained: - `-c`: Enables resuming of an interrupted download. ## Download only if the file is newer Example Usage: `wget -N http://example.com/updated_file.txt` What it does: Downloads the file only if it is newer than the local version, or if the local file does not exist. Command-line Arguments Explained: - `-N`: Enables time-stamping, checking if the remote file is newer than the local copy. ## Download with authentication Example Usage: `wget --user=username --password=secret http://example.com/protected_file.txt` What it does: Downloads a file that requires HTTP authentication credentials. Command-line Arguments Explained: - `--user`: Specifies the username for authentication. - `--password`: Specifies the password for authentication. ## Use a specific user agent Example Usage: `wget --user-agent="Mozilla/5.0" http://example.com` What it does: Sets a custom user agent string to mimic a browser or bypass restrictions based on user agent detection. Command-line Arguments Explained: - `--user-agent`: Defines the user agent string sent in the HTTP request header. ## Download a file with a timeout Example Usage: `wget --timeout=30 http://example.com/really_slow_file.zip` What it does: Sets a timeout for the connection, terminating the download if no response is received within the specified seconds. Command-line Arguments Explained: - `--timeout=30`: Specifies the connection timeout in seconds (e.g., 30 seconds). ## Mirror a website (recursively and retain directory structure) Example Usage: `wget --mirror http://example.com` What it does: Mirrors the website, including all subpages and maintaining the original directory structure. Command-line Arguments Explained: - `--mirror`: A shortcut for `-r --no-parent --convert-links`, which recursively downloads and preserves the site's structure. ## Download from a secure (HTTPS) site without certificate verification Example Usage: `wget --no-check-certificate https://example.com/secured_file.txt` What it does: Downloads a file from an HTTPS site without verifying the SSL certificate, useful for self-signed certificates. Command-line Arguments Explained: - `--no-check-certificate`: Skips SSL certificate verification. ## Download multiple files with a single command Example Usage: `wget http://example.com/file1.txt http://example.com/file2.txt` What it does: Downloads multiple files by specifying their URLs in a single command. Command-line Arguments Explained: - `http://example.com/file1.txt http://example.com/file2.txt`: URLs of the files to be downloaded sequentially. ## Output to a specific directory Example Usage: `wget -P /path/to/dir http://example.com/file.txt` What it does: Saves the downloaded file to the specified directory instead of the current working directory. Command-line Arguments Explained: - `-P`: Specifies the directory where the file should be saved. ## Download a file with a custom header Example Usage: `wget --header="Authorization: Bearer token" http://example.com/api/data` What it does: Attaches a custom HTTP header (e.g., for authentication) to the download request. Command-line Arguments Explained: - `--header="Authorization: Bearer token"`: Adds the specified HTTP header to the request. ## Specify the proxy server Example Usage: `wget --proxy=on http://example.com/file.txt` What it does: Uses the specified proxy server for the download, with proxy settings read from environment variables or the default configuration. Command-line Arguments Explained: - `--proxy=on`: Enables the use of a proxy server. Proxies are typically set via environment variables like `HTTP_PROXY` or `HTTPS_PROXY`. ## Download files with a retry limit Example Usage: `wget --tries=5 http://example.com/retry_file.txt` What it does: Retries the download up to the specified number of times if it fails. Command-line Arguments Explained: - `--tries=5`: Sets the maximum number of retry attempts for failed downloads. ## Download files with a specific output document location Example Usage: `wget --output-document=/path/to/save/file.txt http://example.com/remote_file.txt` What it does: Saves the downloaded file to a specific location, overriding the default filename. Command-line Arguments Explained: - `--output-document`: Specifies the exact path and filename for the saved file. ## Download a file using the POST method Example Usage: `wget --post-data="key=value" http://example.com/post_endpoint` What it does: Sends a POST request with the given data to the specified URL. Command-line Arguments Explained: - `--post-data="key=value"`: Sends the specified data as the body of the POST request. ## Skip parent directories during recursion Example Usage: `wget -r --no-parent http://example.com` What it does: Recursively downloads a website but restricts the process to the exact directory specified, not descending into parent directories. Command-line Arguments Explained: - `-r`: Enables recursive downloading. - `--no-parent`: Prevents `wget` from accessing parent directories of the specified URL.