How To Use Wget To Download Files And Interact With REST APIs

Downloading Files

In this section, you will use Wget to customize your download experience. For example, you will learn to download a single file and multiple files, handle file downloads in unstable network conditions, and, in the case of a download interruption, resume a download. If you’re evaluating curl for similar tasks, see the companion guide, Workflow: Downloading Files with curl.

First, create a directory to save the files that you will download throughout this tutorial:

1. mkdir -p DigitalOcean-Wget-Tutorial/Downloads

With the command above, you have created a directory named DigitalOcean-Wget-Tutorial, and inside of it, you created a subdirectory named Downloads. This directory and its subdirectory will be where you will store the files you download.

Navigate to the DigitalOcean-Wget-Tutorial directory:

1. cd DigitalOcean-Wget-Tutorial

You have successfully created the directory where you will store the files you download.

Downloading a file

In order to download a file using Wget, type wget followed by the URL of the file that you wish to download. Wget will download the file in the given URL and save it in the current directory.

Let’s download a minified version of jQuery using the following command:

wget https://code.jquery.com/jquery-3.6.0.min.js

Don’t worry if you don’t know what jQuery is – you could have downloaded any file available on the internet. All you need to know is that you successfully used Wget to download a file from the internet.

The output will look similar to this:

Output--2021-07-21 16:25:11-- https://code.jquery.com/jquery-3.6.0.min.js Resolving code.jquery.com (code.jquery.com)... 69.16.175.10, 69.16.175.42, 2001:4de0:ac18::1:a:1a, ... Connecting to code.jquery.com (code.jquery.com)|69.16.175.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 89501 (87K) [application/javascript] Saving to: ‘jquery-3.6.0.min.js’ jquery-3.6.0.min.js 100%[===================>] 87.40K 114KB/s in 0.8s 2021-07-21 16:25:13 (114 KB/s) - ‘jquery-3.6.0.min.js’ saved [89501/89501]

According to the output above, you have successfully downloaded and saved a file named jquery-3.6.0.min.js to your current directory.

You can check the contents of the current directory using the following command:

1. ls

The output will look similar to this:

OutputDownloads jquery-3.6.0.min.js jquery.min.js

So far, you have used wget to download files to the current directory. Next, you will download to a specific directory.

Downloading a file to a specific directory

When downloading a file, Wget stores it in the current directory by default. You can change that by using the -P option to specify the name of the directory where you want to save the file.

Download the jQuery file you downloaded previously, but this time save it in the Downloads subdirectory.

wget -P Downloads/ https://code.jquery.com/jquery-3.6.0.min.js

The output will look similar to this:

Output--2021-07-21 16:28:50-- https://code.jquery.com/jquery-3.6.0.min.js Resolving code.jquery.com (code.jquery.com)... 69.16.175.42, 69.16.175.10, 2001:4de0:ac18::1:a:2b, ... Connecting to code.jquery.com (code.jquery.com)|69.16.175.42|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 89501 (87K) [application/javascript] Saving to: ‘Downloads/jquery-3.6.0.min.js’ jquery-3.6.0.min.js 100%[==================================>] 87.40K 43.6KB/s in 2.0s 2021-07-21 16:28:53 (43.6 KB/s) - ‘Downloads/jquery-3.6.0.min.js’ saved [89501/89501]

Notice the last line where it says that the jquery-3.6.0.min.js file was saved in the Downloads directory.

If you use the ls Downloads command to list the contents of the Downloads directory, you will see the jQuery file there:

Run the ls command:

1. ls Downloads

The output will look similar to this:

Outputjquery-3.6.0.min.js

Turning Wget’s output off

By default, Wget outputs a lot of information to the terminal when you download a file. You can use the -q option to turn off all output.

Download the jQuery file, but this time without showing any output:

1. wget -q https://code.jquery.com/jquery-3.6.0.min.js

You won’t see any output, but if you use the ls command to list the contents of the current directory you will find a file named jquery-3.6.0.min.js.1:

1. ls

The output will look similar to this:

OutputDownloads jquery-3.6.0.min.js jquery-3.6.0.min.js.1 jquery.min.js

Before saving a file, Wget checks whether the file exists in the desired directory. If it does, Wget adds a number to the end of the file. If you ran the command above one more time, Wget would create a file named jquery-3.6.0.min.js.2. This number increases every time you download a file to a directory that already has a file with the same name.

You have successfully turned off Wget’s output, but now you can’t monitor the download progress. Let’s look at how to show the download progress bar.

Showing the download progress bar

Wget lets you show the download progress bar but hide any other output by using the -q option alongside the --show-progress option.

Download the jQuery file, but this time only show the download progress bar:

1. wget -q --show-progress https://code.jquery.com/jquery-3.6.0.min.js

The output will look similar to this:

Outputjquery-3.6.0.min.js.2 100%[================================================>] 87.40K 207KB/s in 0.4s

Use the ls command to check the contents of the current directory and you will find the file you have just downloaded with the name jquery-3.6.0.min.js.2

From this point forward you will be using the -q and --show-progress options in most of the subsequent Wget commands.

So far you have only downloaded a single file. Next, you will download multiple files.

Downloading multiple files

In order to download multiples files using Wget, you need to create a .txt file and insert the URLs of the files you wish to download. After inserting the URLs inside the file, use the wget command with the -i option followed by the name of the .txt file containing the URLs.

Create a file named images.txt:

1. nano images.txt

In images.txt, add the following URLs:

images.txt https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg https://cdn.pixabay.com/photo/2016/01/05/17/51/maltese-1123016__340.jpg https://cdn.pixabay.com/photo/2020/06/30/22/34/dog-5357794__340.jpg

The URLs link to three random images of dogs found on Pixabay. After you have added the URLs, save and close the file.

Now you will use the -i option alongside the -P,-q and --show-progress options that you learned earlier to download all three images to the Downloads directory:

1. wget -i images.txt -P Downloads/ -q --show-progress

The output will look similar to this:

Outputpuppy-1903313__340.jp 100%[=========================>] 26.44K 93.0KB/s in 0.3s maltese-1123016__340. 100%[=========================>] 50.81K --.-KB/s in 0.06s dog-5357794__340.jpg 100%[=========================>] 30.59K --.-KB/s in 0.07s

If you use the ls Downloads command to list the contents of the Downloads directory, you will find the names of the three images you have just downloaded:

1. ls Downloads

The output will look similar to this:

Outputdog-5357794__340.jpg jquery-3.6.0.min.js maltese-1123016__340.jpg puppy-1903313__340.jpg

Limiting download speed

So far, you have download files with the maximum available download speed. However, you might want to limit the download speed to preserve resources for other tasks. You can limit the download speed by using the --limit-rate option followed by the maximum speed allowed in kiloBits per second and the letter k.

Download the first image in the images.txt file with a speed of 15 kB/S to the Downloads directory:

1. wget --limit-rate 15k -P Downloads/ -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

The output will look similar to this:

Outputpuppy-1903313__340.jpg.1 100%[====================================================>] 26.44K 16.1KB/s in 1.6s

If you use the ls Downloads command to check the contents of the Downloads directory, you will see the file you have just downloaded with the name puppy-1903313__340.jpg.1.

When downloading a file that already exists, Wget creates a new file instead of overwriting the existing file. Next, you will overwrite a downloaded file.

Overwriting a downloaded file

You can overwrite a file you have downloaded by using the -O option alongside the name of the file. In the code below, you will first download the second image listed in the images.txt file to the current directory and then you will overwrite it.

First, download the second image to the current directory and set the name to image2.jpg:

1. wget -O image2.jpg -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

The output will look similar to this::

Outputimage2.jpg 100%[====================================================>] 26.44K --.-KB/s in 0.04s

If you use the ls command to check the contents of the current directory, you will see the file you have just downloaded with the name image2.jpg.

If you wish to overwrite this image2.jpg file, you can run the same command you ran earlier :

1. wget -O image2.jpg -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

You can run the command above as many times as you like and Wget will download the file and overwrite the existing one. If you run the command above without the -O option, Wget will create a new file each time you run it.

Resuming a download

Thus far, you have successfully downloaded multiple files without interruption. However, if the download was interrupted, you can resume it by using the -c option.

Run the following command to download a random image of a dog found on Pixabay. Note that in the command, you have set the maximum speed to 1 KB/S. Before the image finishes downloading, press Ctrl+C to cancel the download:

1. wget --limit-rate 1k -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

To resume the download, pass the -c option. Note that this will only work if you run this command in the same directory as the incomplete file:

1. wget -c --limit-rate 1k -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

Up until now, you have only downloaded files in the foreground. Next, you will download files in the background.

Downloading in the background

You can download files in the background by using the -b option.

Run the command below to download a random image of a dog from Pixabay in the background:

1. wget -b https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

When you download files in the background, Wget creates a file named wget-log in the current directory and redirects all output to this file. If you wish to watch the status of the download, you can use the following command:

1. tail -f wget-log

The output will look similar to this:

OutputResolving cdn.pixabay.com (cdn.pixabay.com)... 104.18.20.183, 104.18.21.183, 2606:4700::6812:14b7, ... Connecting to cdn.pixabay.com (cdn.pixabay.com)|104.18.20.183|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 33520 (33K) [image/jpeg] Saving to: ‘grass-3206938__340.jpg’ 0K .......... .......... .......... .. 100% 338K=0.1s 2021-07-20 23:49:52 (338 KB/s) - ‘grass-3206938__340.jpg’ saved [33520/33520]

Setting a timeout

Until this point, we have assumed that the server that you are trying to download files from is working properly. However, let’s assume that the server is not working properly. You can use Wget to first limit the amount of time that you wait for the server to respond and then limit the number of times that Wget tries to reach the server.

If you wish to download a file but you are unsure if the server is working properly, you can set a timeout by using the -T option followed by the time in seconds.

In the following command, you are setting the timeout to 5 seconds:

1. wget -T 5 -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

Setting maximum number of tries

You can also set how many times Wget attempts to download a file after being interrupted by passing the --tries option followed by the number of tries.

By running the command below, you are limiting the number of tries to 3:

1. wget --tries=3 -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

If you would like to try indefinitely you can pass inf alongside the --tries option:

1. wget --tries=inf -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

In this section, you used Wget to download a single file and multiple files, resume downloads, and handle network issues. In the next section, you will learn to interact with REST API endpoints.