Download all files from url linux

How to Download a File on Ubuntu Linux using the Command Line

Linux Command line offers more flexibility and control than GUI. A number of people prefer to use the command line than GUI because it is easier and quicker to use than GUI. Using the command line, it is easier to automate the tasks using one line. In addition, it utilizes fewer resources than GUI.

Downloading files is a routine task that is normally performed every day that can include file types like ZIP, TAR, ISO, PNG, etc. you can simply and quickly perform this task using the command line terminal. It requires only using your keyboard. So today, I will show you how you can download a file using the command line in Linux. There are normally two known ways to do this, that is using wget and curl utility. For this article, I am using Ubuntu 20.04 LTS for describing the procedure. But the same commands will work on other Linux distributions like Debian, Gentoo, and CentOS too.

Download files using Curl

Curl can be used to transfer data over a number of protocols. It supports many protocols including HTTP, HTTPS, FTP, TFTP, TELNET, SCP, etc. using Curl, you can download any remote files. It supports pause and resumes functions as well.

To get started with, first, you need to install the curl.

Install curl

Launch command line application in Ubuntu that is Terminal by pressing the Ctrl+Alt+T key combinations. Then enter the below command to install curl with sudo.

When prompted for a password, enter sudo password.

Once the installation is complete, enter the below command to download a file.

Download and save the file using the source file name

To save the file with the same name as the original source file on the remote server, use –O (uppercase O) followed by curl as below:

Instead of -O, you can also specify, “–remote-name” as shown below. Both work the same.

Download and save the file with a different name

If you want to download the file and save it in a different name than the name of the file in the remote server, use -o (lower-case o) as shown below. This is helpful when the remote URL doesn’t contain the file name in the URL as shown in the example below.

[filename] is the new name of the output file.

Download multiple files

To download multiple files, enter the command in the following syntax:

Download files from an FTP Server

To download a file from FTP server, enter the command in following syntax:

To download files from user authenticated FTP servers, use the following syntax:

Pause and resume download

While downloading a file, you can manually pause it using Ctrl+C or sometimes it automatically gets interrupted and stopped due to any reason, you can resume it. Navigate to the same directory where you have previously downloaded the file then enter the command in the following syntax:

Download files using Wget

Using wget, you can download files and contents from Web and FTP servers. Wget is a combination of www and the get. It supports protocols like FTP, SFTP, HTTP, and HTTPS. Also it supports recursive download feature. This feature is very useful if you want to download an entire website for offline viewing or for generating a backup of a static website. In addition, you can use it to retrieve content and files from various web servers.

Install wget

Launch command line application in Ubuntu that is terminal by pressing the Ctrl+Alt+T key combinations. Then enter the below command to install wget with sudo.

When prompted for a password, enter the sudo password.

Download file or webpage using wget

To download a file or a webpage, open the Terminal and enter the command in the following syntax:

To save a single webpage, enter the command in the following syntax:

Download files with a different name

If you want to download and save the file with a different name than the name of the original remote file, use -O (upper-case O) as shown below. This is helpful especially when you are downloading a webpage that automatically get saved with the name “index.html”.

To download a file with a different name, enter the command in the following syntax:

Download files through FTP

To download a file from an FTP server, type the command in the following syntax:

To download files from user authenticated FTP servers, use the below syntax:

Recursively download files

You can use the recursive download feature to download everything under the specified directory whether a website or an FTP site. To use the recursive download feature, enter the command in the below syntax:

Download multiple files

You can use wget to download multiple files. Make a text file with a list of file URLs, then use the wget command in the following syntax to download that list.

For instance, I have the text file named “downloads.txt” in which there is a list of two URLs that I want to download using wget. You can see my text file content in the below image:

I will use the below command to download the file links contained in the text file:

You can see that it is downloading both links one by one.

Pause and Resume download

You can Press Ctrl + C to pause a download. To resume a paused download, go to the same directory where you were downloading the file previously and use –c option after wget as in the below syntax:

Using the above command, you will notice that your download has resumed from where it was paused.

So in this article, we have discussed the basic usage of two command-line methods using which you can download a file. One thing to Note that if you do not specify a directory while downloading a file, the files will be downloaded in the current directory in which you are working.

Karim Buzdar

About the Author: Karim Buzdar holds a degree in telecommunication engineering and holds several sysadmin certifications. As an IT engineer and technical author, he writes for various web sites. You can reach Karim on LinkedIn

Источник

How to Download Files on Debian using curl and wget on the Command Line

Working in a Linux command line gives you more flexibility and control as compared to GUI. Command-line has many uses and is extensively used in server administration. You can automate the task using the command line and also it utilizes fewer resources than GUI. Downloading a file using the command line is also easier and quicker as it requires only a single command as compared to GUI which mostly requires long steps procedure.

In this article, we will explain how to download a file using the Linux command line using two different utilities. Both are free utilities for non-interactive download of files from web. These utilities working in the background even when you are not logged in.

We will use Debian 10 for describing the procedure mentioned in this article.

Method #1 Download files using Curl

Curl is a command-line utility that is used to transfer files to and from the server. We can use it for downloading files from the web. It is designed in such a way that you can run it without a user interaction. It supports various protocols including HTTP, HTTPS, TELNET, SCP, FTP, etc. It is not by default installed in Debian OS. Therefore, we have to install it first. To do so, follow the below steps:

Install Curl

Launch the Terminal application in Debian. For that, go to the Activities tab in the top left corner of the desktop. Then in the search bar, type terminal. When the Terminal icon appears, click on it to launch it.

In the Terminal, type the following command to switch to the superuser account.

When prompted for the password, enter superuser password.

Then run the following command in Terminal to install Curl utility.

Once the installation is completed, we can use the Curl for downloading files.

General syntax of CURL :

To download files using Curl, use the following syntax in Terminal:

Using the [options] parameter, you can specify various functions, for instance, save the download with a specific name, resume a download, specify transfer rate and much more.

Using the [URL] parameter, you can specify the URL of the remote server.

Download and save the file using the source file name

To download and save the file with the same name as the source file name, use the following syntax:

An example of this would be:

It will save the downloaded file as debian-10.0.0-amd64-DVD-1.iso.

Alternatively, you can also specify, “–remote-name” instead of –O to save the file as the remote file name.

Download and save the file with a different name

To download and save the file with the different name as the source file name, use the following syntax:

In the [filename] parameter, specify a new name for the downloaded file.

An example of this would be:

It will save the downloaded file as debian.iso.

Download multiple files simultaneously

Instead of downloading multiple files one by one, you can download all of them simultaneously by running a single command. To download multiple files at the same time, use –O followed by the URL to the file that you wish to download.

Use the following syntax for this purpose:

An example of this would be:

The above command will download both files.

There is an alternative way of doing this. Specify the list of URLs in a file, then use the Curl command along with xargs in the following syntax:

An example of this would be:

Our files.txt file contains two URLs:

The above Curl command will download all the URLs specified in the files.txt file.

Download files from an FTP Server

We can also download files from FTP server using the Curl utility. To do so, run the command in Terminal using the following syntax:

ftp_user and ftp_pass parameters are used to specify FTP login credential. However, you can skip these in case of anonymous FTP connection.

Pause and resume download

You can also resume a download that has been paused manually or due to some other reason. To manually pause a download, use Ctrl+C.

To resume a paused download, navigate to the directory where you have previously downloaded the file, then use the following syntax to resume it.

An example of this would be:

To resume a paused downloaded file debian-10.0.0-amd64-DVD-1.iso file, we have used this command:

From the following output, you can see that it has resumed the download.

Download files using Wget

Similar to Curl, there is another command-line utility Wget that can be used to download files and content from the web. Wget is a combination of the World Wide Web and the word get. It supports protocols like FTP, SFTP, HTTP, and HTTPS. Also, it supports recursive downloading that is very useful if you want to download a whole website for offline viewing or for creating a backup for static website.

Install Wget

If wget is not already installed on your system, you can install it by following the below steps:

Launch the Terminal application in the same way as discussed earlier in this article. In the Terminal, type the following command to switch to the super user account.

When prompted for the password, enter superuser password.

Then run the following command in Terminal to install Wget utility.

General syntax of Wget

To download a file using Wget, use the following syntax:

Download and save the file using the source file name

Using the above syntax for downloading a file without any argument will save the file with the same name as the source file. An example of this would be to download a debian-10.0.0-amd64-DVD-1.iso file.

It will save the download as a debian-10.0.0-amd64-DVD-1.iso.

Download and save the file with a different name

To download and save the file with the different name as the source file name, use the following syntax:

It will save the download as a debian10.

Download files through FTP

To download files from user authenticated FTP servers, use the below syntax:

ftp_user and ftp_pass parameter are used to specify FTP login credential. However, you can skip these in case of anonymous FTP connection.

Download multiple files

To download multiple files using Wget, create a text file with a list of files URLs and then use the below syntax to download all files at simultaneously.

For instance, we have created a text file files.txt that contains two URLs as shown in the image below.

Then we have run the following command:

Running the above command will automatically download both the URLs contained in the files.txt.

Pause and Resume download

To resume a paused download, navigate to the directory where you have previously downloaded the file, then use the following syntax to resume it.

An example of this would be to resume a previously paused debian-10.0.0-amd64-DVD-1.iso file by running the following command.

Recursively download files

Wget supports recursive downloading that is a major feature that differs it from Curl. Recursive download feature allows downloading of everything under a specified directory.

To download a website or FTP site recursively, use the following syntax:

An example of this would be to download the following entire site.

So, in this article, we have learned two different non-interactive command line utilities that let you download files directly from the command line. Both utilities comes in handy and servers a similar purpose. I hope it will be helpful whenever you need to download a file from the internet.

Karim Buzdar

About the Author: Karim Buzdar holds a degree in telecommunication engineering and holds several sysadmin certifications. As an IT engineer and technical author, he writes for various web sites. You can reach Karim on LinkedIn

Источник