Linux curl download file

Downloading files with curl

How to download files straight from the command-line interface

The curl tool lets us fetch a given URL from the command-line. Sometimes we want to save a web file to our own computer. Other times we might pipe it directly into another program. Either way, curl has us covered.

This is the basic usage of curl :

That —output flag denotes the filename ( some.file ) of the downloaded URL ( http://some.url )

Let’s try it with a basic website address:

Besides the display of a progress indicator (which I explain below), you don’t have much indication of what curl actually downloaded. So let’s confirm that a file named my.file was actually downloaded.

Using the ls command will show the contents of the directory:

And if you use cat to output the contents of my.file , like so:

– you will the HTML that powers http://example.com

I thought Unix was supposed to be quiet?

Let’s back up a bit: when you first ran the curl command, you might have seen a quick blip of a progress indicator:

If you remember the Basics of the Unix Philosophy, one of the tenets is:

Rule of Silence: When a program has nothing surprising to say, it should say nothing.

In the example of curl , the author apparently believes that it’s important to tell the user the progress of the download. For a very small file, that status display is not terribly helpful. Let’s try it with a bigger file (this is the baby names file from the Social Security Administration) to see how the progress indicator animates:

Quick note: If you’re new to the command-line, you’re probably used to commands executing every time you hit Enter. In this case, the command is so long (because of the URL) that I broke it down into two lines with the use of the backslash, i.e. \

This is solely to make it easier for you to read. As far as the computer cares, it just joins the two lines together as if that backslash weren’t there and runs it as one command.

Make curl silent

The curl progress indicator is a nice affordance, but let’s just see if we get curl to act like all of our Unix tools. In curl ‘s documentation of options, there is an option for silence:

Silent or quiet mode. Don’t show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.

Repeat and break things

So those are the basics for the curl command. There are many, many more options, but for now, we know how to use curl to do something that is actually quite powerful: fetch a file, anywhere on the Internet, from the simple confines of our command-line.

Before we go further, though, let’s look at the various ways this simple command can be re-written and, more crucially, screwed up:

Shortened options

As you might have noticed in the —silent documentation, it lists the alternative form of -s . Many options for many tools have a shortened alias. In fact, —output can be shortened to -o

Now watch out: the number of hyphens is not something you can mess up on; the following commands would cause an error or other unexpected behavior:

Also, mind the position of my.file , which can be thought of as the argument to the -o option. The argument must follow after the -o …because curl .

If you instead executed this:

How would curl know that my.file , and not -s is the argument, i.e. what you want to name the content of the downloaded URL?

In fact, you might see that you’ve created a file named -s …which is not the end of the world, but not something you want to happen unwittingly.

Order of options

By and large (from what I can think of at the top of my head), the order of the options doesn’t matter:

In fact, the URL, http://example.com , can be placed anywhere in the mix:

A couple of things to note:

The way that the URL, what you might consider the main argument for the curl command, can be placed anywhere after the command is not the way that all commands have been designed. So it always pays to read the documentation with every new command.

Notice how -s http://example.com doesn’t cause a problem. That’s because the -s option doesn’t take an argument. But try the following:

And you will have a problem.

No options at all

The last thing to consider is what happens when you just curl for a URL with no options (which, after all, should be optional). Before you try it, think about another part of the Unix philosophy:

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

If you curl without any options except for the URL, the content of the URL (whether it’s a webpage, or a binary file, such as an image or a zip file) will be printed out to screen. Try it:

Even with the small amount of HTML code that makes up the http://example.com webpage, it’s too much for human eyes to process (and reading raw HTML wasn’t meant for humans).

Standard output and connecting programs

But what if we wanted to send the contents of a web file to another program? Maybe to wc , which is used to count words and lines? Then we can use the powerful Unix feature of pipes. In this example, I’m using curl ‘s silent option so that only the output of wc (and not the progress indicator) is seen. Also, I’m using the -l option for wc to just get the number of lines in the HTML for example.com:

Number of lines in example.com is: 50

Now, you could’ve also done the same in two lines:

Источник

Linux / Unix curl Command Download File Example

curl download file

The syntax is as follows to grab (download) files from remote http/ftp server:
curl -o output.file http://server1.cyberciti.biz/file.tar.gz
curl -O http://server1.cyberciti.biz/file.tar.gz
curl —remote-name http://server1.cyberciti.biz/file.tar.gz
You can download a web page and store in a local file as follows:
curl -o nixcraft.html https://www.cyberciti.biz/low.html
You can grab or download multiple files as follows:
curl -O http://www.cyberciti.biz/low.html -O http://bash.cyberciti.biz/dl/581.sh.zip

curl download file from an ssh server

You can grab file securely using from an SSH server using SFTP:
curl -u username sftp://server1.cyberciti.biz/path/to/file.txt
##

means your $HOME dir ##
curl -u vivek sftp://home1.cyberciti.biz/

/docs/resume.pdf
You can grab a file from an SSH server using SCP using a private key to authenticate. The syntax is:
curl -u username: —key

• -u username – Specify the user name (and optional password) to use for server authentication.
• -u username:password – Specify the user name (and optional password) to use for server authentication.
• –key

/.ssh/id_rsa – SSL or SSH private key file name. Allows you to provide your private key in this separate file.
–pubkey

/.ssh/id_rsa.pub – SSH Public key file name. Allows you to provide your public key in this separate file.
scp://home1.cyberciti.biz/

/Videos/rhn_register.ogv – Use scp protocol and download file from my home server called home1.cyberciti.biz.

Curl: Download a file using username and password

The syntax is as follows to grab a file using ftp username and password:
curl ftp://username:passwd@ftp1.cyberciti.biz:21/path/to/backup.tar.gz
OR
curl -u UserName:PassWord ftp://ftp1.cyberciti.biz:21/backups/07/07/2012/mysql.blog.sql.tar.gz
Secure ftp user (ftp with ssl) can pass the –ftp-ssl option to curl command:
curl —ftp-ssl -u UserName:PassWord ftp://ftp1.cyberciti.biz:21/backups/07/07/2012/mysql.blog.sql.tar.gz

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

HTTP authentication and downloading files with curl

HTTP user can use the following syntax:
curl http://username:passwd@server1.cyberciti.biz/file/path/data.tar.gz
OR
curl -u Username:Password http://server1.cyberciti.biz/file/path/data.tar.gz

Check out related media:

See also:

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Источник

How to download a file with curl on Linux/Unix command line

How to download a file with curl command

The basic syntax:

1. Grab files with curl run: curl https://your-domain/file.pdf
2. Get files using ftp or sftp protocol: curl ftp://ftp-your-domain-name/file.tar.gz
3. You can set the output file name while downloading file with the curl, execute: curl -o file.pdf https://your-domain-name/long-file-name.pdf
4. Follow a 301-redirected file while downloading files with curl, run: curl -L -o file.tgz http://www.cyberciti.biz/long.file.name.tgz

Let us see some examples and usage about the curl to download and upload files on Linux or Unix-like systems.

Installing curl on Linux or Unix

By default curl is installed on many Linux distros and Unix-like systems. But, we can install it as follows:
## Debian/Ubuntu Linux use the apt command/apt-get command ##
$ sudo apt install curl
## Fedora/CentOS/RHEL users try dnf command/yum command ##
$ sudo dnf install curl
## OpenSUSE Linux users try zypper command ##
$ sudo zypper install curl

Verify installation by displaying curl version

Type:
$ curl —version
We see:

Downloading files with curl

The command syntax is:
curl url —output filename
curl https://url -o output.file.name
Let us try to download a file from https://www.cyberciti.biz/files/sticker/sticker_book.pdf and save it as output.pdf
curl https://www.cyberciti.biz/files/sticker/sticker_book.pdf -o output.pdf
OR
curl https://www.cyberciti.biz/files/sticker/sticker_book.pdf —output output.pdf

The -o or —output option allows you to give the downloaded file a different name. If you do not provide the output file name curl will display it to the screen. Let us say you type:
curl —output file.html https://www.cyberciti.biz
We will see progress meter as follows:

The outputs indicates useful information such as:

• % Total : Total size of the whole expected transfer (if known)
• % Received : Currently downloaded number of bytes
• % Xferd : Currently uploaded number of bytes
• Average Dload : Average transfer speed of the entire download so far, in number of bytes per second
• Speed Upload : Average transfer speed of the entire upload so far, in number of bytes per second
• Time Total : Expected time to complete the operation, in HH:MM:SS notation for hours, minutes and seconds
• Time Spent : Time passed since the start of the transfer, in HH:MM:SS notation for hours, minutes and seconds
• Time Left : Expected time left to completion, in HH:MM:SS notation for hours, minutes and seconds
• Current Speed : Average transfer speed over the last 5 seconds (the first 5 seconds of a transfer is based on less time, of course) in number of bytes per second

Resuming interrupted downloads with curl

Pass the -C — to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out:
## Restarting an interrupted download is important task too ##
curl -C — —output bigfilename https://url/file

How to get a single file without giving output name

You can save output file as it is i.e. write output to a local file named like the remote file we get. For example, sticker_book.pdf is a file name for remote URL https://www.cyberciti.biz/files/sticker/sticker_book.pdf. One can save it sticker_book.pdf directly without specifying the -o or —output option by passing the -O (capital
curl -O https://www.cyberciti.biz/files/sticker/sticker_book.pdf

Downloading files with curl in a single shot

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

Dealing with HTTP 301 redirected file

The remote HTTP server might send a different location status code when downloading files. For example, HTTP URLs are often redirected to HTTPS URLs with HTTP/301 status code. Just pass the -L follow the 301 (3xx) redirects and get the final file on your system:
curl -L -O http ://www.cyberciti.biz/files/sticker/sticker_book.pdf

Downloading multiple files or URLs using curl

Try:
curl -O url1 -O url2
curl -O https://www.cyberciti.biz/files/adduser.txt \
-O https://www.cyberciti.biz/files/test-lwp.pl.txt
One can use the bash for loop too:

How to download a file using curl and bash for loop

Grab a password protected file with curl

Try any one of the following syntax
curl ftp://username:passwd@ftp1.cyberciti.biz:21/path/to/backup.tar.gz
curl —ftp-ssl -u UserName:PassWord ftp://ftp1.cyberciti.biz:21/backups/07/07/2012/mysql.blog.sql.tar.gz
curl https://username:passwd@server1.cyberciti.biz/file/path/data.tar.gz
curl -u Username:Password https://server1.cyberciti.biz/file/path/data.tar.gz

Downloading file using a proxy server

Again syntax is as follows:
curl -x proxy-server-ip:PORT -O url
curl -x ‘http://vivek:YourPasswordHere@10.12.249.194:3128’ -v -O https://dl.cyberciti.biz/pdfdownloads/b8bf71be9da19d3feeee27a0a6960cb3/569b7f08/cms/631.pdf

How to use curl command with proxy username/password

Examples

curl command can provide useful information, especially HTTP headers. Hence, one can use such information for debugging server issues. Let us see some examples of curl commands. Pass the -v for viewing the complete request send and response received from the web server.
curl -v url
curl -o output.pdf -v https://www.cyberciti.biz/files/sticker/sticker_book.pdf

Getting HTTP headers information without downloading files

Another useful option is to fetch HTTP headers. All HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document. For instance, when you want to view the HTTP response headers only without downloading the data or actual files:
curl -I url
curl -I https://www.cyberciti.biz/files/sticker/sticker_book.pdf -o output.pdf

Getting header information for given URL

How do I skip SSL skip when using curl?

If the remote server has a self-signed certificate you may want to skip the SSL checks. Therefore, pass pass the -k option as follows:
curl -k url
curl -k https://www.cyberciti.biz/

Rate limiting download/upload speed

You can specify the maximum transfer rate you want the curl to use for both downloads and uploads files. This feature is handy if you have a limited Internet bandwidth and you would like your transfer not to use your entire bandwidth. The given speed is measured in bytes/second, unless a suffix is appended. Appending ‘k’ or ‘K’ will count the number as kilobytes, ‘m’ or ‘M’ makes it megabytes, while ‘g’ or ‘G’ makes it gigabytes. For Examples: 200K, 3m and 1G:
curl —limit-rate url
curl —limit-rate 200 https://www.cyberciti.biz/
curl —limit-rate 3m https://www.cyberciti.biz/

Setting up user agent

Some web application firewall will block the default curl user agent while downloading files. To avoid such problems pass the -A option that allows you to set the user agent.
curl -A ‘user agent name’ url
curl -A ‘Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0’ https://google.com/

Upload files with CURL

The syntax is as follows to upload files:
curl -F «var=@path/to/local/file.pdf» https://url/upload.php
For example, you can upload a file at

/Pictures/test.png to the server https://127.0.0.1/app/upload.php which processes file input with form parameter named img_file, run:
curl -F «img_file=@

/Pictures/test.png» https://127.0.0.1/app/upload.php
One can upload multiple files as follows:
curl -F «img_file1=@

/Pictures/test-1.png» \
-F «img_file2=@

Make curl silent

Want to make hide progress meter or error messages? Try passing the -s or —slient option to turn on curl quiet mode:
curl -s url
curl —silent —output filename https://url/foo.tar.gz

Conclusion

Like most Linux or Unix CLI utilities, you can learn much more about curl command by visiting this help page.

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Источник