Linux copy files with progress

Rsync Show Progress Bar While Copying Files

rsync is free and open source tool. It is useful to copy local or remote file. It reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. You can use any one of the following options to add a total progress indicator when copying files from serverA to serverB or vice versa.

Tutorial details
Difficulty level	Easy
Root privileges	No
Requirements	rsync command on Linux or Unix
Est. reading time	3 minutes

1. rsync command to with —progress option.
2. pv command – monitor the progress of data or data transfer through a pipe. This is a recommend option for most users.

rsync command to show progress bar

You need to use the —progress or -P option which show progress during file transfer. The syntax is as follows:

Examples: Rsync show progress bar in Linux

To copy file from /foo/*.tar.gz to a remote server called server1.cyberciti.biz, enter:

In this example, copy files from a remote server:

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

The —info=progress2 option shows statistics based on the whole transfer, rather than individual files. Use this flag without outputting a filename (e.g. avoid -v or specify –info=name0 if you want to see how the transfer is doing without scrolling the screen with a lot of names.

In this example copy from local computer to remote server:
$ rsync -v —progress debian-9-openstack-amd64.qcow2 vivek@centos7:

Use pv command to monitor progress of rsync command

1. Time elapsed
2. Percentage completed (with progress bar)
3. Current throughput rate
4. Total data transferred
5. ETA

So if you have 42 files in /tmp/software and you would like to copy them to /nas10, enter:

Animated gif 01: rsync and pv command demo

Conclusion

You learned about displaying the progress bar when copying files with the rsync command on Linux, macOS, or Unix-like systems. See pv and rsync command man pages by typing the following commands:
man pv
man rsync

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Category	List of Unix and Linux commands
Documentation	help • mandb • man • pinfo
Disk space analyzers	df • duf • ncdu • pydf
File Management	cat • cp • less • mkdir • more • tree
Firewall	Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04
Linux Desktop Apps	Skype • Spotify • VLC 3
Modern utilities	bat • exa
Network Utilities	NetHogs • dig • host • ip • nmap
OpenVPN	CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04
Package Manager	apk • apt
Processes Management	bg • chroot • cron • disown • fg • glances • gtop • jobs • killall • kill • pidof • pstree • pwdx • time • vtop
Searching	ag • grep • whereis • which
Shell builtins	compgen • echo • printf
Text processing	cut • rev
User Information	groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w
WireGuard VPN	Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04

Comments on this entry are closed.

Also add -h to show human readable sizes (350M instead of 367071842).

um… That’s not a progress bar. That’s progress output.

—progress is what you are looking for. (incase anyone gets this in a google search like i did)

Thanks for the informative post.

If you are running an “update” to a large folder you may not know how many files will be updated in advance, thus making it difficult to supply the file count to PV without doing an rsync –dry-run first. You can capture the “update” file count into a variable like this:

Then you can call rsync and pv with a reference to the file count variable:

A few items to note:

• The file count variable ($FCNT) has an extra five files added. Why is that? The rsync output from your file xfer will have four lines at the end of the files transferred list as a summary; pv will count those four lines as 5 lines. By adding the 5 lines in advance we’ll prevent our progress bar from going over %100 (by much).
• The pv parameters I used are slightly different. I like pv -pteabl because it gives me a files-sent-count, time-elapsed, and average throughput on the progress bar.
• Using the pv -l argument converts the -b parameter from bytes sent to a file count, and converts -s from size of bytes sent to number of lines processed.
• Your second call to rsync (the one that sends the files) needs to have the -v parameter to list one line per file processed otherwise the pv command will stay at 0%. Similarly if you use -vv or -vvv on the rsync call then pv will receive a much larger number of “files” than predicted.
• Using the rsync -v switch will cause rsync to list folders it is entering as another line, i.e. another file processed from the pv perspective. In short, if you are processing more than one sub folder with your rsync command the file count passed to pv will be slightly inaccurate.

–info=progress2 seems to be available only on 3.1.0+
I was using 3.0.9 and it complained no such option exists

Источник

Advanced Copy Command – Shows Progress Bar While Copying Large Files/Folders in Linux

Advanced-Copy is a powerful command line program which is very much similar, but little modified version of original cp command. This modified version of cp command adds a progress bar along with total time taken to complete, while copying large files from one location to another. This additional feature is very useful especially while copying large files, and this gives an idea to user about the status of copy process and how long it takes to complete.

Download and Install Advanced-Copy

There are two methods to install Advanced-Copy utility in Linux systems, either you compile from sources or using pre-compiled binaries. Installing from pre-compiled binaries should always work correctly and requires lesser experience and very effective for Linux newbies.

But I suggest you to compile from sources, for this you required original version of GNU coreutils and latest patchfile of Advacned-Copy. The whole installation should go like this:

Compiling from Sources

First, download the latest version of GNU coreutils and patchfile using wget command and compile and patch it as shown below, you must be root user to perform all commands.

You might get the following error, while running “./configure” command.

Run the following command on the terminal to fix that error and run the “./configure” command again.

Once, compilation completes, two new commands are created under src/cp and src/mv. You need to replace your original cp and mv commands with these two new commands to get the progress bar while copying files.

Note: If you don’t want to copy these commands under standard system paths, you can still run them from source directory like “./cp” and “./mv or create new commands as shown”.

Automatic progress bar

If you want the progress bar to be appear all the time while copying, you need to add the following lines to your

/.bashrc file. Save and close the file

You need to logout and login again to get this work correctly.

How to Use Advacned-Copy Command

The command is same, the only change is adding “-g” or “–progress-bar” option with cp command. The “-R” option is for copying directories recursively. Here is an example screen-shots of a copy process using advanced copy command.

Here is the example of ‘mv‘ command with screen-shot.

Please remember, original commands are not overwritten, if you ever need to use them or you’re not happy with the new progress bar, and want to revert back to original cp and mv commands. You can call them via /usr/bin/cp or /usr/bin/mv.

I really much impressed with this new progress bar feature, at least I would know some information of copy operation time and exactly what’s going on.

Overall I can say, it is really good tool to have in your pocket, especially when you are spending lots of time in copying and moving files through command line.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

We are thankful for your never ending support.

Источник

How can I make a progress bar while copying a directory with cp?

I suppose I could compare the number of files in the source directory to the number of files in the target directory as cp progresses, or perhaps do it with folder size instead? I tried to find examples, but all bash progress bars seem to be written for copying single files. I want to copy a bunch of files (or a directory, if the former is not possible).

10 Answers 10

You can also use rsync instead of cp like this:

rsync -Pa source destination

Which will give you a progress bar and estimated time of completion. Very handy.

To show a progress bar while doing a recursive copy of files & folders & subfolders (including links and file attributes), you can use gcp (easily installed in Ubuntu and Debian by running «sudo apt-get install gcp»):

Here is the typical output while copying a large folder of files:

Notice that it shows just one progress bar for the whole operation, whereas if you want a single progress bar per file, you can use rsync :

You may have a look at the tool vcp . Thats a simple copy tool with two progress bars: One for the current file, and one for overall.

EDIT

Here is the link to the sources: http://members.iinet.net.au/

Most distributions have a package for it.

lynx/vcp link is broken. There seems to be a current fork at github.com/gdm85/curses-vcp with a last commit of May 29, 2016. I was able to compile it just fine on Fedora 23. An independent vcp with a similar concept is at github.com/lynix/vcp but I have not tried it.

Here another solution: Use the tool bar

You could invoke it like this:

You have to go the way over tar, and it will be inaccurate on small files. Also you must take care that the target directory exists. But it is a way.

My preferred option is Advanced Copy, as it uses the original cp source files.

The new programs are now located in src/cp and src/mv. You may choose to replace your existing commands:

Then you can use cp as usual, or specify -g to show the progress bar:

A simple unix way is to go to the destination directory and do watch -n 5 du -s . Perhaps make it more pretty by showing as a bar . This can help in environments where you have just the standard unix utils and no scope of installing additional files . du-sh is the key , watch is to just do every 5 seconds. Pros : Works on any unix system Cons : No Progress Bar

To add another option, you can use cpv . It uses pv to imitate the usage of cp .

It works like pv but you can use it to recursively copy directories

There’s a ubuntu version in apt

How about something like

It finds all the files in the current directory, pipes that through PV while giving PV an estimated size so the progress meter works and then piping that to a CP command with the —parents flag so the DEST path matches the SRC path.

One problem I have yet to overcome is that if you issue this command

the destination path becomes /www/test/home/user/test/. FILES. and I am unsure how to tell the command to get rid of the ‘/home/user/test’ part. That why I have to run it from inside the SRC directory.

Check the source code for progress_bar in the below git repository of mine

Also try custom bash script package supreme to verify how progress bar work with cp and mv comands

(1)Open Apps —-Firefox —-Calculator —-Settings

(2)Manage Files —-Search —-Navigate —-Quick access

(3)Manage Phone —-Move/Copy from phone —-Move/Copy to phone —-Sync folders

(4)Manage USB —-Move/Copy from USB —-Move/Copy to USB

Источник