Copy data from linux to linux

5 commands to copy file from one server to another in Linux or Unix

Table of Contents

In my last article I shared the steps to encrypt a file using gpg key in Linux. Now in this article I will share various commands and tools which you can use to securely copy file from one server to another in Linux. There are additionally other ways to transfer files which I cannot cover here for example you can also use HTTPS to upload and download files.

I may write another article with detail list of steps to use HTTPS and curl for secure file upload and download.If you wish to copy files between Windows and Linux then you can always use Samba but since here we are targeting file transfer between two Linux machines, I will not share any steps related to Samba configuration.

Some more articles on related topic you may be interested in

Using SFTP to copy file from one server to another

In computing, the SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream.

SFTP is easy to work with: You enter sftp along with the name of the remote system on the command line. You are prompted for the account password ; then you are dropped into SFTP with the connection open and waiting.

Once you give the password of deepak user on target node, you will get sftp shell

Next to copy file from host to client (i.e. upload a file from host to client)
Here I have a file ‘ pwd.txt ‘ on my host server under ‘ /home/deepak/pwd.txt ‘ which I wish to copy to my client’s current working directory

To copy a directory and all it’s content use (-r). Here /home/deepak/mydir is available on my host machine which I am copying to the connected client node under current working directory.

So the file was successfully uploaded. You can verify the same

Next copy a file from client node to your host server. I have a file ‘test_file’ on my client node under ‘/home/deepak/test_file’

Validate the same on your host server

You can get more supported options from the man page of sftp .

Using RSYNC to copy file from one server to another

rsync is a utility that you can use to copy file from one server to another very easily, and there are many options available to allow you to be very specific about how you want the data transferred. Another aspect that makes rsync flexible is the many ways you can manipulate the source and target directories. However, you don’t even have to use the network; you can even copy data from one directory to another on the same server.

Copying a file within the same server from one location to another

Here we are using -r to copy recursively, you can also use ( -a ) i.e. for archive which retains as much metadata as possible (in most cases, it should make everything an exact copy).

To copy files between two servers

Using SCP to copy file from one server to another

A useful alternative to rsync is the Secure Copy (SCP) utility to copy file from one server to another, which comes bundled with OpenSSH. It allows you to quickly copy files from one node to another. If your goal is to send a single file or a small number of files to another machine, SCP is a great tool you can use to get the job done. To utilize SCP, we’ll use the scp command. Since you most likely already have OpenSSH installed, you should already have the scp command available

Using SCP is very similar in nature to rsync. The command requires a source, a target, and a filename. To transfer a single file from your local machine to another, the resulting command would look similar to the following:

If you do not specifiy the target directory while doing scp, then the home directory of the target user will be used are destination.

With our previous scp examples, we’ve only been copying a single file. If we want to transfer or download an entire directory and its contents, we will need to use the -r option, which allows us to do a recursive copy:

Using NFS to share file from one server to another

A Network File System (NFS) is a great method of sharing files between Linux or UNIX servers. I have written another article with detailed steps to setup NFSv4 and NFSv3 with examples in RHEL/CentOS 7 and 8 Linux

On my RHEL node I have installed nfs-utils and now will setup my exports. To set up NFS, let’s first create some directories that we will share to other users. Each share in NFS is known as an Export .

• In the first line I have given share access to world for /share directory.
• In the second line, after the directory is called out in a line, we’re also setting which network is able to access them ( 10.0.2.0/255.255.255.0 in our case). This means that if you’re connecting from a different network, your access will be denied.
• As far as what these options do, the first ( rw ) is rather self-explanatory.
• One option you’ll see quite often in the wild is no_root_squash . Normally, the root user on one system will get map to nobody on the other for security reasons. In most cases, one system having root access to another is a bad idea. The no_root_squash option disables this, and it allows the root user on one end to be treated as the root user on the other. .

Next restart your nfs-server services on the server

To check the list of shares currently exported

Now try to mount the directory /share from the client side

So our directory mount is successful, Next validate the content

We can validate the same on our client node using below command

After successfully mounting the share on the client node, you can copy the file locally to your node.

Using SSHFS to copy file from one server to another

SSH Filesystem (SSHFS) is a file sharing solution similar to NFS and Samba. NFS and Samba are great solutions for designating file shares but these technologies may be more complex than necessary if you want to set up a temporary file-sharing service to use for a specific period of time. SSHFS allows you to mount a remote directory on your local machine, and have it treated just like any other directory. The mounted SSHFS directory will be available for the life of the SSH connection and can be used to copy file from one server to another.

Drawbacks of using SSHFS

• Performance of file transfers won’t be as fast as with an NFS mount, since there’s encryption that needs to be taken into consideration as well
• Another downside is that you’d want to save your work regularly as you work on files within an SSHFS mount, because if the SSH connection drops for any reason, you may lose data.

SSHFS is part of EPEL repository, which you can install using yum

For SSHFS to work, we’ll need a directory on both your local Linux machine as well as a remote Linux server. SSHFS can mount any directory from the remote server where you have SSH access.

Here I am mounting /share from node2 on node1

Now validate the content of /mnt and make sure the path is properly mounted

Now you can copy the files from /mnt locally to any other directory

Once your copying is complete, manually unmount the respective directory. There are two ways to do so. First, we can use the umount command as the root (just like we normally would):

You can also use HTTPS for file sharing, although it would be more like of file uploading and downloading via GET and PUT using curl command.

Lastly I hope the commands from this article to copy file from one server to another in Linux or Unix was helpful. So, let me know your suggestions and feedback using the comment section.

Related Posts

Didn’t find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

10 thoughts on “5 commands to copy file from one server to another in Linux or Unix”

Lovely just what I was searching for.Thanks to the author for taking his clock time on this one.

Excellent post. Keep writing such kind of info on your
blog. Im really impressed by your site.

This is the right site for everyone who hopes to understand this topic.
You know so much its almost tough to argue with you (not that I really would want to…HaHa).
You certainly put a brand new spin on a subject that’s been written about for decades.
Great stuff, just wonderful!

Can we transfer 20tb data of file from one system to anothersystem

Is it possible to transfer 20TB data of file from one system to another system using scp

20TB is a big data, theoretically I don’t see any problem. I would recommend using rsync as it will perform incremental copy. But copying such large chunk of data then you need heave bandwidth, memory resources in your system. With rsync atleast you can be safe to pick up from where you lost in case the transfer fails during the transaction.

Can we copy one public key for many systems for passwordless login to another systems

Hello, congratulations on the blog.
To big data, such as 500Gb or 1Tb, rysnc is better than scp. Have you used it to perform backups throughout the day? How to improve throughput, or bandwidth? Thanks

Hello, thank you for the feedback
In our production environment also we perform transfer of multiple file each 500GB+
Now generally when you attempt to copy such big file, the copy tool (rsync or any other tool) will eat up all the available bandwidth so it is very important that you assign a BW limit. In our network we have pre-defined a restriction of 1Gb so that other applications using BW don’t get impacted
You can use —bwlimit to limit the BW with rsync, also —compare-dest to make sure only newer data is copied (assuming the transfer was stopped due to some reason)

Источник

Команда cp: правильное копирование папок с файлами в *nix

В этой статье будут раскрыты некоторые неочевидные вещи связанные с использованием wildcards при копировании, неоднозначное поведение команды cp при копировании, а также способы позволяющие корректно копировать огромное количество файлов без пропусков и вылетов.

Допустим нам нужно скопировать всё из папки /source в папку /target.

Первое, что приходит на ум это:

Сразу исправим эту команду на:

Ключ -a добавит копирование всех аттрибутов, прав и добавит рекурсию. Когда не требуется точное воспроизведение прав достаточно ключа -r .

После копирования мы обнаружим, что скопировались не все файлы — были проигнорированы файлы начинающиеся с точки типа:

.profile
.local
.mc
и тому подобные.

Почему же так произошло?

Потому что wildcards обрабатывает shell ( bash в типовом случае). По умолчанию bash проигнорирует все файлы начинающиеся с точек, так как трактует их как скрытые. Чтобы избежать такого поведения нам придётся изменить поведение bash с помощью команды:

Чтобы это изменение поведения сохранилось после перезагрузки, можно сделать файл wildcard.sh c этой командой в папке /etc/profile.d (возможно в вашем дистрибутиве иная папка).

А если в директории-источнике нет файлов, то shell не сможет ничего подставить вместо звёздочки, и также копирование завершится с ошибкой. Против подобной ситуации есть опции failglob и nullglob . Нам потребуется выставить failglob , которая не даст команде выполниться. nullglob не подойдёт, так как она строку с wildcards не нашедшими совпадения преобразует в пустую строку (нулевой длины), что для cp вызовет ошибку.

Однако, если в папке тысячи файлов и больше, то от подхода с использованием wildcards стоит отказаться вовсе. Дело в том, что bash разворачивает wildcards в очень длинную командную строку наподобие:

На длину командной строки есть ограничение, которое мы можем узнать используя команду:

Получим максимальную длину командной строки в байтах:

Получим что-то типа:

Итак, давайте будем обходиться вовсе без wildcards.

Давайте просто напишем

И тут мы столкнёмся с неоднозначностью поведения cp . Если папки /target не существует, то мы получим то, что нам нужно.

Однако, если папка target существует, то файлы будут скопированы в папку /target/source.

Не всегда мы можем удалить заранее папку /target, так как в ней могут быть нужные нам файлы и наша цель, допустим, дополнить файлы в /target файлами из /source.

Если бы папки источника и приёмника назывались одинаково, например, мы копировали бы из /source в /home/source, то можно было бы использовать команду:

И после копирования файлы в /home/source оказались бы дополненными файлами из /source.

Такая вот логическая задачка: мы можем дополнить файлы в директории-приёмнике, если папки называются одинаково, но если они отличаются, то папка-исходник будет помещена внутрь приёмника. Как скопировать файлы из /source в /target с помощью cp без wildcards?

Чтобы обойти это вредное ограничение мы используем неочевидное решение:

Те кто хорошо знаком с DOS и Linux уже всё поняли: внутри каждой папки есть 2 невидимые папки «.» и «..», являющиеся псевдопапками-ссылками на текущую и вышестоящие директории.

• При копировании cp проверяет существование и пытается создать /target/.
• Такая директория существует и это есть /target
• Файлы из /source скопированы в /target корректно.

Итак, вешаем в жирную рамочку в своей памяти или на стене:

Поведение этой команды однозначно. Всё отработает без ошибок вне зависимости от того миллион у вас файлов или их нет вовсе.

Выводы

Если нужно скопировать все файлы из одной папки в другую, не используем wildcards, вместо них лучше использовать cp в сочетании с точкой в конце папки-источника. Это скопирует все файлы, включая скрытые и не завалится при миллионах файлов или полном отсутствии файлов.

Послесловие

vmspike предложил аналогичный по результату вариант команды:

ВНИМАНИЕ: регистр буквы T имеет значение. Если перепутать, то получите полную белиберду: направление копирования поменяется.
Благодарности:

Источник