Linux compress fs directory on the fly

How do I Compress a Whole Linux or UNIX Directory?

H ow can I compress a whole directory under a Linux / UNIX using a shell prompt?

It is very easy to compress a Whole Linux/UNIX directory. It is useful to backup files, email all files, or even to send software you have created to friends. Technically, it is called as a compressed archive. GNU tar or BSD tar command is best for this work. It can be use on remote Linux or UNIX server. It does two things for you:

Tutorial details
Difficulty level	Easy
Root privileges	No
Requirements	tar
Est. reading time	1 mintue

1. Create the archive
2. Compress the archive
3. Additional operations includes listing, deleting or updating the archive files.

How to compress a whole directory in Linux or Unix

You need to use the tar command as follows (syntax of tar command):
tar -zcvf archive-name.tar.gz source-directory-name
Where,

• -z : Compress archive using gzip program in Linux or Unix
• -c : Create archive on Linux
• -v : Verbose i.e display progress while creating archive
• -f : Archive File name

For example, say you have a directory called /home/jerry/prog and you would like to compress this directory then you can type tar command as follows:
$ tar -zcvf prog-1-jan-2005.tar.gz /home/jerry/prog
Above command will create an archive file called prog-1-jan-2005.tar.gz in current directory. If you wish to restore your archive then you need to use the following command (it will extract all files in current directory):
$ tar -zxvf prog-1-jan-2005.tar.gz
Where,

• -x : Extract files from given archive

If you wish to extract files in particular directory, for example in /tmp then you need to use the following command:
$ tar -zxvf prog-1-jan-2005.tar.gz -C /tmp
$ cd /tmp
$ ls —

Compress an entire directory to a single file

To compress directory named /home/vivek/bin/ in to a /tmp/bin-backup.tar.gz type the tar command on Linux:
tar -zcvf /tmp/bin-backup.tar.gz /home/vivek/bin/

You can compress multiple directories too:
tar -zcvf my-compressed.tar.gz /path/to/dir1/ /path/to/dir2/

A note about non gnu/tar command

The above syntax use GNU tar command for compressing and uncompressing tar files. If your system does not use GNU tar, you can still create a compressed tar file, via the following syntax:
tar -cvf — file1 file2 dir3 | gzip > archive.tar.gz

How to use bzip2 compression instead of gzip compression

The syntax is:
tar -jcvf my-compressed.tar.bz2 /path/to/dir1/

• -j : Compress archive using bzip2 program in Linux or Unix
• -c : Make archive on Linux
• -v : Verbose output
• -f my-compressed.tar.bz2 : Store archive in my-compressed.tar.bz2 file

Conclusion

You learend how to comporess a whole directory in Linux, macOS, *BSD and Unix-like systems using the zip command/tar command. See also:

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

See the zip and tar command man page documentation by typing the following man command:
man tar
man zip

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Источник

Compress a folder with tar?

I’m trying to compress a folder ( /var/www/ ) to

/www_backups/$time.tar where $time is the current date.

This is what I have:

I am completely lost and I’ve been at this for hours now. Not sure if -czf is correct. I simply want to copy all of the content in /var/www into a $time.tar file, and I want to maintain the file permissions for all of the files. Can anyone help me out?

2 Answers 2

To tar and gzip a folder, the syntax is:

Adding — before the options ( czf ) is optional with tar . The effect of czf is as follows:

• c — create an archive file (as opposed to extract, which is x )
• f — filename of the archive file
• z — filter archive through gzip (remove this option to create a .tar file)

If you want to tar the current directory, use . to designate that.

To construct filenames dynamically, use the date utility (look at its man page for the available format options). For example:

This will create a file named something like 20120902-185558.tar.gz .

On Linux, chances are your tar also supports BZip2 compression with the j rather than z option. And possibly others. Check the man page on your local system.

/www_backups/$time.tar /var/www/» Imagine i have a file called test.txt inside /var/www. After making a tar copy of the file, when i extract it it will be placed inside /var/www directories. Does that make sense? I hope it does, kinda hard to explain. I will check for BZip2 support, thanks for the suggestion!

Examples for Most Common Compression Algorithms

The question title for this is simply «Compress a folder with tar?» Since this title is very general, but the question and answer are much more specific, and due to the very large number of views this question has attracted, I felt it would be beneficial to add an up-to-date list of examples of both archiving/compressing and extracting/uncompressing, with various commonly used compression algorithms.

These have been tested with Ubuntu 18.04.4. They are very simple for general use, but could easily be integrated into the OP’s more specific question contents using the the techniques in the accepted answer and helpful comments above.

One thing to note for the more general audience is that tar will not add the necessary extensions (like .tar.gz ) automatically — the user has to explicitly add those, as seen in the commands below:

See the tar man page (best to use man tar on your specific machine) for further details. Below I summarize the options used above directly from the man page:

-x, —extract, —get
extract files from an archive

-v, —verbose
verbosely list files processed

-z, —gzip
filter the archive through gzip

-j, —bzip2
filter the archive through bzip2

-J, —xz
filter the archive through xz

-f, —file=ARCHIVE
use archive file or device ARCHIVE

No need to add the — in front of the combined options, or the = sign between the f option and the filename.

I got all this from my recent article, which will be expanded further into a much more comprehensive article as I have time to work on it.

Источник

compressing dd backup on the fly

Maybe this will sound like dumb question but the way i’m trying to do it doesn’t work.

I’m on livecd, drive is unmounted, etc.

When i do backup this way

. normally it would work but i don’t have enough space on external hd i’m copying to (it ALMOST fits into it). So I wanted to compress this way

. but i got permissions denied. I don’t understand.

3 Answers 3

Do you have access to the sda2-backup. gz file? Sudo only works with the command after it, and doesn’t apply to the redirection. If you want it to apply to the redirection, then run the shell as root so all the children process are root as well:

Alternatively, you could mount the disk with the uid / gid mount options (assuming ext3) so you have write permissions as whatever user you are. Or, use root to create a folder in /media/disk which you have permissions for.

Other Information that might help you:

• The block size only really matters for speed for the most part. The default is 512 bytes which you want to keep for the MBR and floppy disks. Larger sizes to a point should speed up the operations, think of it as analogous to a buffer. Here is a link to someone who did some speed benchmarks with different block sizes. But you should do your own testing, as performance is influenced by many factors. Take also a look at the other answer by andreas
• If you want to accomplish this over the network with ssh and netcat so space may not be as big of an issue, see this serverfault question.
• Do you really need an image of the partition, there might be better backup strategies?
• dd is a very dangerous command, use of instead of if and you end up overwriting what you are trying to backup!! Notice how the keys o and i are next to each other? So be very very very careful.

Источник

Linux compress fs directory on the fly

Команда compress предназначена для сжатия данных без потерь с помощью соответствующей утилиты, использующей алгоритм Лемпела-Зива. Целью использования данной утилиты является экономия дискового пространства. Используемый утилитой алгоритм долгое время подпадал под действие патента, поэтому она не снискала особой популярности. Тем не менее, в данный момент она может беспрепятственно использоваться, ведь срок действия патента истек. В то же время, она не позволяет достичь такой степени сжатия данных, как утилиты bzip2 и xz, посему может использоваться лишь как аналог популярных утилит gzip и zip.

Базовый синтаксис команды выглядит следующим образом:

Чаще всего compress используется вообще без каких-либо параметров, причем в качестве аргументов может передаваться неограниченное количество имен файлов, которые следует сжать. По умолчанию оригинальные версии файлов заменяются на их сжатые версии (с расширением .Z) с соответствующими метаданными (то есть, меткой времени модификации, правами доступа, именами владельца и группы владельцев и так далее). Если вас не устраивает такое положение дел, вы можете воспользоваться параметром -с для сохранения оригинальных версий файлов (в этом случае утилита будет выводить сжатые данные посредством стандартного потока вывода, поэтому вам придется воспользоваться перенаправлением потока данных таким образом, как показано ниже). Параметр -f позволяет принудительно перезаписать существующий файл архива. Параметр -v позволяет выводить информацию о степени сжатия файла. Наконец, параметр -r позволяет осуществить рекурсивное сжатие файлов.

Если же вам нужно создать архив с несколькими файлами внутри, одной утилиты compress будет явно мало. Для этой цели также понадобится утилита tar, с помощью которой можно создать архив с файлами, после чего сжать этот архив с помощью утилиты compress. Например, вы можете использовать следующую последовательность команд для создания архива с именем archive.tar.Z:

$ tar -cf archive.tar
$ compress archive.tar

Параметры -c и -f утилиты tar предназначены для указания на необходимость добавления всех файлов в один архив (-c) и чтения имени файла архива из следующего аргумента (-f).

Альтернативным вариантом является замена последней команды на параметр -Z утилиты tar, позволяющий автоматически сжать полученный архив с помощью compress:

$ tar -cfZ archive.tar.Z

Установка утилиты

В первую очередь вам придется установить рассматриваемую утилиту; проще всего это сделать с помощью терминала путем исполнения соответствующей вашему дистрибутиву команды. Подробнее об установке программного обеспечения рассказано в данном разделе.

Команда для Linux Mint и Ubuntu:

$ sudo apt-get install ncompress

Команда для Fedora Workstation:

$ sudo dnf install ncompress

Примеры использования

1. Сжатие одного файла

В результате оригинальный файл text.txt будет заменен на свою сжатую версию text.txt.Z.

2. Одновременное сжатие нескольких файлов

$ compress text1.txt text2.txt text3.txt

В этом случае также все оригинальные версии файлов (text1.txt, text2.txt, text3.txt) будут заменены на сжатые версии (text1.txt.Z, text2.txt.Z, text3.txt.Z). Добавление нескольких файлов в единый файл архива будет рассмотрено ниже.

3. Сжатие одного файла с сохранением оригинала

$ compress -c text.txt > text.txt.Z

Теперь оригинальный файл text.txt будет оставлен в директории вместе со сжатой версией text.txt.Z.

4. Восстановление оригинальной версии файла из сжатой версии

$ compress -d text.txt.Z

В результате сжатая версия файла text.txt.Z будет заменена на его оригинальную версию text.txt.

5. Создание сжатого архива со всеми файлами из директории

$ tar -cfZ etc.tar.Z /etc/

В результате будет создан архив etc.tar.Z с файлами из директории /etc/.

6. Рекурсивное сжатие файлов из директории

$ compress -rv nolf228

В результате в заданной директории все подвергающиеся сжатию файлы будут заменены на их сжатые версии.

Источник

Linux compress fs directory on the fly

F2FS (Flash-Friendly File System) is a file system intended for NAND-based flash memory equipped with Flash Translation Layer. Unlike JFFS or UBIFS it relies on FTL to handle write distribution. It is supported from kernel 3.8 onwards.

An FTL is found in all flash memory with a SCSI/SATA/PCIe/NVMe interface [1], opposed to bare NAND Flash and SmartMediaCards [2].

Contents

Known issues

fsck failures

F2FS has a weak fsck that can lead to data loss in case of a sudden power loss [3][4].

If power losses are frequent, consider an alternative file system.

Long running fsck delays boot

If the kernel version has changed between boots, the fsck.f2fs utility will perform a full file system check which will take longer to finish[5].

This may be mitigated in the future thanks to a recent commit [6].

Creating a F2FS file system

This article assumes the device has partitions already setup. Install f2fs-tools . Use mkfs.f2fs to format the target partition referred to as /dev/sdxY :

Compression

To use compression, include the compression option. Example:

When mounting the filesystem, specify compress_algorithm=(lzo|lz4|zstd|lzo-rle) . Using compress_extension=txt will cause all txt files to be compressed by default.

In order to tell F2FS to compress a file or a directory, use :

File-based encryption support

Since Linux 4.2, F2FS natively supports file encryption. Encryption is applied at the directory level, and different directories can use different encryption keys. This is different from both dm-crypt, which is block-device level encryption, and from eCryptfs, which is a stacked cryptographic filesystem. To use F2FS’s native encryption support, see the fscrypt article. Create the file system with

or add encryption capability at a later time with fsck.f2fs -O encrypt /dev/sdxY .

Mounting a F2FS file system

The file system can then be mounted manually or via other mechanisms:

Implementation of discard

By default, F2FS is mounted using a hybrid TRIM mode which behaves as continuous TRIM. This implementation creates asynchronous discard threads to alleviate long discarding latency among RW IOs. It keeps candidates in memory, and the thread issues them in idle time [7]. As a result of this, users wanting periodic TRIM will need to implicitly set the nodiscard mount option in /etc/fstab or pass it to mount if mounting manually.

Checking and repair

Checking and repairs to f2fs file systems are accomplished with fsck.f2fs provided by f2fs-tools . See fsck.f2fs(8) for available switches. Example:

Grow an F2FS file system

When the filesystem is unmounted, it can be grown if the partition is expanded. Shrinking is not currently supported.

First use a partition tool to resize the partition: for example, suppose the output of the print command in the parted console is the following:

To resize the f2fs partition to occupy all the space up to the fourth one, just give resizepart 3 31GB and exit . Now expand the filesystem to fill the new partition using:

Источник