Linux tar multi volume

How to Split Large ‘tar’ Archive into Multiple Files of Certain Size

Are you worried of transferring or uploading large files over a network, then worry no more, because you can move your files in bits to deal with slow network speeds by splitting them into blocks of a given size.

In this how-to guide, we shall briefly explore the creation of archive files and splitting them into blocks of a selected size. We shall use tar , one of the most popular archiving utilities on Linux and also take advantage of the split utility to help us break our archive files into small bits.

Create and Split tar into Multiple Files or Parts in Linux

Before we move further, let us take note of, how these utilities can be used, the general syntax of a tar and split command is as follows:

Let us now delve into a few examples to illustrate the main concept of this article.

Example 1: We can first of all create an archive file as follows:

Create Tar Archive File

To confirm that out archive file has been created and also check its size, we can use ls command:

Then using the split utility, we can break the home.tar.bz2 archive file into small blocks each of size 10MB as follows:

Split Tar File into Parts in Linux

As you can see from the output of the commands above, the tar archive file has been split to four parts.

Note: In the split command above, the option -b is used to specify the size of each block and the «home.tar.bz2.part» is the prefix in the name of each block file created after splitting.

Example 2: Similar to the case above, here, we can create an archive file of a Linux Mint ISO image file.

Then follow the same steps in example 1 above to split the archive file into small bits of size 200MB .

Split Tar Archive File to Fixed Sizes

Example 3: In this instance, we can use a pipe to connect the output of the tar command to split as follows:

Create and Split Tar Archive File into Parts

Confirm the files:

Check Parts of Tar Files

In this last example, we do not have to specify an archive name as you have noticed, simply use a — sign.

How to Join Tar Files After Splitting

After successfully splitting tar files or any large file in Linux, you can join the files using the cat command. Employing cat is the most efficient and reliable method of performing a joining operation.

To join back all the blocks or tar files, we issue the command below:

We can see that after running the cat command, it combines all the small blocks we had earlier on created to the original tar archive file of the same size.

Conclusion

The whole idea is simple, as we have illustrated above, you simply need to know and understand how to use the various options of tar and split utilities.

You can refer to their manual entry pages of to learn more other options and perform some complex operations or you can go through the following article to learn more about tar command.

For any questions or further tips, you can share your thoughts via the comment section below.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

We are thankful for your never ending support.

Источник

Linux tar multi volume

To create an archive that is larger than will fit on a single unit of the media, use the ‘ —multi-volume ’ (‘ -M ’) option in conjunction with the ‘ —create ’ option (see section How to Create Archives). A multi-volume archive can be manipulated like any other archive (provided the ‘ —multi-volume ’ option is specified), but is stored on more than one tape or file.

When you specify ‘ —multi-volume ’, tar does not report an error when it comes to the end of an archive volume (when reading), or the end of the media (when writing). Instead, it prompts you to load a new storage volume. If the archive is on a magnetic tape, you should change tapes when you see the prompt; if the archive is on a floppy disk, you should change disks; etc.

Creates a multi-volume archive, when used in conjunction with ‘ —create ’ (‘ -c ’). To perform any other operation on a multi-volume archive, specify ‘ —multi-volume ’ in conjunction with that operation. For example:

The method tar uses to detect end of tape is not perfect, and fails on some operating systems or on some devices. If tar cannot detect the end of the tape itself, you can use ‘ —tape-length ’ option to inform it about the capacity of the tape:

‘ —tape-length= size [ suf ] ’ ‘ -L size [ suf ] ’

Set maximum length of a volume. The suf , if given, specifies units in which size is expressed, e.g. ‘ 2M ’ mean 2 megabytes (see Table 9.1, for a list of allowed size suffixes). Without suf , units of 1024 bytes (kilobyte) are assumed.

This option selects ‘ —multi-volume ’ automatically. For example:

or, which is equivalent:

where n is the ordinal number of the volume to be created and archive is archive file or device name.

When prompting for a new tape, tar accepts any of the following responses:

Request tar to explain possible responses.

Request tar to exit immediately.

Request tar to write the next volume on the file file-name .

Request tar to run a subshell. This option can be disabled by giving ‘ —restrict ’ command line option to tar (27).

Request tar to begin writing the next volume.

(You should only type ‘ y ’ after you have changed the tape; otherwise tar will write over the volume it just finished.)

The volume number used by tar in its tape-changing prompt can be changed; if you give the ‘ —volno-file= file-of-number ’ option, then file-of-number should be an non-existing file to be created, or else, a file already containing a decimal number. That number will be used as the volume number of the first volume written. When tar is finished, it will rewrite the file with the now-current volume number. (This does not change the volume number written on a tape label, as per Including a Label in the Archive, it only affects the number used in the prompt.)

If you want more elaborate behavior than this, you can write a special new volume script, that will be responsible for changing the volume, and instruct tar to use it instead of its normal prompting procedure:

‘ —info-script= command ’ ‘ —new-volume-script= command ’ ‘ -F command ’

Specify the command to invoke when switching volumes. The command can be used to eject cassettes, or to broadcast messages such as ‘ Someone please come change my tape ’ when performing unattended backups.

The name of the archive tar is processing.

Current blocking factor (see section Blocking).

Ordinal number of the volume tar is about to start.

A short option describing the operation tar is executing. See section The Five Advanced tar Operations, for a complete list of subcommand options.

Format of the archive being processed. See section Controlling the Archive Format, for a complete list of archive format names.

File descriptor which can be used to communicate the new volume name to tar .

These variables can be used in the command itself, provided that they are properly quoted to prevent them from being expanded by the shell that invokes tar .

The volume script can instruct tar to use new archive name, by writing in to file descriptor $TAR_FD (see below for an example).

If the info script fails, tar exits; otherwise, it begins writing the next volume.

If you want tar to cycle through a series of files or tape drives, there are three approaches to choose from. First of all, you can give tar multiple ‘ —file ’ options. In this case the specified files will be used, in sequence, as the successive volumes of the archive. Only when the first one in the sequence needs to be used again will tar prompt for a tape change (or run the info script). For example, suppose someone has two tape drives on a system named ‘/dev/tape0’ and ‘/dev/tape1’. For having GNU tar to switch to the second drive when it needs to write the second tape, and then back to the first tape, etc., just do either of:

The second method is to use the ‘ n ’ response to the tape-change prompt.

Finally, the most flexible approach is to use a volume script, that writes new archive name to the file descriptor $TAR_FD . For example, the following volume script will create a series of archive files, named ‘ archive - vol ’, where archive is the name of the archive being created (as given by ‘ —file ’ option) and vol is the ordinal number of the archive being created:

The same script can be used while listing, comparing or extracting from the created archive. For example:

Notice, that the first command had to use ‘ -L ’ option, since otherwise GNU tar will end up writing everything to file ‘archive.tar’.

You can read each individual volume of a multi-volume archive as if it were an archive by itself. For example, to list the contents of one volume, use ‘ —list ’, without ‘ —multi-volume ’ specified. To extract an archive member from one volume (assuming it is described that volume), use ‘ —extract ’, again without ‘ —multi-volume ’.

If an archive member is split across volumes (i.e., its entry begins on one volume of the media and ends on another), you need to specify ‘ —multi-volume ’ to extract it successfully. In this case, you should load the volume where the archive member starts, and use ‘ tar —extract —multi-volume ’— tar will prompt for later volumes as it needs them. See section Extracting an Entire Archive, for more information about extracting archives.

Multi-volume archives can be modified like any other archive. To add files to a multi-volume archive, you need to only mount the last volume of the archive media (and new volumes, if needed). For all other operations, you need to use the entire archive.

If a multi-volume archive was labeled using ‘ —label= archive-label ’ (see section Including a Label in the Archive) when it was created, tar will not automatically label volumes which are added later. To label subsequent volumes, specify ‘ —label= archive-label ’ again in conjunction with the ‘ —append ’, ‘ —update ’ or ‘ —concatenate ’ operation.

Notice that multi-volume support is a GNU extension and the archives created in this mode should be read only using GNU tar . If you absolutely have to process such archives using a third-party tar implementation, read Extracting Members Split Between Volumes.

This document was generated on March 24, 2021 using texi2html 5.0.

Источник

Creating and extracting multivolume tar files

Some servers do not support files larger than 2GB, also when creating archives larger than 2GB most probably server will run out of memory or other resources. So is best to split large archives in multiple files. This is a good way to move large sites to another server.

Creating the multivolume tar file

If for example you have a folder «my_documents» and you want to create multivolume tar archive with its contents, this command will start archiving it and will split the files to 1.2GB:

When first file reached 1.2GB (after some minutes) it will display a prompt to move to next file, it will ask if volume #2 for this archive is ready, type this answer, notice the «2«:

press enter, it will ask if files are ready, press enter again to confirm without typing anything. On next prompt type «n my_documents3.tar» and press enter twice again, and so on. You should now have multiple files of maximum 1.2GB named: my_documents.tar, my_documents2.tar, my_documents3.tar.

Unlike with zip files, it doesn’t seem to be a command for testing multivolume tar files.

If you created the files in order to move a large site, I would recommend to move them with wget triggered from destination server:

You should do some basic testing to compare file integrity if copied from another server, for example use «cksum my_documents.tar» on old and new server to make sure it returns same result, meaning that files are the same.

To unpack the multivolume tar archive

Use this command to unpack the multivolume tar archive, this would probably hapen on the new server if you moved a site:

After some work (minutes on most servers), it will ask for next file, type this:

Then on next question type n my_documents3.tar, etc.

Creating single tar files:

To create one single tar file, for example archiving «images» folder:

unpack single files (not multivolume) with this command:

Источник

Tar Incremental/Multivolume Backup

Experimenting tar incremental backup

— only one file created!

— Failed. Specifying -M won’t help.

investigate why above failed

need to organizate the archive into a sequence of files —info-script=SCRIPT-NAME

better add some label? —label=»SOME-PREFIX `date +SOME-FORMAT`«

Multiple Volumes with tar

The multi-volume with tar kind of assumes you will be swapping tapes out; so what you need to do is create a script to «simulate» a tape switch by renaming files. You can manually rename the file to file1, etc, and let tar «think» you changed tapes.

Below is an example. The following example only counts to 10 correctly, but you can get the idea. The problem with tar multi-volume is that if you want to compress, you have to compress afterwards, and the results are not predictable to know how to fill a cd, etc. There is a backup solution called «dar», (do a google search); which will let you set a «compressed multi-volume size». It is worth checking out.

I have a better set of scripts which does this with perl, but I hate do distribute them because they are suited to my system, but if you want a copy, email me. My scripts count as high as you need, and break each directory into it’s own backup, in case you need to restore just one directory. There are restore scripts too, which do the opposite function to reassemble the tar segments.

And here is the backup-rotate.sh

Multiple Volumes with tar

this is how they work. You run «do-backup root» or «do-backup home». They will tar and gzip all directories, and break them into small pieces, for easier copying. /usr is broken into it’s sub-dirs since it’s so big.

The homedirs are also broken into subdirs, AND all the hidden files and directories are saved into a separate directory «rootbacktemp», so when you restore, remember to copy them back to your homedir root.

When the cd’s are make, 3 programs are copied to each cd, «restore,restore-rotate.pl, and microperl». microperl is a small version of perl, that allows you to restore from scratch, without a perl installation.

To restore, just copy all the files you want into a directory, along with «restore,restore-rotate.pl, and microperl». Then run restore. Everything will be automatic after that. These cd’s are written as ext2, so if you boot with a rescue floppy, and try to mount the cd’s to copy them in, use «mount -t ext2 /dev/hdc /mnt» (or whatever is right for you.)

The «cdwrite» script , way down at the bottom, has the setting for cdrecord. Use «cdrecord -scanbus» to get your dev, and put it in there. The dummy switch is for testing.

If you use the dummy switch, and keep inserting a blank cdr when prompted, you won’t write any cd’s but all the iso’s will be created, in /.

This script isn’t perfect, it requires that you have a good bit of free space on the hard drive for the iso’s. The iso’s are saved in /, but you could modify the script to delete each iso after it is written to cdr, if you are tight on space. There is a line in cdwrite which is commented out «#unlink image.iso». Just uncomment that line to delete isos after they are written.

I suggest using the dummy switch first, and see how it goes, if the iso’s are created, then you are doing it right.

I tried to do some parallel processing in this script, so it will write and process the next iso at the same time. This is very cpu intensive, so it is best to do this when nothing else is happening.

The scripts run fine with a scsi cdwriter, I don’t know if they behave well with ide-scsi.

Good luck, feel free to modify and use the scripts as you desire.

P.S. If you don’t feel comfortable with running the binary «microperl» which I’ve include (I wouldn’t run any binary sent to me); you can make your own or you can change the restore scripts to run with #!/usr/bin/perl

Microperl can be made from the perl5.8 distibution, with «make -f Makefile.micro».

These scripts run good once you understand them.

Backup under Linux (Re: Dump Backup failed. )

You can do incremental backups with tar. Here is my tar script for backing up everything which changed since the last backup. In my case, I do it to a zipdisk, but you could modify this to do it to a tape. You would have to store the date of the last incremental backup somewhere; I do it on the zipdisk. You would also have to use mt commands to position yourself on the tape. And I would strongly recommend using the eof option of mt to add an EOF after each record. Then you can use mt fsf skipping twice for each record to position the tape.

Источник