Merge two files linux

Linux merge command

On Unix-like operating systems, the merge command performs a three-way file merge.

The merge process analyzes three files: a base version, and two conflicting modified versions. It attempts to automatically combine both sets of modifications, based on the shared base version, into a single merged file. If automatic merge is not possible, it facilitates manual merging.

This page describes the GNU/Linux version of merge.

Description

merge is part of the RCS version control package. It is used to perform a three-way file merge.

merge analyzes three files — an original file, and two modified versions of the original — and compares them, line-by-line, attempting to resolve the differences between the two sets of modifications to create a single, unified file which represents both sets of changes.

Depending on the differences between the two sets of changes, this may be an automatic process or may require user input.

If neither set of changes conflicts with the other, merge can usually figure out what to do on its own. But if the two sets of changes conflict — for example, if the same line of text is worded differently in both modified files — merge shows the conflict in the resulting merged file.

How merging works

merge incorporates all changes that lead from file2 to file3 into file1. The result ordinarily goes into file1.

Suppose file2 is the original, and both file1 and file3 are modifications of file2. Then merge combines both changes.

A conflict occurs if both file1 and file3 have changes in a common segment of lines. If a conflict is found, merge normally outputs a warning and brackets the conflict with » >>>>>>» lines. For instance, a typical conflict looks like this:

If there are conflicts, the user should edit the result and delete one of the alternatives.

Syntax

Options

-A	Output conflicts using the -A style of diff3(1) (if supported by diff3). This merges all changes leading from file2 to file3 into file1, and generates the most verbose output.
-E, -e	These options specify conflict styles that generate less information than -A. See diff3(1) for details. The default is -E. With -e, merge does not warn about conflicts.
-L label	This option may be given up to three times, and specifies labels to be used in place of the corresponding file names in conflict reports. That is, merge -L x -L y -L z a b c generates output that looks like it came from files x, y, and z instead of from files a, b, and c.
-p	Send results to standard output instead of overwriting file1.
-q	Quiet mode. Do not warn about conflicts.
-V	Print RCS’s version number.

Examples

Let’s say we have a file named orig.txt with the following contents.

. and a file named mod1.txt, which is a modified version of orig.txt:

. and a file named mod2.txt, which is also a modified version of orig.txt:

. and we run merge as follows:

It analyzes all three files, write to mod1.txt, and display the following warning:

This means the merge was successful, but we should be aware that there was a conflict. If we open mod1.txt — which by default is the file where the merge is written — we find that it now contains the following text:

It is up to us to decide which «Oranges are. » line to keep (or to combine them in our own way), and make the edit to the file manually.

Related commands

diff — Identify the differences between two files.

Источник

how to merge multiple files into one single file in linux

Many a times you may have multiple files that needs to merged into one single file. It could be that you previously split a single file into multiple files, and want to just merge them back or you have several log files that you want merged into one. Whatever the reason, it is very easy to merge multiple text files into a single file in Linux.

The command in Linux to concatenate or merge multiple files into one file is called cat. The cat command by default will concatenate and print out multiple files to the standard output. You can redirect the standard output to a file using the ‘>‘ operator to save the output to disk or file system.

Another useful utility to merge files is called join that can join lines of two files based on common fields. It can however work only on two files at a time, and I have found it to be quite cumbersome to use. We will cover mostly the cat command in this post.

Merge Multiple files into One in Order

The cat command takes a list of file names as its argument. The order in which the file names are specified in the command line dictates the order in which the files are merged or combined. So, if you have several files named file1.txt, file2.txt, file3.txt etc…

bash$ cat file1.txt file2.txt file3.txt file4.txt > ./mergedfile.txt

The above command will append the contents of file2.txt to the end of file1.txt. The content of file3.txt is appended to the end of merged contents of file1.txt and file2.txt and so on…and the entire merged file is saved with the name mergedfile.txt in the current working directory.

Many a time, you might have an inordinately large number of files which makes it harder to type in all the file names. The cat command accepts regular expressions as input file names, which means you can use them to reduce the number of arguments.

bash$ cat file*.txt my*.txt > mergedfile.txt

This will merge all the files in the current directory that start with the name file and has a txt extension followed by the files that start with my and has a txt extension. You have to be careful about using regular expressions, if you want to preserve the order of files. If you get the regular expression wrong, it will affect the exact order in which the files are merged.

A quick and easy way to make sure the files get merged in the exact order you want, is to use the output of another file listing program such as ls or find and pipe it to the cat command. First execute the find command with the regular expression and verify the file order…

bash$ find . -name «file*.txt» -o -name «my*.txt»

This will print the files in order such that you can verify it to be correct or modify it to match what you want. You can then pipe that output into the cat command.

bash$ find . -name «file*.txt» -o -name «my*.txt» | xargs cat > ./mergedfile.txt

When you merge multiple files into one file using regular expressions to match them, especially when it is piped and where the output file is not very obvious, make sure that the regular expression does not match the filename of the merged file. In the case that it does match, usually the cat command is pretty good at error-ing out with the message “input file is output file”. But it helps to be careful to start with.

Merge Two Files at Arbitrary Location

Sometimes you might want to merge two files, but at a particular location within the content of a file. This is more like the process of inserting contents of one file into an another at a particular position in the file.

If the file sizes are small and manageable, then vi is a great editor tool to do this. Otherwise the option is to split the file first and then merge the resulting files in order. The easiest way is to split the file is based on the line numbers, exactly at where you want to insert the other file.

bash$ split -l 1234 file1.txt

You can split the file into any number of output files depending on your requirement. The above example will split the file file1.txt to chunks of 1234 lines. It is quite possible that you might end up with more than two files, named xaa, xab, xac etc..You can merge all of it back using the same cat command as mentioned earlier.

bash$ cat xaa file2.txt xa

The above command will merge the files in order with the contents of file2.txt in between the contents of xaa and xab.

Another use case is when you need to merge only specific parts of certain files depending on some condition. This is especially useful for me when I have to analyze several large log files, but am only interested in certain messages or lines. So, I will need to extract the important log messages based on some criteria from several log files and save them in a different file while also maintaining or preserving the order of the messages.

Though you can do this using cat and grep commands, you can do it with just the grep command as well.

bash$ grep -h «[Error]» logfile*.log > onlyerrors.log

The above will extract all the lines that match the pattern [Error] and save it to another file. You will have to make sure that the log files are in order when using the regular expression to match them, as mentioned earlier in the post.

Источник

How do I «merge» two text files?

I recently upgraded from Ubuntu 12.04 to 12.10 and at one point, it encountered an Apache config file conflict in apache2.conf . I didn’t give me a merge option at that point, so I just rejected the new file and the installer saved the new file as apache2.conf.dpkg-dist .

I can diff the two files with diff apache2.conf apache2.conf.dpkg-dist and get just the lines that are different. But I want to manually merge the two sort of like how I resolve merge conflicts in SVN or git. How can I do that?

3 Answers 3

Use vimdiff if you like vim . Otherwise, diffuse works great as well.

Version control has more information available when it resolves conflicts: it has not only your version and the other guy’s version but also the common ancestor, and thus it can do a three-way merge. Here, the common ancestor is the original version of the configuration file in the distribution, or the official version that you last merged with your changes.

Unfortunately neither Ubuntu nor any other major distribution I know of makes it completely seamless to do three-way merges when a configuration file is updated. You can get close, however, with etckeeper. Etckeeper is an add-on for APT, the package management tool used by Debian and derivatives, that manages /etc in a version control system (Bazaar, Darcs, Git, Mercurial); it’s been ported to other systems, including Yum in Fedora. I recommend using etckeeper; it’s also a great way of keeping track of the changes you make in /etc .

Some programs manage their configuration files with ucf, but that’s not something you have control of as a user.

More generally, when you have the ancestor and two versions, you can do a three-way merge with the merge utility shipped with RCS or with diff3 -m from diffutils.

There are also a great many interactive diff and merge programs. Emacs and Vim have interfaces for that, as do most diff viewers.

Источник

how to merge 2 big files [closed]

Want to improve this question? Update the question so it’s on-topic for Stack Overflow.

Closed 8 years ago .

Suppose I have 2 files with size of 100G each. And I want to merge them into one, and then delete them. In linux we can use

cat file1 file2 > final_file

But that needs to read 2 big files, and then write a bigger file. Is it possible just append one file to the other, so that no IO is required? Since metadata of file contains the location of the file, and the length, I am wondering whether it is possible to change the metadata of the file to do the merge, so no IO will happen.

3 Answers 3

Can you merge two files without writing one file onto the other?

Only in obscure theory. Since disk storage is always based on blocks and filesystems therefore store things on block boundaries, you could only append one file to another without rewriting if the first file ended perfectly on a block boundary. There are some rare filesystem configurations that use tail packing, but that would only help if the first file where already using the tail block of the previous file.

Unless that perfect scenario occurs or your filesystem is able to mark a partial block in the middle of the file (I’ve never heard of this), this won’t work. Just to kick the edge case around, there’s also no way outside of changing the kernel interace to make such a call (re: Link to a specific inode)

Can we make this better than doubling the size of both files?

Yes, we can use the append ( >> ) operation instead.

That will still result in using all the space of consumed by file2 twice over until we can delete it.

Can we avoid using extra space?

No. Unless somebody comes back with something I don’t know, you’re basically out of luck there. It’s possible to truncate a file, forgetting about the existence of the end of it, but there is no way to forget about the existence of the start unless we get back to modifying inodes directly and having to alter the kernel interface to the filesystem since that’s definitely not a a POSIX operation.

What about writing a little bit at a time, then deleting what we wrote?

No again. Since we can’t chop the start of a file off, we’d have to rewrite everything from the point of interest all the way to the end of the file. This would be very costly for IO and only useful after we’ve already read half the file.

What about sparse files?

Maybe! Sparse file allow us to store a long string of zeroes without using up nearly that much space. If we were to read file2 in large chunks starting at the end, we could write those blocks to the end of file1 . file1 would immediately look (and read) as if it were the same size as both, but it would be corrupted until we were done because everything we hadn’t written would be full of zeroes.

Explaining all this is another answer in itself, but if you can do a spare allocation, you would be able to use only your chunk read size + a little bit extra in disk space to perform this operation. For a reference talking about sparse blocks in the middle of files, see http://lwn.net/Articles/357767/ or do a search involving the term, SEEK_HOLE .

Why is this «maybe» instead of «yes»? Two parts: you’d have to write your own tool (at least we’re on the right site for that), and sparse files are not universally respected by file systems and other processes alike. Fortunately you probably won’t have to worry about other processes respecting your file, but you will have to worry about setting the right flags and making sure your filesystem is amenable. Last of all, you’ll still be reading and re-writing the length of file2 , which isn’t what you want. This method does mean you can append with just a small amount of disk space, though, rather at using at least 2*file2 amount of space.

Источник