How to join two text files, removing duplicates, in Windows

Is it possible to do using cmd.exe?

5 Answers 5

If you can affort to use a case insensitive comparison, and if you know that none of the lines are longer than 511 bytes (127 for XP), then you can use the following:

First part (merging two text files) is possible. (See Documentation of copy command)

For part 2, you can use sort and uniq utilities from CoreUtils for Windows. This are windows port of the linux utilities.

This has a limitation that you will lose track of original sequencing.

Update 1

Windows also ships with a native SORT.EXE.

Update 2

You may also use the same approach of Unix or PowerShell with pure Batch, developing a simple uniq.bat filter program:

EDIT: The program below is a Batch-JScript hybrid version of uniq program, more reliable and faster; copy this program in a file called uniq.bat :

This way, you may use this solution:

However, in this case the result lost the original order.

The solution below assume that both input files are sorted in ascending order using the same order of IF command’s comparison operators and that does not contain empty lines.

Not the answer you’re looking for? Browse other questions tagged batch-file or ask your own question.

Linked

Related

Hot Network Questions

Subscribe to RSS

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

site design / logo © 2021 Stack Exchange Inc; user contributions licensed under cc by-sa. rev 2021.4.16.39093

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

6 Ways To Combine or Merge Multiple Text Files

Many users will come into contact with text files in Windows on a daily basis. Whether it’s reading a Readme file, viewing system or application logs, editing configuration files or writing your own files. Text files can easily be viewed, created and edited using the built in Windows Notepad or more advanced programs like Notepad++. Making your own text file is simple and requires no coding or programming knowledge.

If you have a number of text files spread across your computer it can become a bit inconvenient to keep opening different files to read or edit specific text. What would make things easier is being able to merge a number of text files into one single file. This makes the text faster to read and search through. It’s especially useful for multiple logs or report files that could be spread across dozens or even hundreds of files.

Here we list 6 different tools and methods for you to be able to merge or combine multiple text based files into a single file to hopefully make things a little easier.

One advantage TXTCollector has over many other tools of this type is its ability to optionally include all the files inside subfolders and not just the main folder when merging. It also has a text filtering drop down box which can automatically add files of a specific type. They are BAT, LOG, XML, TXT, DAT, TAG, REG, INI, INF, HTM, CFG and CSV.

After installing TXTCollector, run it and browse for the folder containing your files, optionally include subfolders. All you have to do then is decide what type of separator to use, whether to add a simple line break between each text file or use no separation at all. Pressing Combine all files will ask you where to save the merged text file and what name to give it.

By default, TXTCollector cannot select all files but only files with specific file extensions. You can add custom extensions by clicking “Extensions and Separators” at the bottom and opening Extensions.txt in Notepad. Enter an asterisk (*) on its own line and then save the file. The extension drop down will now have a * which will select all files for merging. You can also add custom separators by editing the Separators.txt.

2. Text File Merge

While Text File Merge is an old tool dating to 2002, joining text files together isn’t complicated or operating system dependant and it still works fine today in Windows 10. It is a portable executable but comes as a Zip self extracting executable, so you need to browse for a folder and unzip the Text File Merge program before using it.

After browsing for the folder containing the text files, you can either press the Select All button, drag the mouse over a range of names or use Ctrl/Shift+Click to multi select. The box just below is for using a different extension filter such as *.log. The other options are for stripping lines from the top of text files or adding line spacing to the bottom. Each merged text file will be saved in the same folder with the name “merged file[number].txt”.

3. Text File Joiner

The website and developer of Text File Joiner is no longer around but the program still works and is simple enough to use. It’s only a 38KB standalone executable but the .NET Framework 4 is required which will need to be installed on Windows 7 and below operating systems.

Press button 1 to select and load the files and then button 2 to choose where to save the text file and what to call it. The order of files can be altered with the up and down buttons. You can also insert each file’s name and separate each text file with extra spacing. You have to select files to load and not whole folders so select everything in a folder or multiple files by using Ctrl+A or Shift/Ctrl+Click.

4. Use Notepad++ to Combine Multiple Text files

While the option to merge a number of text files into a single text file is not available as a standard feature in Notepad++, it can be added in by making use of the internal plugin system. It also works in a different way because only text files which are opened inside Notepad++ will be combined.

Firstly, download Notepad++ 32-bit version and install it or extract the portable version. Run the program and go to the Plugins menu > Plugins Admin. Find the Combine v1 plugin, check its box and click Install. After a compulsory restart of Notepad++, open all the files you want to merge and then go to Plugins > Combine > Start.

The Combine window will pop up and offer to insert the filename and a line of dashes before each text file and a line of asterisks with optional line feed after. By pressing OK a new text file will be created in Notepad++ with all opened text files combined, save it as a new TXT file. To rearrange the order of merged text, drag around the file tabs to the preferred order. The leftmost tab will be top, the rightmost will be the bottom text file.

Combine has a webpage where you can read more, download the plugin DLL separately or possibly make a donation if you find it useful.

Note: The Combine plugin does not work with the 64-bit version of Notepad++ and won’t be visible in the plugin list. It will also popup an error if you try to install the plugin manually. Make sure you use the 32-bit version of Notepad++, even if you are running a 64-bit Windows.

5. Merge Multiple Text Files Online

If your text files are relatively small and do not contain private or confidential information, using an online text file merging tool is a useful option. There are several online services out there that can merge text files. We’ll look at two that are essentially the same service with the same features, so you can use either one.

Visit Bear File Converter or Filemerge and you have the option of dropping text files onto the webpage, supplying a URL or browsing for files on your PC of up to 50MB.

Place files in a specific order using the sort area or delete them from the list. You can also add prefix text (before each file), suffix text (after each file), custom separator text, and choose between a DOS or UNIX output format. Then just press Merge and the new combined text file can be downloaded. The service also accepts around a dozen input file formats besides TXT but outputs only to TXT.

6. Merge Text Files Using Built-in Windows Tools

Windows has some methods to join multiple files together using command line tools such as Type and Copy. All you have to do is supply the right syntax to the command. For instance, open a Command Prompt, CD to the folder containing the text files and type:

That very simple command will read the contents of all the text files in the current folder and output it all to the combined text file. You can also do a similar thing with Copy:

The above will join all .TXT files together into one file called Combined.txt and has the same result as the Type command. The Copy command is quite powerful for joining files and you can join together just about anything including archives and video files. One drawback is if the text files don’t each have a line break at the end they will append directly onto the text from the previous file.

To stop that happening and split the text from each file with a line break you need to use something a bit more advanced:

For %f in (*.txt) do type “%f” >> Combined.txt & echo. >> Combined.txt

The above command will add an echo. after each file is copied into the text file which makes sure all text files start on a new line. Do note that you will need to change Combined.txt to a different file name if you run the command more than once in the same folder. Otherwise, all text files will simply append to the same file.

Add Line Breaks To The End Of Your Text Files

An option to make sure all your text files merge together properly is for each of them to contain a line break at the end. Then all methods will work without having to remember to add them yourself, even the simple Copy or Type commands.

Add a Line Break To All Files From The Command Line

This is a variation on the above command and will simply append a line feed to all TXT files in a folder.

Add a Line Break To All Files Using Notepad++

Notepad++ is also able to easily add a new line to the end of text files using the Replace function. Load all text files into Notepad++, go to Search > Replace (Ctrl+H) and change the following:

Replace with: \1\n

Search mode: Select “Regular expression” and enable “. matches newline”

Then press “Replace All in All Opened Documents” and all files will get a line feed at the end of the last line of text. This ensures anything appended to the text will start on its own line. Press Save All (Ctrl+Shift+S) to save all files when you’re finished.

Every subsequent press of the replace button will add another new line to each file. You can easily add 3 or 5 blank lines so separate files can be clearly recognized inside a long merged text file. Also, custom text can be added to the Replace with box after “\1\n”, such as “\1\n—END OF FILE—\n” or anything else you require.

Final Note: You might notice that if your files are numbered, sometimes they are not sorted in the proper numeric order. For example, the command line and some tools will order the files as file1, file10, file2 while Windows Explorer will order them as file1, file2, file10. The workaround is to pad the numbering with zero’s as file01, file02, file10 etc, which will add the files to the combined text file in proper numeric order.

8 Ways to Split and Join Large Files

As time moves on, the size of software and files found on your computer gets bigger. A photo will have more pixels in it and be larger, video files are now being distributed in 4K which means bigger file sizes than before. The same applies to games, some can now be as much as a 200GB download on platforms such as Steam. Applications are also growing, the latest Microsoft Office installation is now much bigger than it used to be.

As everything grows in size, it becomes more of a problem when you are trying to move or copy the files to a more restricted medium. For instance, you might be trying to send a video file via email which has an attachment size limit. Or you could be trying to backup or copy large files, such as ISO images, onto optical media or flash drives. If the file is too big for the space limit you have available, it needs to be split into separate parts.

Splitting files into smaller parts that can then be joined together again, later on, is done all the time on the internet. It’s also quite common to split a large file when you have to fit it onto several CDs, DVDs, or even floppies. Curiously Windows itself has the capacity to join already split files, but can’t split the file in the first place. For that you need a third party method, here are 8 ways for you to do it. All methods were tested on Windows 7 and 10.

FFSJ is small, efficient, and easy to use and comes as either a Lite or Standard version. Lite is a portable executable while the Standard installer version includes the context menu option that can split and join by right click. FFSJ also has a command line interface so files can be split and joined from batch scripts or the Command Prompt.

Splitting a file is easy and you only need to drop a file onto the window or manually locate it, choose a different output directory if required, and set the split method. The file can be split according to a specific number of parts or a specific size. The Encrypt data option is useful as only those that know the password can rejoin the files (you must rejoin the files with FFSJ).

For joining files you only have to drag and drop or locate the first file in the set and supply a save to directory. A password will be required if you encrypted the files. The MD5 Checksum tab allows you to get the checksum of the file before splitting, then a joined file can be verified to see if its checksum matches.

2. KFK

KFK is by KC Softwares who also make other well known tools like the software update monitor Sumo. In addition to splitting and joining files, KFK also has an option to burn the files to disc (if you have the shareware VSO CopytoDVD software) and also offers disk spanning by splitting a file over multiple floppy disks.

It’s possible to drop a file onto the KFK window or you can browse for one manually. For splitting, you can choose either to set a specific number of parts or a specific size of each part. The “Generate automatic rebuild file” checkbox creates a batch file with the files so you can join them together without using KFK.

Make sure to download the Lite or portable version of KFK. In the past, KC softwares have bundled adware with their installers. At the time of writing, they appear to be adware free but be aware when downloading.

3. GSplit 3

GSplit is definitely the program with the most advanced options here. Not only can it automatically span a file across several removable devices (floppies, USB flash drives, etc), GSplit can also create custom tags for the files and a custom self uniting program. There’s even a batch option where a number of files can be split one after the other (using the Several Files button).

There are four standard split methods; by numbers of files, by the size of each file, entering the size of each file individually, or even splitting a text file by line. The self uniting function (SFU) adds an executable to the files so they can be joined without GSplit.

The SFU file can be customized with your own messages, a picture, a chosen destination folder, run a program after splitting and auto start/end. The Express button is a faster way to quickly split a file from a single window. Both installer and portable versions of GSplit 3 are available.

4. Puran Splitter

Splitter is also part of the Puran Utilities tweaking software but this is the standalone version. It’s unfortunate that the program has to be installed but you can easily copy the folder from Program Files and then uninstall to create a portable version. Delete the file “Default.cjstyles” from the program folder if you prefer not to have the light blue theme.

Splitting a file is self explanatory and you simply drop or browse for the file to split, browse for an output folder, set the size of each file, and press the button. Sizes are in Kilobytes or Megabytes so multiply Megabytes by 1024 to use Gigabytes.

A useful feature is the option to either create a batch file to rejoin the files back together or create an executable. Either option works without the need for Puran Splitter. The batch join option uses the Windows Copy command which is useful to get around any restrictions that don’t allow an unknown executable to run on the computer.

5. Split Files Using an Archiving Program (PeaZip)

One of the most known methods of splitting files into multiple parts is by using a file archiver. Most archiving software has the ability to create a multi part Zip or 7z archive. We’re looking at PeaZip because it can actually split files by creating a multi part archive or using the standard method of splitting the file like the tools above. PeaZip is also a very good general archiver with a portable version if you would rather not install it.

To create a multi part archive, press Add, and drop the file onto the window. Choose where to save the split files, and in the dropdown boxes select Self-extracting 7Z, and Store. Select the split file size from one of the presets or choose Custom and enter a specific size in KB, MB, or GB.

With the self extracting 7Z you can join the files back together without an archiver installed. If that’s not required, use the 7Z or Zip methods instead. To create a standard split file, choose Split in the top drop down and pick the split size in the bottom drop down and start the process.

An alternative split window is in Tools > PeaUtils although here the output path is locked to the same as the input file. The PeaUtils window is also where you can join split parts together again. Click the drop down and select Join spanned file, then locate the first file in the set (.001) or drop it onto the window.

PeaUtils is a standalone program that doesn’t require PeaZip to work. Therefore, you can use split, join, or any other options available in the drop down without using or installing PeaZip.