How to find out line-endings in a text file?

I’m trying to use something in bash to show me the line endings in a file printed rather than interpreted. The file is a dump from SSIS/SQL Server being read in by a Linux machine for processing.

Are there any switches within vi , less , more , etc?

In addition to seeing the line-endings, I need to know what type of line end it is ( CRLF or LF ). How do I find that out?

11 Answers 11

You can use the file utility to give you an indication of the type of line endings.

To convert from «DOS» to Unix:

To convert from Unix to «DOS»:

Converting an already converted file has no effect so it’s safe to run blindly (i.e. without testing the format first) although the usual disclaimers apply, as always.

simple cat -e works just fine.

This displays Unix line endings ( \n or LF) as $ and Windows line endings ( \r\n or CRLF) as ^M$ .

:set list to see line-endings.

:set nolist to go back to normal.

While I don’t think you can see \n or \r\n in vi , you can see which type of file it is (UNIX, DOS, etc.) to infer which line endings it has.

Alternatively, from bash you can use od -t c or just od -c to display the returns.

In the bash shell, try cat -v . This should display carriage-returns for windows files.

(This worked for me in rxvt via Cygwin on Windows XP).

Editor’s note: cat -v visualizes \r (CR) chars. as ^M . Thus, line-ending \r\n sequences will display as ^M at the end of each output line. cat -e will additionally visualize \n , namely as $ . ( cat -et will additionally visualize tab chars. as ^I .)

To show CR as ^M in less use less -u or type — u once less is open.

Try file then file -k then dos2unix -ih

file will usually be enough. But for tough cases try file -k or dosunix -ih .

Try file -k

Short version: file -k somefile.txt will tell you.

• It will output with CRLF line endings for DOS/Windows line endings.
• It will output with LF line endings for MAC line endings.
• And for Linux/Unix line «CR» it will just output text . (So if it does not explicitly mention any kind of line endings then this implicitly means: «CR line endings».)

Long version see below.

Real world example: Certificate Encoding

I sometimes have to check this for PEM certificate files.

The trouble with regular file is this: Sometimes it’s trying to be too smart/too specific.

Let’s try a little quiz: I’ve got some files. And one of these files has different line endings. Which one?

(By the way: this is what one of my typical «certificate work» directories looks like.)

Let’s try regular file :

Huh. It’s not telling me the line endings. And I already knew that those were cert files. I didn’t need «file» to tell me that.

What else can you try?

You might try dos2unix with the —info switch like this:

So that tells you that: yup, «0.example.end.cer» must be the odd man out. But what kind of line endings are there? Do you know the dos2unix output format by heart? (I don’t.)

But fortunately there’s the —keep-going (or -k for short) option in file :

Excellent! Now we know that our odd file has DOS ( CRLF ) line endings. (And the other files have Unix ( LF ) line endings. This is not explicit in this output. It’s implicit. It’s just the way file expects a «regular» text file to be.)

(If you wanna share my mnemonic: «L» is for «Linux» and for «LF».)

Now let’s convert the culprit and try again:

Good. Now all certs have Unix line endings.

Try dos2unix -ih

I didn’t know this when I was writing the example above but:

Actually it turns out that dos2unix will give you a header line if you use -ih (short for —info=h ) like so:

And another «actually» moment: The header format is really easy to remember: Here’s two mnemonics:

1. It’s DUMB (left to right: d for Dos, u for Unix, m for Mac, b for BOM).
2. And also: «DUM» is just the alphabetical ordering of D, U and M.

Further reading

You can use xxd to show a hex dump of the file, and hunt through for «0d0a» or «0a» chars.

You can use cat -v as @warriorpostman suggests.

You may use the command todos filename to convert to DOS endings, and fromdos filename to convert to UNIX line endings. To install the package on Ubuntu, type sudo apt-get install tofrodos .

You can use vim -b filename to edit a file in binary mode, which will show ^M characters for carriage return and a new line is indicative of LF being present, indicating Windows CRLF line endings. By LF I mean \n and by CR I mean \r . Note that when you use the -b option the file will always be edited in UNIX mode by default as indicated by [unix] in the status line, meaning that if you add new lines they will end with LF, not CRLF. If you use normal vim without -b on a file with CRLF line endings, you should see [dos] shown in the status line and inserted lines will have CRLF as end of line. The vim documentation for fileformats setting explains the complexities.

Also, I don’t have enough points to comment on the Notepad++ answer, but if you use Notepad++ on Windows, use the View / Show Symbol / Show End of Line menu to display CR and LF. In this case LF is shown whereas for vim the LF is indicated by a new line.

I dump my output to a text file. I then open it in notepad ++ then click the show all characters button. Not very elegant but it works.

Git status ignore line endings / identical files / windows & linux environment / dropbox / mled

ignore line ending differences?

I use randomly Windows and Linux to work on the project. The project is in Dropbox.

I found a lot about how do make git diff ignore line endings. Since i use meld git diff opens meld for each file. And meld says «identical file».

So how do I avoid this. Git should only open meld for changed files. And git status should not report files as changed if only the file ending is different.

EDIT: Cause:

This happened because of this setting on Windows

So I checked out the working copy on Linux and set core.autocrlf false on Windows.

It would be still nice to know how to make git status ignore different new lines.

6 Answers 6

Try setting core.autocrlf value like this :

This answer seems relevant since the OP makes reference to a need for a multi-OS solution. This Github help article details available approaches for handling lines endings cross-OS. There are global and per-repo approaches to managing cross-os line endings.

Global approach

Configure Git line endings handling on Linux or OS X:

Configure Git line endings handling on Windows:

Per-repo approach:

In the root of your repo, create a .gitattributes file and define line ending settings for your project files, one line at a time in the following format: path_regex line-ending-settings where line-ending-settings is one of the following:

• text
• binary (files that Git should not modify line endings for — as this can cause some image types such as PNGs not to render in a browser)

The text value can be configured further to instruct Git on how to handle line endings for matching files:

• text — Changes line endings to OS native line endings.
• text eol=crlf — Converts line endings to CRLF on checkout.
• text eol=lf — Converts line endings to LF on checkout.
• text=auto — Sensible default that leaves line handle up to Git’s discretion.

Here is the content of a sample .gitattributes file:

More on how to refresh your repo after changing line endings settings here. Tldr:

backup your files with Git, delete every file in your repository (except the .git directory), and then restore the files all at once. Save your current files in Git, so that none of your work is lost.

git commit -m «Saving files before refreshing line endings»

Remove the index and force Git to rescan the working directory.

Rewrite the Git index to pick up all the new line endings.

Show the rewritten, normalized files.

In some cases, this is all that needs to be done. Others may need to complete the following additional steps:

Add all your changed files back, and prepare them for a commit. This is your chance to inspect which files, if any, were unchanged.

It is perfectly safe to see a lot of messages here that read[s] «warning: CRLF will be replaced by LF in file.»

Commit the changes to your repository.

git commit -m «Normalize all the line endings»

Convert Unix line endings to Windows

I recently moved back to Windows from Linux. I have some files with CRLFs, some with LFs and some that are mixed. Is there a utility that will help me find all my Unix-touched files and convert them to proper CRLF terminated files?

The utility must run on Windows, not Linux. I have already moved. I’d rather not install Cygwin if I can avoid it.

14 Answers 14

You can convert them with the unix2dos utility on your Linux platform. There are unix2dos versions available for Windows as well.

If you have Perl installed you can also use this one liner:

Here is an easy and quick way.

Drag and drop the text file into Chrome (I don’t know about other browsers) and then cut and paste back into the original file 🙂

The one I found best for recursively going through folders, allowing file filters and allowing a simple search for «\r\n» and replacing it with just «\n» was Notepad++.

Notepad++ is one of the best, free, open source notepad programs for Windows. It is very simple and powerful. It handled the line ending search/replace just fine. A contractor check a bunch of .c and .h files in to our repository with Linux \r\n line endings, but since most people have standardized on Windows/Eclipse build tools, the files won’t build until the line endings are converted.

For example: sfk addcr -dir . -file .txt -norec
changes LF endings into CR/LF for Windows, on all .txt files of the current directory, but NOT within subdirectories (no recursion).

But this program does a lot more than just that.

On Cygwin, you can convert between Unix and «DOS» AKA Windows files using two built-in utilities:

Convert to DOS CR/LF format:

Convert back to Unix CR format:

The file is left in place with the same name.

I’m going to throw this solution out there. Git will do this. See this post about it

So theoretically you could do this to convert an entire tree

Change crlf to lf if you want to go the other way. NOTE: you’re not done yet, keep reading

Type git status to see which files will be affected. You might have to add lines like

etc to .gitattributes to avoid converting certain files. You can also explicit mark certain files as text

Then just repeat these 2 lines after you’ve edited .gitattributes

Then use git status again to see which files will be changed. When you’re sure all the files you want affected are listed by git status then commit

now check all the files out again

They should now have whatever your desired line endings are

** NOTE: If you were already using git skip the first 3 commands git commands. If you were not using git you can now delete the .gitattributes file and the .git folder.

** Back up your files: the git rm —cached -r deletes them all (although they are theoretically in your git repo (the .git folder) which is how they get restored by the last command git reset —hard . It’s just since files are getting deleted it’s probably best to back them up.

How to convert Windows end of line in Unix end of line (CR/LF to LF)

I’m a Java developer and I’m using Ubuntu to develop. The project was created in Windows with Eclipse and it’s using the Windows-1252 encoding.

To convert to UTF-8 I’ve used the recode program:

This command gives this error:

Convert line endings from CR/LF to a single LF: Edit the file with Vim, give the command :set ff=unix and save the file. Recode now should run without errors.

Nice, but I’ve many files to remove the CR/LF character from, and I can’t open each to do it. Vi doesn’t provide any option to command line for Bash operations.

Can sed be used to do this? How?

8 Answers 8

There should be a program called dos2unix that will fix line endings for you. If it’s not already on your Linux box, it should be available via the package manager.

sed cannot match \n because the trailing newline is removed before the line is put into the pattern space, but it can match \r , so you can convert \r\n (DOS) to \n (Unix) by removing \r:

Warning: this will change the original file

However, you cannot change from Unix EOL to DOS or old Mac ( \r ) by this. More readings here:

Actually, Vim does allow what you’re looking for. Enter Vim, and type the following commands:

The first of these commands sets the argument list to every file matching **/*.java , which is all Java files, recursively. The second of these commands does the following to each file in the argument list, in turn:

• Sets the line-endings to Unix style (you already know this)
• Writes the file out iff it’s been changed
• Proceeds to the next file

The tr command can also do this:

and should be available to you.

You’ll need to run tr from within a script, since it cannot work with file names. For example, create a file myscript.sh:

Running myscript.sh would process all the java files in the current directory and its subdirectories.

I’ll take a little exception to jichao’s answer. You can actually do everything he just talked about fairly easily. Instead of looking for a \n , just look for carriage return at the end of the line.

To change from Unix back to DOS, simply look for the last character on the line and add a form feed to it. (I’ll add -r to make this easier with grep regular expressions.)

Theoretically, the file could be changed to Mac style by adding code to the last example that also appends the next line of input to the first line until all lines have been processed. I won’t try to make that example here, though.

Warning: -i changes the actual file. If you want a backup to be made, add a string of characters after -i . This will move the existing file to a file with the same name with your characters added to the end.