Linux delete all lines from file

Your Own Linux.

Linux How To’s | Bash Scripting | Python

Monday, 20 April 2015

Sed Command in Linux — Delete Lines from a File

This is the third article of the «Super sed ‘ Series», in which we will learn how to delete lines from a file using line numbers and regular expressions. In the first two articles, we have learned, how to print lines using sed and how to append and insert lines in a file using sed. If you are familiar with sed print syntaxes, then this article would be pretty easy to understand.

For those learners who are new to sed , let’s have a brief introduction of the Super sed :

• sed stand for Stream EDitor and it being based on the ed editor, it borrows most of the commands from the ed . It was developed by Lee E. McMahon of Bell Labs.
• sed offers large range of text transformations that include printing lines, deleting lines, editing line in-place, search and replace, appending and inserting lines, etc.
• sed is useful whenever you need to perform common editing operations on multiple lines without using ‘vi’ editor.
• Whenever sed is executed on an input file or on the contents from stdin, sed reads the file line-by-line and after removing the trailing newline, places it in the «Pattern space», where the commands are executed on them after conditions (as in case of regex matching) are verified, and then printed on the stdout.

Deleting Lines from a File using sed

Before we start, just remember two points:

1. sed «d» command lets us print specific lines based on the line number or regex provided.
2. When ^ means beginning of the line and $ denotes end of the line, ^$ makes a «Blank Line», very useful while removing empty lines from a file.

For our better understanding, let us have a file sedtest.txt with contents as follows:

A. sed — Delete Lines with Line Number

1. Delete ‘N’th line

This will remove ‘N’th line in the FILE.txt .

Example:
To delete 1st line,

While, to remove last line,

If you run above commands and inspect the file sedtest.txt , you would find that, the original contents of that file would not change. In case you want to remove lines in the file and save the changes (i.e. edit the file in place), you will have to use the option -i as shown in below example:

2. Delete all Lines starting from ‘M’th up to ‘N’th

This will remove the block of lines starting at line number M and ending at line number N .

Example:
To delete block of lines from 3rd line to 8th line.

Similarly, in order to delete lines starting from 5th up to last line, you would run-

3. Delete Every ‘M’th Line Starting from ‘N’th Line

This will start from Nth line and delete every Mth line coming after that. Note that, Nth line will also be deleted.

Example:
To delete every alternate line staring from 2nd one.

B. sed — Delete Lines using Regular Expression/Pattern

1. Delete lines containing a specific Pattern

This will delete all those lines that contain the pattern provided.

2. Delete all those lines not containing the Pattern

This will delete all those lines which do not include the pattern provided.

3. Delete block of lines starting from pattern matching line

This will remove the lines from the one where pattern matches, till ‘N’th line.

Similarly, so as to delete all the lines starting from pattern matching line till the end, you would use ‘/PATTERN/,$ d’ as follows —

4. Delete block of lines starting from Nth and ending at pattern matching line

This will delete the lines starting from the ‘N’th line, till the one where pattern matches.

5. Delete block of lines between two Pattern matches

This will start deleting lines from 1st pattern match till 2nd pattern match.

6. Deleting all blank lines

7. Deleting lines containing either of two patterns

That was all about the third article on sed command. More articles on sed are coming soon. So, stay tuned. Of course, do not forget to share your feedback in the comment section below.

Источник

How to remove blank lines from a Unix file

I need to remove all the blank lines from an input file and write into an output file. Here is my data as below.

8 Answers 8

This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn’t support that you can write the output to a temporary file and replace the original:

If you also want to remove lines consisting only of whitespace (not just empty lines) then use:

Edit: also remove whitespace at the end of lines, because apparently you’ve decided you need that too:

The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.

Use grep to match any line that has nothing between the start anchor ( ^ ) and the end anchor ( $ ):

If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways:

or, without Perl regular expressions:

Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you’re using GNU or BSD sed , you can do the edit in-place, if that’s what you want, with the -i option.

If I execute the above command still I have blank lines in my output file. What could be the reason?

There could be several reasons. It might be that you don’t have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that’s the problem, then:

The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs.

Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn’t a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete ( -d ) character code octal 15, aka control-M or \r or carriage return:

If neither of those works, then you need to show a hex dump or octal dump ( od -c ) of the first two lines of the file, so we can see what we’re up against:

Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.)

You can try the POSIX named character classes such as sed -e ‘/^[[:space:]]*$/d’ ; it will probably work, but is not guaranteed. You can try it with:

If it works, there’ll be three spaces between the ‘Hello’ and the ‘World’. If not, you’ll probably get an error from sed . That might save you grief over getting tabs typed on the command line.

Источник

How to remove the lines which appear on file B from another file A?

I have a large file A (consisting of emails), one line for each mail. I also have another file B that contains another set of mails.

Which command would I use to remove all the addresses that appear in file B from the file A.

So, if file A contained:

and file B contained:

Then file A should be left with:

Now I know this is a question that might have been asked more often, but I only found one command online that gave me an error with a bad delimiter.

Any help would be much appreciated! Somebody will surely come up with a clever one-liner, but I’m not the shell expert.

11 Answers 11

If the files are sorted (they are in your example):

-23 suppresses the lines that are in both files, or only in file 2. If the files are not sorted, pipe them through sort first.

grep -Fvxf

• works on non-sorted files
• maintains the order
• is POSIX

• -F : use literal strings instead of the default BRE
• -x : only consider matches that match the entire line
• -v : print non-matching
• -f file : take patterns from the given file

This method is slower on pre-sorted files than other methods, since it is more general. If speed matters as well, see: Fast way of finding lines in one file that are not in another?

Here’s a quick bash automation for in-line operation:

awk to the rescue!

This solution doesn’t require sorted inputs. You have to provide fileB first.

How does it work?

NR==FNR idiom is for storing the first file in an associative array as keys for a later «contains» test.

NR==FNR is checking whether we’re scanning the first file, where the global line counter (NR) equals to the current file line counter (FNR).

a[$0] adds the current line to the associative array as key, note that this behaves like a set, where there won’t be any duplicate values (keys)

!($0 in a) we’re now in the next file(s), in is a contains test, here it’s checking whether current line is in the set we populated in the first step from the first file, ! negates the condition. What is missing here is the action, which by default is and usually not written explicitly.

Note that this can now be used to remove blacklisted words.

with a slight change it can clean multiple lists and create cleaned versions.

Источник

How can I delete all lines in a file using vi?

How can I delete all lines in a file using vi?

At moment I do that using something like this to remove all lines in a file:

How can I delete all lines using vi ?

Note: Using dd is not a good option. There can be many lines.

14 Answers 14

to delete all lines.

The : introduces a command (and moves the cursor to the bottom).
The 1,$ is an indication of which lines the following command ( d ) should work on. In this case the range from line one to the last line (indicated by $ , so you don’t need to know the number of lines in the document).
The final d stands for delete the indicated lines.

There is a shorter form ( :%d ) but I find myself never using it. The :1,$d can be more easily «adapted» to e.g. :4,$-2d leaving only the first 3 and last 2 lines, deleting the rest.

• : tells vi to go in command mode
• % means all the lines
• d : delete

On the command line,

What is the problem with dd?

• /dev/null is a special 0 byte file
• if is the input file
• of is the ouput file

I’d recommend that you just do this (should work in any POSIX-compliant shell):

If you really want to do it with vi, you can do:

• 1G (go to first line)
• dG (delete to last line)

If your cursor is on the first line (if not, type: gg or 1G ), then you can just use dG . It will delete all lines from the current line to the end of file. So to make sure that you’ll delete all the lines from the file, you may mix both together, which would be: ggdG (while in command mode).

Or %d in Ex mode, command-line example: vim +%d foo.bar .

I’m a lazy dude, and I like to keep it simple. ggdG is five keystrokes including Shift

gg goes to the first line in the file, d is the start of the d elete verb and G is the movement to go to the bottom of the file. Verbosely, it’s go to the beginning of the file and delete everything until the end of the tile.

Go to the beginning of the file and press d G .

I always use ggVG

• gg jumps to the start of the current editing file
• V (capitalized v) will select the current line. In this case the first line of the current editing file
• G (capitalized g) will jump to the end of the file. In this case, since I selected the first line, G will select the whole text in this file.

Then you can simply press d or x to delete all the lines.

note that in your question, echo > test.txt creates a file with a single line break in it, not an empty file.

From the shell, consider using echo -n > test.txt or : > test.txt .

While I’d generally use a vi editing command (I use ggdG ), you can also call out to the shell with a reference to the current file like so:

It’s nearly as concise as ggdG , but harder to type, and you also have to confirm that you want to reload the modified file, so I don’t particularly recommend it in this case, but knowing how to use shell commands from vi like this is useful.

breaking it down:

• : initiate a vi command
• ! initate a shell command
• : this is a shell builtin command with empty output
• > redirect the output
• % vi substitutes this with the name of the current file

The suggested :1,$d is also a good one of course, and just while I’m at it there’s also 1GdG

Источник

Remove the last line from a file in Bash

I have a file, foo.txt , containing the following lines:

I want a simple command that results in the contents of foo.txt being:

15 Answers 15

The -i option does not exist in GNU sed versions older than 3.95, so you have to use it as a filter with a temporary file:

Of course, in that case you could also use head -n -1 instead of sed .

MacOS:

On Mac OS X (as of 10.7.4), the equivalent of the sed -i command above is

This is by far the fastest and simplest solution, especially on big files:

if You want to delete the top line use this:

which means output lines starting at line 2.

Do not use sed for deleting lines from the top or bottom of a file — it’s very very slow if the file is large.

For large files

I had trouble with all the answers here because I was working with a HUGE file (

300Gb) and none of the solutions scaled. Here’s my solution:

Or alternatively, as a one liner:

In words: Find out the length of the file you want to end up with (length of file minus length of length of its last line, using bc ), and set that position to be the end of the file (by dd ing one byte of /dev/null onto it).

This is fast because tail starts reading from the end, and dd will overwrite the file in place rather than copy (and parse) every line of the file, which is what the other solutions do.

NOTE: This removes the line from the file in place! Make a backup or test on a dummy file before trying it out on your own file!

Источник