Linux remove line from all files

Your Own Linux.

Linux How To’s | Bash Scripting | Python

Monday, 20 April 2015

Sed Command in Linux — Delete Lines from a File

This is the third article of the «Super sed ‘ Series», in which we will learn how to delete lines from a file using line numbers and regular expressions. In the first two articles, we have learned, how to print lines using sed and how to append and insert lines in a file using sed. If you are familiar with sed print syntaxes, then this article would be pretty easy to understand.

For those learners who are new to sed , let’s have a brief introduction of the Super sed :

• sed stand for Stream EDitor and it being based on the ed editor, it borrows most of the commands from the ed . It was developed by Lee E. McMahon of Bell Labs.
• sed offers large range of text transformations that include printing lines, deleting lines, editing line in-place, search and replace, appending and inserting lines, etc.
• sed is useful whenever you need to perform common editing operations on multiple lines without using ‘vi’ editor.
• Whenever sed is executed on an input file or on the contents from stdin, sed reads the file line-by-line and after removing the trailing newline, places it in the «Pattern space», where the commands are executed on them after conditions (as in case of regex matching) are verified, and then printed on the stdout.

Deleting Lines from a File using sed

Before we start, just remember two points:

1. sed «d» command lets us print specific lines based on the line number or regex provided.
2. When ^ means beginning of the line and $ denotes end of the line, ^$ makes a «Blank Line», very useful while removing empty lines from a file.

For our better understanding, let us have a file sedtest.txt with contents as follows:

A. sed — Delete Lines with Line Number

1. Delete ‘N’th line

This will remove ‘N’th line in the FILE.txt .

Example:
To delete 1st line,

While, to remove last line,

If you run above commands and inspect the file sedtest.txt , you would find that, the original contents of that file would not change. In case you want to remove lines in the file and save the changes (i.e. edit the file in place), you will have to use the option -i as shown in below example:

2. Delete all Lines starting from ‘M’th up to ‘N’th

This will remove the block of lines starting at line number M and ending at line number N .

Example:
To delete block of lines from 3rd line to 8th line.

Similarly, in order to delete lines starting from 5th up to last line, you would run-

3. Delete Every ‘M’th Line Starting from ‘N’th Line

This will start from Nth line and delete every Mth line coming after that. Note that, Nth line will also be deleted.

Example:
To delete every alternate line staring from 2nd one.

B. sed — Delete Lines using Regular Expression/Pattern

1. Delete lines containing a specific Pattern

This will delete all those lines that contain the pattern provided.

2. Delete all those lines not containing the Pattern

This will delete all those lines which do not include the pattern provided.

3. Delete block of lines starting from pattern matching line

This will remove the lines from the one where pattern matches, till ‘N’th line.

Similarly, so as to delete all the lines starting from pattern matching line till the end, you would use ‘/PATTERN/,$ d’ as follows —

4. Delete block of lines starting from Nth and ending at pattern matching line

This will delete the lines starting from the ‘N’th line, till the one where pattern matches.

5. Delete block of lines between two Pattern matches

This will start deleting lines from 1st pattern match till 2nd pattern match.

6. Deleting all blank lines

7. Deleting lines containing either of two patterns

That was all about the third article on sed command. More articles on sed are coming soon. So, stay tuned. Of course, do not forget to share your feedback in the comment section below.

Источник

Remove line of text from multiple files in Linux

Is there a simple way to remove the same line of text from a folder full of text documents at the command line?

5 Answers 5

If your version of sed allows the -i.bak flag (edit in place):

If not, simply put it in a bash loop:

To find a pattern and remove the line containing the pattern below command can be used

Example : if you want to remove the line containing word sleep in all the xml files

NOTE : Be careful before choosing a pattern as it will delete the line recursively in all the files in the current directory hierarchy 🙂

Consider grep -v:

Grep -v shows all lines except those that match, they go into a temp file, and then the tmpfile is moved back to the old file name.

This tells perl to loop over your file (-n), edit in place (-i), and print the line if it does not match your regular expression.

Somewhat related, here’s a handy way to perform a substitution over several files.

I wrote a Perl script for this:

Note that it removes lines based on a regex. If you wanted to do exact match, the inner foreach would look like:

Источник

Unix Sed Command to Delete Lines in File — 15 Examples

Sed Command to Delete Lines: Sed command can be used to delete or remove specific lines which matches a given pattern or in a particular position in a file. Here we will see how to delete lines using sed command with various examples.

The following file contains a sample data which is used as input file in all the examples:

Sed Command to Delete Lines — Based on Position in File

In the following examples, the sed command removes the lines in file that are in a particular position in a file.

1. Delete first line or header line

The d option in sed command is used to delete a line. The syntax for deleting a line is:

Here N indicates Nth line in a file. In the following example, the sed command removes the first line in a file.

2. Delete last line or footer line or trailer line

The following sed command is used to remove the footer line in a file. The $ indicates the last line of a file.

3. Delete particular line

This is similar to the first example. The below sed command removes the second line in a file.

4. Delete range of lines

The sed command can be used to delete a range of lines. The syntax is shown below:

Here m and n are min and max line numbers. The sed command removes the lines from m to n in the file. The following sed command deletes the lines ranging from 2 to 4:

5. Delete lines other than the first line or header line

Use the negation (!) operator with d option in sed command. The following sed command removes all the lines except the header line.

6. Delete lines other than last line or footer line

7. Delete lines other than the specified range

Here the sed command removes lines other than 2nd, 3rd and 4th.

8. Delete first and last line

You can specify the list of lines you want to remove in sed command with semicolon as a delimiter.

9. Delete empty lines or blank lines

The ^$ indicates sed command to delete empty lines. However, this sed do not remove the lines that contain spaces.

Sed Command to Delete Lines — Based on Pattern Match

In the following examples, the sed command deletes the lines in file which match the given pattern.

10. Delete lines that begin with specified character

^ is to specify the starting of the line. Above sed command removes all the lines that start with character ‘u’.

11. Delete lines that end with specified character

$ is to indicate the end of the line. The above command deletes all the lines that end with character ‘x’.

12. Delete lines which are in upper case or capital letters

13. Delete lines that contain a pattern

14. Delete lines starting from a pattern till the last line

Here the sed command removes the line that matches the pattern fedora and also deletes all the lines to the end of the file which appear next to this matching line.

15. Delete last line only if it contains the pattern

Here $ indicates the last line. If you want to delete Nth line only if it contains a pattern, then in place of $ place the line number.

Note: In all the above examples, the sed command prints the contents of the file on the unix or linux terminal by removing the lines. However the sed command does not remove the lines from the source file. To Remove the lines from the source file itself, use the -i option with sed command.

If you dont wish to delete the lines from the original source file you can redirect the output of the sed command to another file.

If you like this article, then please share it or click on the google +1 button.

Источник

How do I remove newlines from a text file?

I have the following data, and I need to put it all into one line.

None of these commands is working perfectly.

Most of them let the data look like this:

20 Answers 20

Edit:

If none of the commands posted here are working, then you have something other than a newline separating your fields. Possibly you have DOS/Windows line endings in the file (although I would expect the Perl solutions to work even in that case)?

If that doesn’t work then you’re going to have to inspect your file more closely (e.g. in a hex editor) to find out what characters are actually in there that you want to remove.

Must do the job.

This page here has a bunch of other methods to remove newlines.

edited to remove feline abuse 🙂

You can edit the file in vim:

to figure WHAT is the offending character. then use

Use sed with POSIX classes

This will remove all lines containing only whitespace (spaces & tabs)

sed ‘/^[[:space:]]*$/d’

Just take whatever you are working with and pipe it to that

Example

cat filename | sed ‘/^[[:space:]]*$/d’

Expanding on a previous answer, this removes all new lines and saves the result to a new file (thanks to @tripleee):

Which is better than a «useless cat» (see comments):

Also useful for getting rid of new lines at the end of the file, e.g. created by using echo blah > file.txt .

Note that the destination filename is different, important, otherwise you’ll wipe out the original content!

Was having the same case today, super easy in vim or nvim, you can use gJ to join lines. For your use case, just do

this will join all your 99 lines. You can adjust the number 99 as need according to how many lines to join. If just join 1 line, then only gJ is good enough.

If the data is in file.txt, then:

The ‘ $( ‘ reads the file and gives the contents as a series of words which ‘echo’ then echoes with a space between them. The ‘tr’ command then deletes any spaces:

xargs consumes newlines as well (but adds a final trailing newline):

Assuming you only want to keep the digits and the semicolons, the following should do the trick assuming there are no major encoding issues, though it will also remove the very last «newline»:

You can easily modify the above to include other characters, e.g. if you want to retain decimal points, commas, etc.

Nerd fact: use ASCII instead.

(Edited cause i didn’t see the friggin’ answer that had same solution, only difference was that mine had ASCII)

Using the gedit text editor (3.18.3)

1. Click Search
2. Click Find and Replace.
3. Enter \n\s into Find field
4. Leave Replace with blank (nothing)
5. Check Regular expression box
6. Click the Find button

Note: this doesn’t exactly address the OP’s original, 7 year old problem but should help some noob linux users (like me) who find their way here from the SE’s with similar «how do I get my text all on one line» questions.

Источник

Remove the last line from a file in Bash

I have a file, foo.txt , containing the following lines:

I want a simple command that results in the contents of foo.txt being:

15 Answers 15

The -i option does not exist in GNU sed versions older than 3.95, so you have to use it as a filter with a temporary file:

Of course, in that case you could also use head -n -1 instead of sed .

MacOS:

On Mac OS X (as of 10.7.4), the equivalent of the sed -i command above is

This is by far the fastest and simplest solution, especially on big files:

if You want to delete the top line use this:

which means output lines starting at line 2.

Do not use sed for deleting lines from the top or bottom of a file — it’s very very slow if the file is large.

For large files

I had trouble with all the answers here because I was working with a HUGE file (

300Gb) and none of the solutions scaled. Here’s my solution:

Or alternatively, as a one liner:

In words: Find out the length of the file you want to end up with (length of file minus length of length of its last line, using bc ), and set that position to be the end of the file (by dd ing one byte of /dev/null onto it).

This is fast because tail starts reading from the end, and dd will overwrite the file in place rather than copy (and parse) every line of the file, which is what the other solutions do.

NOTE: This removes the line from the file in place! Make a backup or test on a dummy file before trying it out on your own file!

Источник