Linux remove first line from file

How do I delete the first n lines of an ascii file using shell commands?

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

If it’s any help, I’m using RedHat and an Ubuntu linux systems.

7 Answers 7

As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile . This won’t echo anything, it will just modify the file in-place. If you don’t need to pipe the result to another command, this is easier.

sed -i ‘1,3d’ file.txt

This deletes first 3 line from file.txt.

1GB left of space, this solution won’t work as creates a tmp file

If the tabulated lines are the ones that have a tab character:

( ␉ being a literal tab character) or equivalently

In a bash/ksh/zsh script, you can write $’\t’ for a tab, e.g. grep $’\t’ or sed -n $’/\t/p’ .

If you want to eliminate 10 lines at the beginning of the file:

(note that it’s +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or

On Linux, you can take advantage of GNU sed’s -i option to modify files in place:

Or you can use a shell loop and temporary files:

Or if you don’t want to modify the files in place, but instead give them a different name:

Источник

Remove First n Lines of a Large Text File

I need to remove the first 42 lines of a 2GB SQL dump.

I know I can view the first lines using:

But is there anyway to edit or remove them?

8 Answers 8

If you want to just view the lines from the 43rd on you can use

The + sign is important — without it, tail will print the last 43 lines instead. Alternatively with ‘sed’

If you want to really delete the first 42 lines from the original file then you can make sed make the change inplace with the -i option

700MB. Takes awhile for the file to load, though.

This seems to be the easiest:

Remove lines 1-42 from test.sql and save as test2.sql

tail -n +43 dump.sql > dump_new.sql

You can use Vim in Ex mode:

1 move to first line

42 select 42 lines

x save and close

Because of sed discrepancies across Linux and Mac, I resolved to use tail -n +43 dump.sql > new.sql format.

Just to add this. If you’re on a mac you need to add the backup extension. Answer from this post.

Sorry, I can’t give you actual code right now. However, try looking at something along the lines of

What this should do (once properly formatted) is count the number of lines in the file (wc -l), subtract 44 from it (-44) and then print out everything starting with the 45th line in the file.

Источник

How can I remove the first line of a text file using bash/sed script?

I need to repeatedly remove the first line from a huge text file using a bash script.

Right now I am using sed -i -e «1d» $FILE — but it takes around a minute to do the deletion.

Is there a more efficient way to accomplish this?

100MB. MacBook Air 2013 with SSD.

16 Answers 16

-n x : Just print the last x lines. tail -n 5 would give you the last 5 lines of the input. The + sign kind of inverts the argument and make tail print anything but the first x-1 lines. tail -n +1 would print the whole file, tail -n +2 everything but the first line, etc.

GNU tail is much faster than sed . tail is also available on BSD and the -n +2 flag is consistent across both tools. Check the FreeBSD or OS X man pages for more.

The BSD version can be much slower than sed , though. I wonder how they managed that; tail should just read a file line by line while sed does pretty complex operations involving interpreting a script, applying regular expressions and the like.

Note: You may be tempted to use

but this will give you an empty file. The reason is that the redirection ( > ) happens before tail is invoked by the shell:

1. Shell truncates file $FILE
2. Shell creates a new process for tail
3. Shell redirects stdout of the tail process to $FILE
4. tail reads from the now empty $FILE

If you want to remove the first line inside the file, you should use:

The && will make sure that the file doesn’t get overwritten when there is a problem.

You can use -i to update the file without using ‘>’ operator. The following command will delete the first line from the file and save it to the file.

For those who are on SunOS which is non-GNU, the following code will help:

You can easily do this with:

on the command line; or to remove the first line of a file permanently, use the in-place mode of sed with the -i flag:

No, that’s about as efficient as you’re going to get. You could write a C program which could do the job a little faster (less startup time and processing arguments) but it will probably tend towards the same speed as sed as files get large (and I assume they’re large if it’s taking a minute).

But your question suffers from the same problem as so many others in that it pre-supposes the solution. If you were to tell us in detail what you’re trying to do rather then how, we may be able to suggest a better option.

For example, if this is a file A that some other program B processes, one solution would be to not strip off the first line, but modify program B to process it differently.

Let’s say all your programs append to this file A and program B currently reads and processes the first line before deleting it.

You could re-engineer program B so that it didn’t try to delete the first line but maintains a persistent (probably file-based) offset into the file A so that, next time it runs, it could seek to that offset, process the line there, and update the offset.

Then, at a quiet time (midnight?), it could do special processing of file A to delete all lines currently processed and set the offset back to 0.

It will certainly be faster for a program to open and seek a file rather than open and rewrite. This discussion assumes you have control over program B, of course. I don’t know if that’s the case but there may be other possible solutions if you provide further information.

Источник

Your Own Linux.

Linux How To’s | Bash Scripting | Python

Monday, 20 April 2015

Sed Command in Linux — Delete Lines from a File

This is the third article of the «Super sed ‘ Series», in which we will learn how to delete lines from a file using line numbers and regular expressions. In the first two articles, we have learned, how to print lines using sed and how to append and insert lines in a file using sed. If you are familiar with sed print syntaxes, then this article would be pretty easy to understand.

For those learners who are new to sed , let’s have a brief introduction of the Super sed :

• sed stand for Stream EDitor and it being based on the ed editor, it borrows most of the commands from the ed . It was developed by Lee E. McMahon of Bell Labs.
• sed offers large range of text transformations that include printing lines, deleting lines, editing line in-place, search and replace, appending and inserting lines, etc.
• sed is useful whenever you need to perform common editing operations on multiple lines without using ‘vi’ editor.
• Whenever sed is executed on an input file or on the contents from stdin, sed reads the file line-by-line and after removing the trailing newline, places it in the «Pattern space», where the commands are executed on them after conditions (as in case of regex matching) are verified, and then printed on the stdout.

Deleting Lines from a File using sed

Before we start, just remember two points:

1. sed «d» command lets us print specific lines based on the line number or regex provided.
2. When ^ means beginning of the line and $ denotes end of the line, ^$ makes a «Blank Line», very useful while removing empty lines from a file.

For our better understanding, let us have a file sedtest.txt with contents as follows:

A. sed — Delete Lines with Line Number

1. Delete ‘N’th line

This will remove ‘N’th line in the FILE.txt .

Example:
To delete 1st line,

While, to remove last line,

If you run above commands and inspect the file sedtest.txt , you would find that, the original contents of that file would not change. In case you want to remove lines in the file and save the changes (i.e. edit the file in place), you will have to use the option -i as shown in below example:

2. Delete all Lines starting from ‘M’th up to ‘N’th

This will remove the block of lines starting at line number M and ending at line number N .

Example:
To delete block of lines from 3rd line to 8th line.

Similarly, in order to delete lines starting from 5th up to last line, you would run-

3. Delete Every ‘M’th Line Starting from ‘N’th Line

This will start from Nth line and delete every Mth line coming after that. Note that, Nth line will also be deleted.

Example:
To delete every alternate line staring from 2nd one.

B. sed — Delete Lines using Regular Expression/Pattern

1. Delete lines containing a specific Pattern

This will delete all those lines that contain the pattern provided.

2. Delete all those lines not containing the Pattern

This will delete all those lines which do not include the pattern provided.

3. Delete block of lines starting from pattern matching line

This will remove the lines from the one where pattern matches, till ‘N’th line.

Similarly, so as to delete all the lines starting from pattern matching line till the end, you would use ‘/PATTERN/,$ d’ as follows —

4. Delete block of lines starting from Nth and ending at pattern matching line

This will delete the lines starting from the ‘N’th line, till the one where pattern matches.

5. Delete block of lines between two Pattern matches

This will start deleting lines from 1st pattern match till 2nd pattern match.

6. Deleting all blank lines

7. Deleting lines containing either of two patterns

That was all about the third article on sed command. More articles on sed are coming soon. So, stay tuned. Of course, do not forget to share your feedback in the comment section below.

Источник

How to delete lines from a file using sed command

sed command stands for Stream Editor. It is used to perform basic text transformations in Linux.

sed is one of the important command, which plays major role in file manipulations. It can be used for many purposes some of which are listed below:

• To delete or remove specific lines which matches with given pattern.
• To remove a particular line based on position in a file.
• Removing lines using regular expressions.

There are 27 examples listed in this article, which will help you to become a master in sed command.

If you could memorize these commands, it will save a lot of your time when you have the requirement to perform various admin tasks in your projects.

Also, check this article on how you can use the sed command to replace a matching string in a file.

Note: Since this is a demonstration article, we use sed command without the -i option, which is similar to the “dry run” option, and will display the actual output without making any changes in the file.

But, if you would like to remove the lines from the source file in real environment then use the -i option with sed command.

To demonstrate sed command, we have created the ‘sed-demo.txt’ file and added the following contents with line numbers for better understanding.

Part-I) Removing lines based on a position in the file

The first part shows how to use sed command to delete lines based on a position in the file.

1) How to delete first line from a file?

To delete the first line from a file, use the following syntax:

Syntax:

• N – Denotes ‘Nth’ line in a file
• d – Indicates the deletion of a line.

The below sed command removes the first line from the ‘sed-demo.txt’ file:

2) How to delete last line from a file?

If you would like to delete the last line from a file, use the following syntax ( $ denotes the last line of a file).

The below sed command removes the last line from the ‘sed-demo.txt’ file:

The following sed command removes the first and last line from the file:

3) Deleting a particular line from a file

In this example, we will delete the third line from the ‘sed-demo.txt’ file using sed command as shown below. Similarly, any line can be removed by entering the line number instead of the number 3.

4) Deleting range of lines

The sed command removes any range of given lines from a file. We need to enter the ‘minimum’ and ‘maximum’ line numbers. The below example removes lines ranging from 5 to 7.

5) How to remove multiple lines

The sed command is capable of removing a set of given lines.

In this example, the following sed command will remove 1st , 5th , 9th , and the last line.

5.a) Deleting lines other than the specified range

Use the following sed command to remove all the lines from the file, except the specified range of lines:

Details;

• ! – Negation operator is used to keep the specified lines

Use the below command to delete all lines other than the first line:

Use the below command to delete all lines except the last line:

6) Deleting empty or blank lines

The following sed command will remove the empty or blank lines from ‘sed-demo.txt’ file.

Part-II) Removing lines based on pattern match

The second part shows, how to use sed command to remove lines in a file that match a given pattern.

7) Removing lines that contain a pattern

The following sed command will remove the lines which match the System pattern in ‘sed-demo.txt’ file .

8) Deleting lines that contain one of several strings

The following sed command removes the lines which match with the System or Linux pattern from the file ‘sed-demo.txt’:

9) Deleting lines that begin with specific character

The following sed command will remove all the lines that begin with a given character. To demonstrate this, I have created another file called ‘sed-demo-1.txt’ with following contents:

The following sed command removes all the lines that start with character R :

The following sed command will remove the lines that begin with the character either R or F :

The following sed command will remove all the lines that starts with character L and ends with System string :

10) Removing lines that end with specified character

The following sed command removes all the lines that end with character m :

The following sed command removes all the lines that ends with character either x or m :

11) Deleting all lines that start with uppercase

Use the following sed command to remove all the lines that start with an Uppercase letter:

12) Deleting a matching pattern lines with specified range

The below sed command removes the pattern Linux only if it is present in the lines from 1 to 6 :

12.a) Deleting only the last line if it contains a given pattern

The below sed command only removes the last line if it contains the pattern “openSUSE“

13) How to delete pattern matching lines and also the next Line?

Use the following sed command to delete the line which matches the pattern System and the subsequent line in the file as well.

13.a) Deleting lines starting from a pattern till the last line

The below sed command removes the line that matches the pattern “CentOS”, and also deletes all the subsequent lines till the end of the file:

14) Deleting lines that contains Digits/Numbers

The below sed command removes all the lines that contains ‘digits’:

The below sed command removes all the lines which only begins with digits:

The below sed command removes all the lines which ends with digits:

15) Deleting lines that contains Alphabetic Characters from a file

The below sed command removes all the lines that contains any alphabetic characters.

Closing Notes

In this guide, we have shown you several examples to delete lines from a file using the sed command.

If you have any questions or feedback, feel free to comment below.

Источник