Linux delete string from file

How to delete a string from a file in bash

I want to delete a string from a file. This string is provided by the user. I tried the following command which didn’t work.

I tried without double quotes(«») as well. But it didn’t work.

Could you guys please help me?

2 Answers 2

but make sure input doesn’t contain any regex special character.

awk with index function might be safer due to non-regex:

The problem is that variables won’t be expanded when put inside single quotes.

So you need to use double quotes:

The original file will be kept as xyz.txt.bak and the modified file will be xyz.txt .

If you do not want a backup:

Not the answer you’re looking for? Browse other questions tagged string bash shell or ask your own question.

Related

Hot Network Questions

Subscribe to RSS

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

site design / logo © 2021 Stack Exchange Inc; user contributions licensed under cc by-sa. rev 2021.10.8.40416

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

Источник

Remove the last line from a file in Bash

I have a file, foo.txt , containing the following lines:

I want a simple command that results in the contents of foo.txt being:

15 Answers 15

The -i option does not exist in GNU sed versions older than 3.95, so you have to use it as a filter with a temporary file:

Of course, in that case you could also use head -n -1 instead of sed .

MacOS:

On Mac OS X (as of 10.7.4), the equivalent of the sed -i command above is

This is by far the fastest and simplest solution, especially on big files:

if You want to delete the top line use this:

which means output lines starting at line 2.

Do not use sed for deleting lines from the top or bottom of a file — it’s very very slow if the file is large.

For large files

I had trouble with all the answers here because I was working with a HUGE file (

300Gb) and none of the solutions scaled. Here’s my solution:

Or alternatively, as a one liner:

In words: Find out the length of the file you want to end up with (length of file minus length of length of its last line, using bc ), and set that position to be the end of the file (by dd ing one byte of /dev/null onto it).

This is fast because tail starts reading from the end, and dd will overwrite the file in place rather than copy (and parse) every line of the file, which is what the other solutions do.

NOTE: This removes the line from the file in place! Make a backup or test on a dummy file before trying it out on your own file!

Источник

Delete specific line number(s) from a text file using sed?

I want to delete one or more specific line numbers from a file. How would I do this using sed?

7 Answers 7

If you want to delete lines from 5 through 10 and line 12th:

This will print the results to the screen. If you want to save the results to the same file:

This will back the file up to file.bak , and delete the given lines.

Note: Line numbers start at 1. The first line of the file is 1, not 0.

You can delete a particular single line with its line number by

This will delete the line on 33 line number and save the updated file.

and awk as well

This is very often a symptom of an antipattern. The tool which produced the line numbers may well be replaced with one which deletes the lines right away. For example;

(where deletelines is the utility you are imagining you need) is the same as

Having said that, if you are in a situation where you genuinely need to perform this task, you can generate a simple sed script from the file of line numbers. Humorously (but perhaps slightly confusingly) you can do this with sed .

This accepts a file of line numbers, one per line, and produces, on standard output, the same line numbers with d appended after each. This is a valid sed script, which we can save to a file, or (on some platforms) pipe to another sed instance:

On some platforms, sed -f does not understand the option argument — to mean standard input, so you have to redirect the script to a temporary file, and clean it up when you are done, or maybe replace the lone dash with /dev/stdin or /proc/$pid/fd/1 if your OS (or shell) has that.

As always, you can add -i before the -f option to have sed edit the target file in place, instead of producing the result on standard output. On *BSDish platforms (including OSX) you need to supply an explicit argument to -i as well; a common idiom is to supply an empty argument; -i » .

Источник

How can I remove the first line of a text file using bash/sed script?

I need to repeatedly remove the first line from a huge text file using a bash script.

Right now I am using sed -i -e «1d» $FILE — but it takes around a minute to do the deletion.

Is there a more efficient way to accomplish this?

100MB. MacBook Air 2013 with SSD.

16 Answers 16

-n x : Just print the last x lines. tail -n 5 would give you the last 5 lines of the input. The + sign kind of inverts the argument and make tail print anything but the first x-1 lines. tail -n +1 would print the whole file, tail -n +2 everything but the first line, etc.

GNU tail is much faster than sed . tail is also available on BSD and the -n +2 flag is consistent across both tools. Check the FreeBSD or OS X man pages for more.

The BSD version can be much slower than sed , though. I wonder how they managed that; tail should just read a file line by line while sed does pretty complex operations involving interpreting a script, applying regular expressions and the like.

Note: You may be tempted to use

but this will give you an empty file. The reason is that the redirection ( > ) happens before tail is invoked by the shell:

1. Shell truncates file $FILE
2. Shell creates a new process for tail
3. Shell redirects stdout of the tail process to $FILE
4. tail reads from the now empty $FILE

If you want to remove the first line inside the file, you should use:

The && will make sure that the file doesn’t get overwritten when there is a problem.

You can use -i to update the file without using ‘>’ operator. The following command will delete the first line from the file and save it to the file.

For those who are on SunOS which is non-GNU, the following code will help:

You can easily do this with:

on the command line; or to remove the first line of a file permanently, use the in-place mode of sed with the -i flag:

No, that’s about as efficient as you’re going to get. You could write a C program which could do the job a little faster (less startup time and processing arguments) but it will probably tend towards the same speed as sed as files get large (and I assume they’re large if it’s taking a minute).

But your question suffers from the same problem as so many others in that it pre-supposes the solution. If you were to tell us in detail what you’re trying to do rather then how, we may be able to suggest a better option.

For example, if this is a file A that some other program B processes, one solution would be to not strip off the first line, but modify program B to process it differently.

Let’s say all your programs append to this file A and program B currently reads and processes the first line before deleting it.

You could re-engineer program B so that it didn’t try to delete the first line but maintains a persistent (probably file-based) offset into the file A so that, next time it runs, it could seek to that offset, process the line there, and update the offset.

Then, at a quiet time (midnight?), it could do special processing of file A to delete all lines currently processed and set the offset back to 0.

It will certainly be faster for a program to open and seek a file rather than open and rewrite. This discussion assumes you have control over program B, of course. I don’t know if that’s the case but there may be other possible solutions if you provide further information.

Источник

How to delete from a text file, all lines that contain a specific string?

How would I use sed to delete all lines in a text file that contain a specific string?

19 Answers 19

To remove the line and print the output to standard out:

To directly modify the file – does not work with BSD sed:

Same, but for BSD sed (Mac OS X and FreeBSD) – does not work with GNU sed:

To directly modify the file (and create a backup) – works with BSD and GNU sed:

There are many other ways to delete lines with specific string besides sed :

Ruby (1.9+)

Shell (bash 3.2 and later)

GNU grep

And of course sed (printing the inverse is faster than actual deletion):

You can use sed to replace lines in place in a file. However, it seems to be much slower than using grep for the inverse into a second file and then moving the second file over the original.

The first command takes 3 times longer on my machine anyway.

The easy way to do it, with GNU sed :

You may consider using ex (which is a standard Unix command-based editor):

• + executes given Ex command ( man ex ), same as -c which executes wq (write and quit)
• g/match/d — Ex command to delete lines with given match , see: Power of g

The above example is a POSIX-compliant method for in-place editing a file as per this post at Unix.SE and POSIX specifications for ex .

The difference with sed is that:

sed is a Stream EDitor, not a file editor. BashFAQ

Unless you enjoy unportable code, I/O overhead and some other bad side effects. So basically some parameters (such as in-place/ -i ) are non-standard FreeBSD extensions and may not be available on other operating systems.

I was struggling with this on Mac. Plus, I needed to do it using variable replacement.

sed -i » «/$pattern/d» $file

where $file is the file where deletion is needed and $pattern is the pattern to be matched for deletion.

I picked the » from this comment.

The thing to note here is use of double quotes in «/$pattern/d» . Variable won’t work when we use single quotes.

You can also use this:

Here -v will print only other than your pattern (that means invert match).

I have made a small benchmark with a file which contains approximately 345 000 lines. The way with grep seems to be around 15 times faster than the sed method in this case.

I have tried both with and without the setting LC_ALL=C, it does not seem change the timings significantly. The search string (CDGA_00004.pdbqt.gz.tar) is somewhere in the middle of the file.

Here are the commands and the timings:

To get a inplace like result with grep you can do this:

The first command edits the file(s) inplace (-i).

The second command does the same thing but keeps a copy or backup of the original file(s) by adding .bk to the file names (.bk can be changed to anything).

You can also delete a range of lines in a file. For example to delete stored procedures in a SQL file.

sed ‘/CREATE PROCEDURE.*/,/END ;/d’ sqllines.sql

This will remove all lines between CREATE PROCEDURE and END ;.

I have cleaned up many sql files withe this sed command.

echo -e «/thing_to_delete\ndd\033:x\n» | vim file_to_edit.txt

Just in case someone wants to do it for exact matches of strings, you can use the -w flag in grep — w for whole. That is, for example if you want to delete the lines that have number 11, but keep the lines with number 111:

It also works with the -f flag if you want to exclude several exact patterns at once. If «blacklist» is a file with several patterns on each line that you want to delete from «file»:

to show the treated text in console

to save treated text into a file

to append treated text info an existing file

to treat already treated text, in this case remove more lines of what has been removed

the | more will show text in chunks of one page at a time.

Curiously enough, the accepted answer does not actually answer the question directly. The question asks about using sed to replace a string, but the answer seems to presuppose knowledge of how to convert an arbitrary string into a regex.

Many programming language libraries have a function to perform such a transformation, e.g.

But how to do it on the command line?

Since this is a sed-oriented question, one approach would be to use sed itself:

So given an arbitrary string $STRING we could write something like:

or as a one-liner:

with variations as described elsewhere on this page.

Источник