Linux fast remove file

“Fastest way to delete large amount of files in linux”

what will happen when you use rm to delete large amount of files

I had this problem lately, I had over 27 million small files, each less than 100KB in one single directory by mistake. and when I want to delete them all, the rm command does not work properly, it takes way to much time to finish, so what’s wrong about it? the problem is that rm command is invoked for each and every file in the list. For example, if there are 50 files in the folder which are bigger than 7M, then 50 rm commands are invoked for deleting each of them. This will take a much longer time.

So are there any other method to solve this?

Find Command with -exec

The above shown command, will delete all the files inside /test directory. First the find command will look for all files inside the directory, and then for each result, it will execute and rm

essentially this command is not different from rm command, however,in practice, this would be a little faster than the raw rm command. it tooks around 14 Minute for half a million files, depending on your individual file size.

Find Command with -delete

this command is actually much faster than the above command. it tooks around 5 Minute for half a million files, depending on your individual file size.

this is actually the fatstest option. it tooks around 1 Minute for half a million files, depending on your individual file size.

But yeah if you are interested in deleting files using Perl, you need to have some good hands on with Perl regular expressions.

There is one more lesser used and less known method that can be used to delete large number of files inside a folder. This method is none other than our famous tool RSYNC used for transferring and synchronizing files between two local as well as remote locations in Linux

RSYNC with -delete

This can be achieved by simply synchronizing a target directory which has the large number of files, with an empty directory. In our case test directory has half a million files, lets create a directory called as blanktest, which will be kept empty for the purpose of simply synchronization. Now along with this we will be using -delete option in rsync, which will delete all those files in the target directory, which are are not present in the source

Empty Directory: /home/blanktest

Directory to be emptied: /test

this command tooks around 2 minutes to delete all files. Cool!

One last note

Linux and other operating systems have some limits in their file systems, it is not a good idea to put too many files in one single directory, if you want to store large amount of files, a good idea would be store these files subdirectories and the ideal amount of files in one single subdirecotries is less than 10000.

Источник

5 Ways to Empty or Delete a Large File Content in Linux

Occasionally, while dealing with files in Linux terminal, you may want to clear the content of a file without necessarily opening it using any Linux command line editors. How can this be achieved? In this article, we will go through several different ways of emptying file content with the help of some useful commands.

Caution: Before we proceed to looking at the various ways, note that because in Linux everything is a file, you must always make sure that the file(s) you are emptying are not important user or system files. Clearing the content of a critical system or configuration file could lead to a fatal application/system error or failure.

With that said, below are means of clearing file content from the command line.

Important: For the purpose of this article, we’ve used file access.log in the following examples.

1. Empty File Content by Redirecting to Null

A easiest way to empty or blank a file content using shell redirect null (non-existent object) to the file as below:

Empty Large File Using Null Redirect in Linux

2. Empty File Using ‘true’ Command Redirection

Here we will use a symbol : is a shell built-in command that is essence equivalent to the true command and it can be used as a no-op (no operation).

Another method is to redirect the output of : or true built-in command to the file like so:

Empty Large File Using Linux Commands

3. Empty File Using cat/cp/dd utilities with /dev/null

In Linux, the null device is basically utilized for discarding of unwanted output streams of a process, or else as a suitable empty file for input streams. This is normally done by redirection mechanism.

And the /dev/null device file is therefore a special file that writes-off (removes) any input sent to it or its output is same as that of an empty file.

Additionally, you can empty contents of a file by redirecting output of /dev/null to it (file) as input using cat command:

Empty File Using cat Command

Next, we will use cp command to blank a file content as shown.

Empty File Content Using cp Command

In the following command, if means the input file and of refers to the output file.

Empty File Content Using dd Command

4. Empty File Using echo Command

Here, you can use an echo command with an empty string and redirect it to the file as follows:

Empty File Using echo Command

Note: You should keep in mind that an empty string is not the same as null. A string is already an object much as it may be empty while null simply means non-existence of an object.

For this reason, when you redirect the out of the echo command above into the file, and view the file contents using the cat command, is prints an empty line (empty string).

To send a null output to the file, use the flag -n which tells echo to not output the trailing newline that leads to the empty line produced in the previous command.

Empty File Using Null Redirect

5. Empty File Using truncate Command

The truncate command helps to shrink or extend the size of a file to a defined size.

You can employ it with the -s option that specifies the file size. To empty a file content, use a size of 0 (zero) as in the next command:

Truncate File Content in Linux

That’s it for now, in this article we have covered multiple methods of clearing or emptying file content using simple command line utilities and shell redirection mechanism.

These are not probably the only available practical ways of doing this, so you can also tell us about any other methods not mentioned in this guide via the feedback section below.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

We are thankful for your never ending support.

Источник

How To: Linux / UNIX delete a file using rm command

H ow do I delete a file under a Linux / UNIX / *BSD / AIX / HP-UX operating system using command line options?

To remove or delete a file or directory in Linux, FreeBSD, Solaris, macOS, or Unix-like operating systems, use the rm command or unlink command. This page explains how to delete a given file on a Linux or Unix like system using the command line option.

Tutorial details
Difficulty level	Easy
Root privileges	No
Requirements	rm and unlink command on Linux or Unix
Est. reading time	4 minutes

Syntax: rm command to remove a file

rm (short for remove) is a Unix / Linux command which is used to delete files from a filesystem. Usually, on most filesystems, deleting a file requires write permission on the parent directory (and execute permission, in order to enter the directory in the first place). The syntax is as follows to delete the specified files and directories:

• -f : Forcefully remove file
• -r : Remove the contents of directories recursively

When rm command used just with the file names, rm deletes all given files without confirmation by the user.

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

Warning : Be careful with filenames as Unix and Linux, by default, won’t prompt for confirmation before deleting files. Always keep verified backups of all critical files and data.

Unix Remove or delete a file example

Say you have a file named abc.txt and you want to remove it:
$ rm abc.txt

Linux delete multiple files

Delete three files named foo.mp4, bar.doc, and demo.txt, run:

Linux recursively delete all files

Remove all files and sub-directories from a directory (say deltree like command from MS-DOS world), enter:
$ rm -rf mydir

Linux delete a file and prompt before every removal

To request confirmation before attempting to remove each file pass the -i option to the rm command:
$ rm -i filename
Sample outputs:

Gif 01: rm command demo

Force rm command to explain what is being done with file

Pass the -v option as follows:
$ rm -v moiz.list.txt bios-updates.doc
removed ‘moiz.list.txt’
removed ‘bios-updates.doc’

How to delete empty directories

To remove empty directory use rmdir command and not the rm command:
$ rmdir mydirectory
$ rmdir dirNameHere
$ rmdir docs

How to read a list of all files to delete from a text file

The rm command is often used in conjunction with xargs to supply a list of files to delete. Create a file called file.txt:
$ cat file.txt
List of to delete:

Now delete all file listed in file.txt, enter:
$ xargs rm

How do I delete a file named -foo.txt or a directory named -bar?

To delete a file called -foo.txt :
rm — -foo.txt
OR
rm — ./-foo.txt
To delete a directory called -bar :
rm -r -f — -bar
The two — dashes tells rm command the end of the options and rest of the part is nothing but a file or directory name begins with a dash.

Never run rm -rf / as an administrator or normal UNIX / Linux user

WARNING! These examples will delete all files on your computer if executed.

$ rm -rf /
$ rm -rf *

rm -rf (variously, rm -rf /, rm -rf *, and others) is frequently used in jokes and anecdotes about Unix disasters. The rm -rf / variant of the command, if run by an administrator, would cause the contents of every writable mounted filesystem on the computer to be deleted. Do not try these commands.

Conclusion

You learned how to delete files on Linux and Unix-like operating systems. Here are all important options for GNU rm command (read man page here)

Remove files command summary

Option	Description
-f	Ignore nonexistent files and arguments, never prompt
-i	Prompt before every file removal
-I	Prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes —interactive[=WHEN] prompt according to WHEN: never, once (-I), or always (-i); without WHEN, prompt always
—one-file-system	when removing a hierarchy recursively, skip any directory that is on a file system different from that of the corresponding command line argument
—no-preserve-root	do not treat ‘/’ specially
—preserve-root[=all]	do not remove ‘/’ (default); with ‘all’, reject any command line argument on a separate device from its parent
-r	remove directories and their contents recursively
-R	same as above
-d	rmove empty directories
-v	Explain what is being done

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Источник

Which is the fastest method to delete files in Linux

Creating, deleting, and modifying files is one of the common task that a user does in any operating system. This kind of task comes under a day to day list of tasks that a user does. Although it is quite fast and seamless operation when it comes to deleting a single or a handful of files in Linux or any other operating system. But if the number of files is quite large, then the deletion operation takes quite long to complete.

What happens when you delete a file in Linux, depends on the kind of file system, on which the file you want to delete resides. There are many operational differences in deleting files under different types of file system. When we talk about files in Linux, its all about inodes rather than files. How an inode gets modified, during file deletion is an important aspect to understand.

Inodes are the building blocks of Linux operating system. If you are interested in understanding inodes, i would recommend reading the below post, before going ahead, as we will not be discussing inode related details in this post.

I am writing this post, to find out the fastest method to delete large number of files in Linux. We will begin this tutorial with some simple file deletion methods, and then will compare the speed with which different method completed the task of file deletion. Another major reason for writing this post is the time i spend on one of our crawler servers, to delete millions of files with very small size (few KBs).

As i told, if you are dealing with small number of files, then the operation will be fast compared to a large number of files which are of very small size. Let’s begin this with some simple commands in Linux used to delete files.

Commands to delete files in Linux and their example usage

To delete files in Linux, the most commonly used command is rm command. Let’s see some example’s of rm command.

-f used in the above command, will delete the file forcefully without asking for a confirmation.

The above command will delete the directory named «testdirectory» as well as all the contents inside that directory(-r option used is to delete files recursively)

The above command rmdir, will only delete the directory if its empty.

Let’s now have a look at some different methods of deleting files in Linux. One of my favorite methods out there is to use find command. Find commands is a very handy tool that can be used to search files according to its type, size, created date, modified date, and much more different criteria. To find out more about this wonderful searching tool in Linux, read the below post.

The above shown command, will delete all the files inside /test directory. First the find command will look for all files inside the directory, and then for each result, it will execute and rm.

Let’s see some different methods that can be used with find command, to delete files.

In the above shown example, find command will search all those files inside the /test directory which are modified 7 days ago, and then delete each of them.

Above shown example, will search for all those files in the directory /test which are larger than 7M, and then delete each of them.

In all of the above shown examples of find command, rm command is invoked for each and every file in the list. For example, in the last find command shown above, if the result is 50 files which are bigger than 7M, then 50 rm commands are invoked for deleting each of them. This will take a much longer time.

Instead of using the above command of rm with the help of -exec argument in find, there is a better alternative. We will see the alternative and then calculate the difference between speed in each of them.

As i told before, the main idea behind finding the deletion speed, is when you delete large number of files. So lets first create half a million files with the help of a simple bash for loop. And after creating half a million files, we will try to delete it with rm command, find command with exec argument and then will see a better find command alternative .

The above command will create 5 lakh files (half a million) in the current working directory, with the name 1.txt to 500000.txt, and each file contains the text «testing», so the file size will be at least in the range of some kilo bytes. Let’s now test the speed of deleting these number of files with different commands. First let’s use the simple rm command, and then will use find command with -exec and then delete option to calculate the time taken to delete these number of files.

If you see the above rm command i ran on the test directory, containing half a million files, it gave me a message saying /bin/rm: Argument list too long . Which means the command didn’t complete the deletion, because the number of files given to rm command was too big to complete. So rm command didn’t even stand the test, because it gave up. Don’t pay attention to the time displayed by the time command, because rm command didn’t complete its operation, and time command displays the output without bothering about the end result of the command.

Now let’s use our previously seen find command with -exec argument.

From the output we got by using time command, it is clear that it took 14 minutes and 51 seconds to delete 5 lakh files from a single directory. This is quite a long time, because for each file a separate rm command is executed, until the complete list of files gets deleted.

Now lets test the time consumed, by using -delete, option in find command.

Wow you saw that result!! -delete option only took 5 minutes 11 seconds. That’s a wonderful improvement in the speed, when you delete millions of files in Linux.

Let’s now have a look at how deleting files using Perl language works, and its speed compared to other options we saw earlier.

That’s insanely fast compared to other find command, and rm command options we saw earlier. Till now this seems to be the best method that can be used to delete all the files in a directory. That’s a remarkable achievement in speed for deleting files in Linux. If you see the output Perl only took around 1 minute to delete half a million files in that directory.

But yeah if you are interested in finding more complex options while using Perl, you need to have some good hands on with Perl regular expressions.

There is one more lesser used and less known method that can be used to delete large number of files inside a folder. This method is none other than our famous tool RSYNC used for transferring and synchronizing files between two local as well as remote locations in Linux.

Let’s have a look at that method of deleting all files inside a folder with the help of RSYNC command. The method and logic used behind deleting files with the help of rsync is based on the fact that rsync is commonly used for synchronizing files between two different locations.

This can be achieved by simply synchronizing a target directory which has the large number of files, with an empty directory. In our case test directory has half a million files, lets create a directory called as blanktest , which will be kept empty for the purpose of simply synchronization. Now along with this we will be using -delete option in rsync, which will delete all those files in the target directory, which are are not present in the source(in our case the source is an empty directory, so all the files in the destination directory will be deleted.)

Empty Directory: /home/blanktest

Directory to be emptied: /test

The results are pretty impressive, so its much better to use rsync if you want to empty a directory containing millions of files, compared to find command.

The below shown table summarizes the speed for file deletion in Linux, using different methods in Linux.

Источник