Execute command on all files in directory linux

cd into all directories, execute command on files in that directory, and return to previous current directory

I’m attempting to write a script that will be run in a given directory with many single level sub directories. The script will cd into each of the sub directories, execute a command on the files in the directory, and cd out to continue onto the next directory. What is the best way to do this?

5 Answers 5

The best way is to not use cd at all:

execdir is like exec , but the working directory is different:

It is not POSIX.

The above command doesn’t need to do any subshells — it just tracks its progress in the current shell by alternating $OLDPWD and $PWD . When you cd — the shell exchanges the value of these two variables, basically, as it changes directories. It also prints the name for each directory as it works there to stderr.

I just had a second look at it and decided I could do a better job with error handling. It will skip a dir into which it cannot cd — and cd will print a message about why to stderr — and it will break w/ a non-zero exit code if your command does not execute successfully or if running command somehow affects its ability to return to your original directory — $OLDPWD . In that case it also does a cd — last — and writes the resulting current working directory name to stderr.

Источник

Execute command on all files in a directory

Could somebody please provide the code to do the following: Assume there is a directory of files, all of which need to be run through a program. The program outputs the results to standard out. I need a script that will go into a directory, execute the command on each file, and concat the output into one big output file.

For instance, to run the command on 1 file:

10 Answers 10

The following bash code will pass $file to command where $file will represent every file in /dir

• -maxdepth 1 argument prevents find from recursively descending into any subdirectories. (If you want such nested directories to get processed, you can omit this.)
• -type -f specifies that only plain files will be processed.
• -exec cmd option <> tells it to run cmd with the specified option for each file found, with the filename substituted for <>
• \; denotes the end of the command.
• Finally, the output from all the individual cmd executions is redirected to results.out

However, if you care about the order in which the files are processed, you might be better off writing a loop. I think find processes the files in inode order (though I could be wrong about that), which may not be what you want.

I’m doing this on my raspberry pi from the command line by running:

The accepted/high-voted answers are great, but they are lacking a few nitty-gritty details. This post covers the cases on how to better handle when the shell path-name expansion (glob) fails, when filenames contain embedded newlines/dash symbols and moving the command output re-direction out of the for-loop when writing the results to a file.

When running the shell glob expansion using * there is a possibility for the expansion to fail if there are no files present in the directory and an un-expanded glob string will be passed to the command to be run, which could have undesirable results. The bash shell provides an extended shell option for this using nullglob . So the loop basically becomes as follows inside the directory containing your files

This lets you safely exit the for loop when the expression ./* doesn’t return any files (if the directory is empty)

or in a POSIX compliant way ( nullglob is bash specific)

This lets you go inside the loop when the expression fails for once and the condition [ -f «$file» ] check if the un-expanded string ./* is a valid filename in that directory, which wouldn’t be. So on this condition failure, using continue we resume back to the for loop which won’t run subsequently.

Also note the usage of — just before passing the file name argument. This is needed because as noted previously, the shell filenames can contain dashes anywhere in the filename. Some of the shell commands interpret that and treat them as a command option when the name are not quoted properly and executes the command thinking if the flag is provided.

The — signals the end of command line options in that case which means, the command shouldn’t parse any strings beyond this point as command flags but only as filenames.

Double-quoting the filenames properly solves the cases when the names contain glob characters or white-spaces. But *nix filenames can also contain newlines in them. So we de-limit filenames with the only character that cannot be part of a valid filename — the null byte ( \0 ). Since bash internally uses C style strings in which the null bytes are used to indicate the end of string, it is the right candidate for this.

So using the printf option of shell to delimit files with this NULL byte using the -d option of read command, we can do below

The nullglob and the printf are wrapped around (..) which means they are basically run in a sub-shell (child shell), because to avoid the nullglob option to reflect on the parent shell, once the command exits. The -d » option of read command is not POSIX compliant, so needs a bash shell for this to be done. Using find command this can be done as

For find implementations that don’t support -print0 (other than the GNU and the FreeBSD implementations), this can be emulated using printf

Another important fix is to move the re-direction out of the for-loop to reduce a high number of file I/O. When used inside the loop, the shell has to execute system-calls twice for each iteration of the for-loop, once for opening and once for closing the file descriptor associated with the file. This will become a bottle-neck on your performance for running large iterations. Recommended suggestion would be to move it outside the loop.

Extending the above code with this fixes, you could do

which will basically put the contents of your command for each iteration of your file input to stdout and when the loop ends, open the target file once for writing the contents of the stdout and saving it. The equivalent find version of the same would be

Источник

How to list all files in a directory with absolute paths

I need a file (preferably a .list file) which contains the absolute path of every file in a directory.

Example dir1: file1.txt file2.txt file3.txt

How can I accomplish this in linux/mac?

10 Answers 10

You can use find. Assuming that you want only regular files, you can do:

You can adjust the type parameter as appropriate if you want other types of files.

It’s the shell that computes the list of (non-hidden) files in the directory and passes the list to ls . ls just prints that list here, so you could as well do:

Note that it doesn’t include hidden files, includes files of any type (including directories) and if there’s no non-hidden file in the directory, in POSIX/csh/rc shells, you’d get /current/wd/* as output. Also, since the newline character is as valid as any in a file path, if you separate the file paths with newline characters, you won’t be able to use that resulting file to get back to the list of file reliably.

With the zsh shell, you could do instead:

• -rC1 prints r aw on 1 C olumn.
• -N , output records are NUL-delimited instead of newline-delimited (lines) as NUL is the only character that can’t be found in a file name.
• N : expands to nothing if there’s no matching file ( nullglob )
• D : include hidden files ( dotglob ).
• -. : include only regular files ( . ) after symlink resolution ( — ).

Then, you’d be able to do something like:

To remove those files for instance.

You could also use the 😛 modifier in the glob qualifiers to get the equivalent of realpath() on the files expanded from the globs (gets a full path exempt of any symlink component):

To see just regular files —

Another way with tree , not mentioned here, it goes recursively and unlike find or ls you don’t have any errors (like: Permission denied , Not a directory ) you also get the absolute path in case you want to feed the files to xargs or other command

the options meaning:

To install tree :

sudo apt install tree on Ubuntu/Debian

sudo yum install tree on CentOS/Fedora

sudo zypper install tree on OpenSUSE

You can just use realpath or readlink this naughty way:

When ls prints to a TTY it formats the file names in columns, but when it’s writing to a file, pipe, or other non-TTY it behaves like ls -1 and prints one file name per line. You can check this by running ls | cat in place of ls . [1]

• xargs build and execute command lines from standard input.
• realpath : return the canonicalized absolute pathname
• readlink : read value of a symbolic link
• Use realpath — to make it treat everything that follows as parameters instead of options, if files could have » -something «.
• If some files have spaces you could:

In a past Linux environment, I had a resolve command that would standardize paths, including making a relative path into an absolute path. I can’t find it now, so maybe it was written by someone in that organization.

You can make your own script using functions in the Python or Perl standard libraries (and probably other languages too).

Then, you would solve your problem with:

With this command, you can also do things like this:

Recursive files can be listed by many ways in linus. Here i am sharing one liner script to clear all logs of files(only files) from /var/log/ directory and second check recently which logs file has made an entry.

note: for directory location you can also pass $PWD instead of /var/log.

To list the full path of all commands (apps/programs) accessible to the user. (revised to address most, but not all limitations outlined in the comments)

NOTE
The PATH variable would normally have a colon ( : ) either at the beginning or at the end, but not both. A colon at the beginning or end signifies to search the current directory as well. Standard practice is for it to be at the end so as to never override standard utility programs. The sed substitutions here handle either case.

Explanation.

• eval
  Frankly, I don’t know why eval is needed here, but it is. Without it, I get.

ls: cannot access ‘./<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/home/alpha/bin/<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/usr/local/sbin/<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/usr/local/bin/<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/usr/sbin/<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/usr/bin/<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/sbin/<.,>[[:word:]]‘: No such file or directory
ls: cannot access ‘/bin/<.,>[[:word:]]‘: No such file ordirectory

Источник

tar: add all files and directories in current directory INCLUDING .svn and so on

I try to tar.gz a directory and use

The resulting tar includes .svn directories in subdirs but NOT in the current directory (as * gets expanded to only ‘visible’ files before it is passed to tar

tar -czf workspace.tar.gz . instead but then I am getting an error because ‘.’ has changed while reading:

Is there a trick so that * matches all files (including dot-prefixed) in a directory?

(using bash on Linux SLES-11 (2.6.27.19)

16 Answers 16

Don’t create the tar file in the directory you are packing up:

does the trick, except it will extract the files all over the current directory when you unpack. Better to do:

or, if you don’t know the name of the directory you were in:

(This assumes that you didn’t follow symlinks to get to where you are and that the shell doesn’t try to second guess you by jumping backwards through a symlink — bash is not trustworthy in this respect. If you have to worry about that, use cd -P .. to do a physical change directory. Stupid that it is not the default behaviour in my view — confusing, at least, for those for whom cd .. never had any alternative meaning.)

One comment in the discussion says:

I [. ] need to exclude the top directory and I [. ] need to place the tar in the base directory.

The first part of the comment does not make much sense — if the tar file contains the current directory, it won’t be created when you extract file from that archive because, by definition, the current directory already exists (except in very weird circumstances).

The second part of the comment can be dealt with in one of two ways:

Источник

Linux Delete All Files In Directory Using Command Line

Linux Delete All Files In Directory

The procedure to remove all files from a directory:

1. Open the terminal application
2. To delete everything in a directory run: rm /path/to/dir/*
3. To remove all sub-directories and files: rm -r /path/to/dir/*

Let us see some examples of rm command to delete all files in a directory when using Linux operating systems.

How to remove all the files in a directory?

Suppose you have a directory called /home/vivek/data/. To list files type the ls command:
$ ls

Understanding rm command option that deleted all files in a directory

• -r : Remove directories and their contents recursively.
• -f : Force option. In other words, ignore nonexistent files and arguments, never prompt. Dangerous option. Be careful.
• -v : Verbose option. Show what rm is doing on screen.

Deleting hidden vs non-hidden files

In Linux, any file or directory that starts with a dot character called a dot file. It is to be treated as hidden file. To see hidden files pass the -a to the ls command:
ls
ls -a
ls -la
To remove all files except hidden files in a directory use:
rm /path/to/dir/*
rm -rf /path/to/dir/*
rm *
In this example, delete all files including hidden files, run:
rm -rf /path/to/dir1/<*,.*>
rm -rfv /path/to/dir1/

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

Bash remove all files from a directory including hidden files using the dotglob option

If the dotglob option set, bash includes filenames beginning with a ‘.’ in the results of pathname expansion. In other words, turn on this option to delete hidden files:

See GNU/bash man page for the shopt command online here:
man bash
help shopt

Linux Remove All Files In Directory

As I said earlier one can use the unlink command too. The syntax is:
unlink filename
For example, delete file named foo.txt in the current working directory, enter:
unlink foo.txt
It can only delete a single file at a time. You can not pass multiple files or use wildcards such as *. Therefore, I strongly recommend you use the rm command as discussed above.

Conclusion

In this quick tutorial, you learned how to remove or delete all the files in a directory using the rm command. Linux offers a few more options to find and delete files. Please see the following tutorials:

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Источник