Linux argument list to long

mv, rm, cp, ls: Argument list too long

Выполняя в консоли Linux команду mv или rm, рано или поздно вы можете наткнуться на сообщение об ошибке «argument list too long», используя cp, mv, rm и подобные команды, которые обрабатывают большое количество файлов. Ошибка может возникнуть, если в одном каталоге оказалось много файлов — стремясь экономить память, команда не может обработать полный список и вываливается с ошибкой. Есть несколько возможностей, как обойти данное ограничение.
Для тестов создадим два каталога — «source» и «target» и 20,000 пустых текстовых файлов:

Пытаясь выполнить команды mv, cp, rm или ls над этим количеством наткнемся на ограничение:

Выполнение команд выдаст ошибку:

Чтобы обойти данную проблему, можно использовать xargs или find, либо комбинацию обоих.

Использование ls и xargs для перемещения файлов

Используя вывод команды «ls», можно переместить все файлы из source в target:

Флаг -i в xargs заставит команду подставить вместо <> полученное от предыдущей команды строковое значение. Использовать для вывода команду типа «ls *.txt» нельзя, поскольку список аргументов снова станет слишком большим, прежде чем дойдет до выполнения xargs. Если нужно обработать определенные файлы по маске, лучше использовать команду find.

Использование find для перемещения большого числа файлов из source в target

Вместо флага -exec можно использовать xargs:

Убрав из параметров команды find часть » -name *.txt» позволит перенести все. Чтобы обработать только файлы, можно использовать параметр « -type f «.

Все три приведенных примера выполнятся примерно за одинаковое время, возможно вариант с -exec будет чуть быстрее чем xargs.

Использование find для обработки большого числа файлов лучше чем ls, поскольку дает гораздо больше контроля — можно указать шаблон для имен файлов, которые будут обрабатываться.

Как удалить большое число файлов

Приведенные ниже команды позволят удалить любое число файлов, находящихся в директории target:

Использование приведенных команд выглядит намного сложнее простой «rm -f source/*», но может оказаться единственным вариантом удалить большое число файлов нужного формата.

mv, rm, cp, ls: Argument list too long: 1 комментарий

Спасибо, полезная статья, помогла. Как раз столкнулся с такой проблемой. xargs редко приходится применять, поэтому всё время забывается как по конвейеру передавать аргументы.

Источник

Argument list too long when copying/deleting/moving files on Linux

If you do a lot of command line stuff in Linux, sooner or later you’ll come across the «argument list too long» error when copying, moving, deleting, listing etc files in a directory that contains a lot of files. This post looks at a couple of ways of solving this problem.

For the tests conducted in this post I created two directories, «source» and «target» and created 20,000 empty text files like so:

Doing any of these:

will result in errors like this:

To workaround the problem you can use xargs or find, or a combination of the two.

Using the «ls» command, you could do this to move all the files from source into target:

The -i flag in xargs tells it to use string replacements and <> is the placeholder. You cannot use a pattern for ls such as «ls *.txt» because you’ll get the arugment list too long error before the data can be piped through to xargs. In order to do pattern matching you have to use the find command.

Using find to do the same move from source to target, you could do this:

or this, using xargs instead of the -exec flag:

In the «find» examples above you could omit the -name *.txt part to move everything, and/or add further pattern matches such as -type f to only match files.

All three examples above take roughly the same amount of time to run although I found in testing the find … -exec method to be slightly faster than find … xargs

The find method is better than the ls method because it allows you to specify filename patterns and can give you a lot more control over what you are copying, moving, deleting etc.

As a final set of examples, you could do the following to then delete the files from the target directory:

Having to delete/move/etc files in this way is trickier and more time consuming than being able to simply do something like rm -f source/* but sometimes this is the only way to do it if there are too many files matching your pattern.

Источник

«Argument list too long»: How do I deal with it, without changing my command?

When I run a command like ls */*/*/*/*.jpg , I get the error

I know why this happens: it is because there is a kernel limit on the amount of space for arguments to a command. The standard advice is to change the command I use, to avoid requiring so much space for arguments (e.g., use find and xargs ).

What if I don’t want to change the command? What if I want to keep using the same command? How can I make things «just work», without getting this error? What solutions are available?

4 Answers 4

On Linux, the maximum amount of space for command arguments is 1/4th of the amount of available stack space. So, a solution is to increase the amount of space available for the stack.

Short version: run something like

Longer version: The default amount of space available for the stack is something like 8192 KB. You can see the amount of space available, as follows:

Choose a larger number, and set the amount of space available for the stack. For instance, if you want to try allowing up to 65536 KB for the stack, run this:

You may need to play around with how large this needs to be, using trial-and-error. In many cases, this is a quick-and-dirty solution that will eliminate the need to modify the command and work out the syntax of find , xargs , etc. (though I realize there are other benefits to doing so).

I believe that this is Linux-specific. I suspect it probably won’t help on any other Unix operating system (not tested).

Instead of ls */*/*/*/*.jpg , try:

xargs (1) knows, what the maximum number of arguments is on the system, and will break up its standard input to call the specified command-line multiple times with no more arguments than that limit, whatever it is (you can also set it lower than the OS’ maximum using the -n option).

For example, suppose, the limit is 3 arguments and you have five files. In that case xargs will execute ls twice:

Often this is perfectly suitable, but not always — for example, you can not rely on ls (1) sorting all of the entries for you properly, because each separate ls -invocation will sort only the subset of entries given to it by xargs .

Though you can bump the limit as suggested by others, there will still be a limit — and some day your JPG-collection will outgrow it again. You should prepare your script(s) to deal with an infinite number.

This Linux Journal article gives 4 solutions. Only the fourth solution does not involve changing the command:

Method #4 involves manually increasing the number of pages that are allocated within the kernel for command-line arguments. If you look at the include/linux/binfmts.h file, you will find the following near the top:

In order to increase the amount of memory dedicated to the command-line arguments, you simply need to provide the MAX_ARG_PAGES value with a higher number. Once this edit is saved, simply recompile, install and reboot into the new kernel as you would do normally.

On my own test system I managed to solve all my problems by raising this value to 64. After extensive testing, I have not experienced a single problem since the switch. This is entirely expected since even with MAX_ARG_PAGES set to 64, the longest possible command line I could produce would only occupy 256KB of system memory—not very much by today’s system hardware standards.

The advantages of Method #4 are clear. You are now able to simply run the command as you would normally, and it completes successfully. The disadvantages are equally clear. If you raise the amount of memory available to the command line beyond the amount of available system memory, you can create a D.O.S. attack on your own system and cause it to crash. On multiuser systems in particular, even a small increase can have a significant impact because every user is then allocated the additional memory. Therefore always test extensively in your own environment, as this is the safest way to determine if Method #4 is a viable option for you.

I agree that the limitation is seriously annoying.

Источник

Argument list too long when copying files

I just asked a question related to how I can count the files of particular extension. Now I want to cp these files to a new dir .

but they are giving the same error,

bash: /bin/cp: Argument list too long

How do I copy them?

4 Answers 4

cp *.prj ../prjshp/ is the right command, but you’ve hit a rare case where it runs into a size limitation. The second command you tried doesn’t make any sense.

One method is to run cp on the files in chunks. The find command knows how to do this:

• find traverses the current directory and the directories below it recursively.
• -maxdepth 1 means to stop at a depth of 1, i.e. don’t recurse into subdirectories.
• -name ‘*.prj’ means to only act on the files whose name matches the specified pattern. Note the quotes around the pattern: it will be interpreted by the find command, not by the shell.
• -exec … <> + means to execute the specified command for all the files. It invokes the command multiple times if necessary, taking care not to exceed the command line limit.
• mv -t ../prjshp moves the specified files into ../prjshp . The -t option is used here because of a limitation of the find command: the found files (symbolized by <> ) are passed as the last argument of the command, you can’t add the destination after it.

Another method is to use rsync .

• rsync -r … . ../prjshp copies the current directory into ../prjshp recursively.
• —include=’*.prj’ —exclude=’*’ means to copy files matching *.prj and exclude everything else (including subdirectories, so .prj files in subdirectories won’t be found).

This command copies the files one by one and will work even if there are too many of them for * to expand into a single cp command:

1/4 million jpegs I had extracted from a video for a project. This is the approach I used.

There’s 3 key points to keep in mind when facing Argument list too long error:

The length of command-line arguments is limited by ARG_MAX variable, which by POSIX definition is «. [m]aximum length of argument to the exec functions including environment data» (emphasis added)». That is, when shell executes a non-built-it command, it has to call one of exec() to spawn that command’s process, and that’s where ARG_MAX comes into play. Additionally, the name or path to the command itself ( for example, /bin/echo ) plays a role.

Shell built-in commands are executed by shell, which means the shell doesn’t use exec() family of functions and therefore aren’t affected by ARG_MAX variable.

Certain commands, such as xargs and find are aware of ARG_MAX variable and repeatedly perform actions under that limit

From the points above and as shown in Kusalananda’s excellent answer on related question, the Argument list too long can also occur when environment is big. So taking in consideration that each user’s environment may vary, and the argument size in bytes is relevant, it’s hard to come up with a single number of files/arguments.

How to handle such error ?

The key thing is to focus not on the number of files, but focus on whether or not the command you’re going to use involves exec() family of function and tangentially — the stack space.

Use shell built-ins

As discussed before, the shell built-ins are immune to ARG_MAX limit, that is things such as for loop, while loop, built-in echo , and built-in printf — all those will perform well enough.

On related question about deleting files, there was a solution as such:

Note that this uses shell’s built-in printf . If we’re calling the external printf , that will involve exec() , hence will fail with large number of arguments:

bash arrays

According to an answer by jlliagre, bash doesn’t impose limits on arrays, so building array of filenames and using slices per iteration of loop can be done as well, as shown in danjpreron’s answer:

This, however, has limitation of being bash-specific and non-POSIX.

Increase stack space

Sometimes you can see people suggest increasing the stack space with ulimit -s ; on Linux ARG_MAX value is 1/4th of stack space for each program, which means increasing stack space proportionally increases space for arguments.

According to answer by Franck Dernoncourt, which cites Linux Journal, one can also recompile Linux kernel with larger value for maximum memory pages for arguments, however, that’s more work than necessary and opens potential for exploits as stated in the cited Linux Journal article.

Avoid shell

Another way, is to use python or python3 which come by default with Ubuntu. The python + here-doc example below, is something I personally used to copy a large directory of files somewhere in the range of 40,000 items:

Источник

Argument list too long for ls

I get the following error when trying to ls *.txt | wc -l a directory that contains many files:

Does the threshold of this «Argument list» dependent on distro or computer’s spec? Usually, I’d pipe the result of such big result to some other commands ( wc -l for example), so I’m not concerned with limits of the terminal.

5 Answers 5

Your error message argument list too long comes from the ***** of ls *.txt .

This limit is a safety for both binary programs and your Kernel. See ARG_MAX, maximum length of arguments for a new process for more information about it, and how it’s used and computed.

There is no such limit on pipe size. So you can simply issue this command:

NB: On modern Linux, weird characters in filenames (like newlines) will be escaped with tools like ls or find , but still displayed from *****. If you are on an old Unix, you’ll need this command

NB2: I was wondering how one can create a file with a newline in its name. It’s not that hard, once you know the trick:

It depends mainly on your version of the Linux kernel.

You should be able to see the limit for your system by running

which tells you the maximum number of bytes a command line can have after being expanded by the shell.

In Linux = 2.6.25, the limit is either 128 KB, or 1/4 of your stack size (see ulimit -s ), whichever is larger.

See the execve(2) man page for all the details.

Unfortunately, piping ls *.txt isn’t going to fix the problem, because the limit is in the operating system, not the shell.

The shell expands the *.txt , then tries to call

and you have so many files matching *.txt that you’re exceeding the 128 KB limit.

You’ll have to do something like

(And see Shawn J. Goff’s comments below about file names that contain newlines.)

Even though ls produces more output than ls *.txt produces (or attempts to produce), it doesn’t run into the «argument too long» problem, because you’re not passing any arguments to ls . Note that grep takes a regular expression rather than a file matching pattern.

You might want to use:

(assuming your version of ls supports this option). This tells ls not to sort its output, which could save both time and memory — and in this case the order doesn’t matter, since you’re just counting files. The resources spent sorting the output are usually not significant, but in this case we already know you have a very large number of *.txt files.

And you should consider reorganizing your files so you don’t have so many in a single directory. This may or may not be feasible.

Источник