Linux pipe errors to file

Linux pipes tips & tricks

Pipe — что это?

Pipe (конвеер) – это однонаправленный канал межпроцессного взаимодействия. Термин был придуман Дугласом Макилроем для командной оболочки Unix и назван по аналогии с трубопроводом. Конвейеры чаще всего используются в shell-скриптах для связи нескольких команд путем перенаправления вывода одной команды (stdout) на вход (stdin) последующей, используя символ конвеера ‘|’:

grep выполняет регистронезависимый поиск строки “error” в файле log, но результат поиска не выводится на экран, а перенаправляется на вход (stdin) команды wc, которая в свою очередь выполняет подсчет количества строк.

Логика

Конвеер обеспечивает асинхронное выполнение команд с использованием буферизации ввода/вывода. Таким образом все команды в конвейере работают параллельно, каждая в своем процессе.

Размер буфера начиная с ядра версии 2.6.11 составляет 65536 байт (64Кб) и равен странице памяти в более старых ядрах. При попытке чтения из пустого буфера процесс чтения блокируется до появления данных. Аналогично при попытке записи в заполненный буфер процесс записи будет заблокирован до освобождения необходимого места.
Важно, что несмотря на то, что конвейер оперирует файловыми дескрипторами потоков ввода/вывода, все операции выполняются в памяти, без нагрузки на диск.
Вся информация, приведенная ниже, касается оболочки bash-4.2 и ядра 3.10.10.

Простой дебаг

Исходный код, уровень 1, shell

Т. к. лучшая документация — исходный код, обратимся к нему. Bash использует Yacc для парсинга входных команд и возвращает ‘command_connect()’, когда встречает символ ‘|’.
parse.y:
Также здесь мы видим обработку пары символов ‘|&’, что эквивалентно перенаправлению как stdout, так и stderr в конвеер. Далее обратимся к command_connect():make_cmd.c:
где connector это символ ‘|’ как int. При выполнении последовательности команд (связанных через ‘&’, ‘|’, ‘;’, и т. д.) вызывается execute_connection():execute_cmd.c:

PIPE_IN и PIPE_OUT — файловые дескрипторы, содержащие информацию о входном и выходном потоках. Они могут принимать значение NO_PIPE, которое означает, что I/O является stdin/stdout.
execute_pipeline() довольно объемная функция, имплементация которой содержится в execute_cmd.c. Мы рассмотрим наиболее интересные для нас части.
execute_cmd.c:
Таким образом, bash обрабатывает символ конвейера путем системного вызова pipe() для каждого встретившегося символа ‘|’ и выполняет каждую команду в отдельном процессе с использованием соответствующих файловых дескрипторов в качестве входного и выходного потоков.

Исходный код, уровень 2, ядро

Обратимся к коду ядра и посмотрим на имплементацию функции pipe(). В статье рассматривается ядро версии 3.10.10 stable.
fs/pipe.c (пропущены незначительные для данной статьи участки кода):
Если вы обратили внимание, в коде идет проверка на флаг O_NONBLOCK. Его можно выставить используя операцию F_SETFL в fcntl. Он отвечает за переход в режим без блокировки I/O потоков в конвеере. В этом режиме вместо блокировки процесс чтения/записи в поток будет завершаться с errno кодом EAGAIN.

Максимальный размер блока данных, который будет записан в конвейер, равен одной странице памяти (4Кб) для архитектуры arm:
arch/arm/include/asm/limits.h:
Для ядер >= 2.6.35 можно изменить размер буфера конвейера:
Максимально допустимый размер буфера, как мы видели выше, указан в файле /proc/sys/fs/pipe-max-size.

Tips & trics

В примерах ниже будем выполнять ls на существующую директорию Documents и два несуществующих файла: ./non-existent_file и. /other_non-existent_file.

Перенаправление и stdout, и stderr в pipe

или же можно использовать комбинацию символов ‘|&’ (о ней можно узнать как из документации к оболочке (man bash), так и из исходников выше, где мы разбирали Yacc парсер bash):

Перенаправление _только_ stderr в pipe

Shoot yourself in the foot
Важно соблюдать порядок перенаправления stdout и stderr. Например, комбинация ‘>/dev/null 2>&1′ перенаправит и stdout, и stderr в /dev/null.

Получение корректного кода завершения конвейра

По умолчанию, код завершения конвейера — код завершения последней команды в конвеере. Например, возьмем исходную команду, которая завершается с ненулевым кодом:
И поместим ее в pipe:
Теперь код завершения конвейера — это код завершения команды wc, т.е. 0.

Обычно же нам нужно знать, если в процессе выполнения конвейера произошла ошибка. Для этого следует выставить опцию pipefail, которая указывает оболочке, что код завершения конвейера будет совпадать с первым ненулевым кодом завершения одной из команд конвейера или же нулю в случае, если все команды завершились корректно:
Shoot yourself in the foot
Следует иметь в виду “безобидные” команды, которые могут вернуть не ноль. Это касается не только работы с конвейерами. Например, рассмотрим пример с grep:
Здесь мы печатаем все найденные строки, приписав ‘new_’ в начале каждой строки, либо не печатаем ничего, если ни одной строки нужного формата не нашлось. Проблема в том, что grep завершается с кодом 1, если не было найдено ни одного совпадения, поэтому если в нашем скрипте выставлена опция pipefail, этот пример завершится с кодом 1:
В больших скриптах со сложными конструкциями и длинными конвеерами можно упустить этот момент из виду, что может привести к некорректным результатам.

Источник

How to Fix Linux Broken Pipe Errors

While it’s quite rare, it’s possible that you’ll find yourself unable to install any packages at all in an otherwise seemingly stable installation of Linux. Debian, Ubuntu and those distributions derived from them rely on the apt-get package manager. Generally, if you have administrator superuser access, then typing apt-get install -f should install additional packages if needed. However, as rare as it is, you might get a message that reads dpk-deb: error and then proceeds to instruct that you have a broken pipe.

Usually the term broken pipe refers to a misuse of the character | at the command line, which is often called a pipe, especially to those with a background in MS-DOS. In this case, though, you might not have typed a command incorrectly. Rather, if you didn’t use the | character when typing the apt-get command, then you might have a problem involving your file system that forced a read-only mount. While it might seem embarrassing, the first step is to check and ensure you’ve typed the installation command correctly before proceeding. If there’s no problem, then the next step is to inspect the file system.

Method 1: Inspecting the Typed Command

While more than likely this isn’t the case, it’s important to make sure that you typed the installation command correctly before you assume the worst. Even the most elite programmers sometimes end up mistyping something, and it’s easy to type the | character by mistake. Don’t push up on the keyboard to recall the last command. Type it again individually. Try running apt-get install -f or apt-get update, since these commands aren’t destructive. If you still get the error after repeatedly reissuing it, then try rebooting the machine. If you still receive it, then you might be suffering from a file system problem. The broken pipe error message might be nothing more than a symptom of a bigger problem.

Method 2: Recovering from Broken Pipe Errors Caused by File System Dysfunctions

Should you be unable to fix it merely by typing the command over again, then pay special attention to whether or not you see some sort of an error that reads “Read-only file system” in the terminal output. This indicates that, for whatever reason, your boot partition is mounted as a read-only volume. Generally, this means it’s suffered some sort of file system problem and the GNU/Linux operating system doesn’t feel it’s safe to write to it. This safety mechanism means that while some data might be corrupt, Linux has done it’s best to salvage your install.

You might possibly receive some kind of strange message when you try to use the sudo command that is also caused by this same type of error. Such a message would begin with “sudo: unable to open” followed by the name of some resource. Since all resources in Unix are stored as files, a read-only partition mount might keep sudo from working.

If either of these cases is true, then you need to shutdown your system. If you have a pre-made USB drive with an ISO of a clean Linux install, then insert it and hold down whichever key your BIOS or UEFI system requires to boot from a removable device. Should you instead have a separate Linux partition on your boot drive, then you should access this through GRUB when you restart. In either case, you need to have a stable environment to work from that can access the damaged partition without booting from it.

Assuming you have some type of data that you haven’t backed up yet, then take this opportunity to back that data up to another partition or a removable device if Linux allows you to attach mounted devices in this state. Usually you can’t mount devices to a read-only file system, so you will have needed to reboot your system into a live USB or potentially DVD environment to perform backups. You’ll want to run the appropriate fsck command for the file system installed on your boot drive to restore it to a stable state. For instance, let’s assume you have a damaged /dev/sda1 partition that you formatted with an ext4 file structure. If that’s the case, then from a root terminal inside of the live USB or DVD environment you could issue fsck.ext4 -fv /dev/sda1 to check the file system. You can also use fsck.ext2 or fsck.ext3 if you booted from either of these two types of ext file system. In theory, all three of these call the e2fsck program anyway, and may just be soft links.

This will provide verbose output and force it to scan even if it appears clean. You may be interested in using -c or -ck to run a surface scan with the badblocks program in case you suspect that there was some sort of geometry problem with the disk. Once you’ve run the program, reboot from the partition if it’s stable and then issue the sudo mount -o rw,remount / command from a terminal inside this boot to restore read-write access. You’ll need root access to look in the /lost+found directory, but it’s important you do since fsck.ext# might have put some lost clusters there. If you’re missing files once you’re back inside of your GNU/Linux install, then they might be here, though they’ll be renamed. Use the file command in the terminal to tell which type each is if at all possible.

If you don’t have a live USB or DVD to boot from, then you’ll need to make one from a clean machine from an ISO downloaded for a distribution of Linux. Since you’re only using terminal commands, you might not even need a boot device for your distribution. Some, like KNOPPIX, are designed explicitly for fixing problems like this. If you’re working with a netbook or laptop that features an SD or microSD card slot, then you could also boot from an ISO burned to an SDHC or microSDHC card. The same goes for, say, Ubuntu tablets that have such a slot.

Источник

How to fix Broken Pipe Error in Linux

Have you ever encountered a situation in which you were unable to download any package on your Linux machine ? Or you might have probably seen an error like package not installed? This kind of error can easily be fixed with a command like “sudo apt install –f”. On rare occasions, you may have experienced a broken pipe error.

A pipe in Linux / Unix connects two processes , one of them has read-end of the file and the other one has the write-end of the file. When a process writes to a Pipe, it gets stored in a buffer and gets retrieved by the other process. Broken pipe occurs when a process prematurely exits from either end and the other process has not yet closed the pipe.

Example use case:

A user has just recently reinstalled RVM (Ruby Version Manager) after he performed a fresh install of Ubuntu.

He then opened up the terminal and issued the command:

type rvm | head -1

This issued the following error:

rvm is a function -bash: type: write error: Broken pipe

What happened here is that when the user runs the command type rvm | head -1 , bash has actually executed type rvm in one process and head -1 in another process. The stdout of the type part is connected to the “write” end of a pipe whereas the stdin of the head part is hooked up to the “read” end. Note that the two processes have run concurrently ( at the same time ).

The head -1 process has carried out a read operation of data from stdin , then prints out a single line (as dictated by the -1 option) before exiting, causing therefore the “read” end of the pipe to be closed. Since the rvm function has quite a long data stream (about 11 kB after having been bash parsed and reconstructed), which means that head exits yet at the same time type still has some data to write out (few KB).

Since type is trying to carry out a write operation to a pipe whose other end has therefore been closed – a brokenpipe routine or the write() function that it invoked, will return an EPIPE error which is known as “Broken pipe”.

Inspecting the Command

In most cases, this might not be the case but the first step you should check is whether the command issued was right or not. You should reissue the command and check whether it gets executed or not. You can also try issuing commands like “sudo apt update” and “sudo apt install –f” as these commands are not destructive in nature. If your problem still persists, try rebooting the machine and see whether the problem was resolved or not.

Fixing a Problem with File System

When you have issued the commands mentioned earlier multiple times and you still get the error, check whether the error reads something like “read-only file system” in the terminal output. This may be caused when your boot partition gets mounted as read-only for some reason. The problem could be caused by some faulty software installation when the system decides that it is not safe to write to the drive.

The other cause might be when you try to install something from apt and the installer needs to access some resource in read mode, but cannot perform the read operation properly. It may throw an error like “sudo: cannot mount”. This error occurs because most of the ‘entities’ in Linux are files and in order to read a resource, Linux would need to open that file and read it. If however another process is currently using that resource, then it may not be possible to read the file. Also, when the reading process exits abruptly and does not close the file, it may corrupt the file until the next boot.

If you still cannot access the files even after rebooting, then the problem could be bigger than anticipated. You may have a broken file system. To resolve this issue, you may need a stable Linux environment in order to work on the broken system. The best way to do this is to boot from a Live Linux USB drive and work from it.

This is the right moment to backup all your data. Although the following steps are safe, you should make sure to store your data on a secure device.

Once you boot into a Live USB drive you should start to check for the partitions with a corrupt file system. To do so, issue the following command:

sudo fsck.ext4 -fv /dev/sdaX”

Note that here X stands for the partition that you are trying to scan. Note that this command is for partitions of type ext4. If you have a partition of type ext3 or ext2 you will need to replace the command with “fsck.ext3” and “fsck.ext2” respectively. This will scan your drive and print the output on the terminal (note the -v flag). Alternatively, you can specify a -c flag to surface-scan the drive; it will look for bad sectors on the drive.

Once you have done this, your partition should hopefully been fixed. Now Boot into your machine and issue the command:

sudo mount -o rw,remount /

This will restore the read/write permission on your drive and will therefore solve the broken pipe issue.

Afterwords

You have just seen one solution to resolve the broken pipe issue, but a broken pipe is not a problem, it could be a symptom of a larger problem more often than not. You can have a broken pipe whenever you are trying to read a stream like resource that gets closed prematurely. .

If you like the content, we would appreciate your support by buying us a coffee. Thank you so much for your visit and support.

Источник