What is core dump linux

Core dump

A core dump is a file containing a process’s address space (memory) when the process terminates unexpectedly. Core dumps may be produced on-demand (such as by a debugger), or automatically upon termination. Core dumps are triggered by the kernel in response to program crashes, and may be passed to a helper program (such as systemd-coredump) for further processing. A core dump is not typically used by an average user, but may be passed on to developers upon request where it can be invaluable as a post-mortem snapshot of the program’s state at the time of the crash, especially if the fault is hard to reliably reproduce.

Contents

Disabling automatic core dumps

Users may wish to disable automatic core dumps for a number of reasons:

• Performance: generating core dumps for memory-heavy processes can waste system resources and delay the cleanup of memory.
• Disk space: core dumps of memory-heavy processes may consume disk space equal to, if not greater, than the process’s memory footprint if not compressed.
• Security: core dumps, although typically readable only by root, may contain sensitive data (such as passwords or cryptographic keys), which are written to disk following a crash.

Using sysctl

sysctl can be used to set the kernel.core_pattern to nothing to disable core dump handling. Create this file[1]:

To apply the setting immediately, use sysctl :

Using systemd

systemd’s default behavior is defined in /usr/lib/sysctl.d/50-coredump.conf , which sets kernel.core_pattern to call systemd-coredump . It generates core dumps for all processes in /var/lib/systemd/coredump . systemd-coredump behavior can be overridden by creating a configuration snippet in the /etc/systemd/coredump.conf.d/ directory with the following content[2][3]:

Then reload systemd’s configuration.

This method alone is usually sufficient to disable userspace core dumps, so long as no other programs enable automatic core dumps on the system, but the coredump is still generated in memory and systemd-coredump run.

Using PAM limits

The maximum core dump size for users logged in via PAM is enforced by limits.conf. Setting it to zero disables core dumps entirely. [4]

Using ulimit

Command-line shells such as bash or zsh provide a builtin ulimit command which can be used to report or set resource limits of the shell and the processes started by the shell. See bash(1) § SHELL BUILTIN COMMANDS or zshbuiltins(1) for details.

To disable core dumps in the current shell:

Making a core dump

To generate a core dump of an arbitrary process, first install the gdb package. Then find the PID of the running process, for example with pgrep:

Attach to the process:

Then at the (gdb) prompt:

Now you have a coredump file called core.2071 .

Where do they go?

The kernel.core_pattern sysctl decides where automatic core dumps go. By default, core dumps are sent to systemd-coredump which can be configured in /etc/systemd/coredump.conf . By default, all core dumps are stored in /var/lib/systemd/coredump (due to Storage=external ) and they are compressed with zstd (due to Compress=yes ). Additionally, various size limits for the storage can be configured.

To retrieve a core dump from the journal, see coredumpctl(1) .

Examining a core dump

Use coredumpctl to find the corresponding dump:

You need to uniquely identify the relevant dump. This is possible by specifying a PID , name of the executable, path to the executable or a journalctl predicate (see coredumpctl(1) and journalctl(1) for details). To see details of the core dumps:

Pay attention to «Signal» row, that helps to identify crash cause. For deeper analysis you can examine the backtrace using gdb:

When gdb is started, use the bt command to print the backtrace:

See Debugging/Getting traces if debugging symbols are requested, but not found.

Источник

Understand and configure core dumps on Linux

Every system needs running processes to fulfill its primary goal. But sometimes things go wrong and a process may crash. Depending on the configuration of the system a core dump is created. In other words, a memory snapshot of the crashed process is stored. The term core actually refers to the old magnetic core memory from older systems. Although this type of memory is no longer being used, we still use this term on Linux systems. Enough for history, let’s configure our Linux system to properly handle core dumps.

Table of Contents

Linux and core dumps

Most Linux systems have core dumps enabled by default. As always, there is a tradeoff to make here. On one hand, we want to gather data for improved stability and troubleshooting. On the other, we want to limit the debug data and avoid leaking sensitive data.

The first option is good for machines where unstable programs need to be investigated, like the workstation of a developer. The second option is better suited for production systems storing or processing sensitive data.

Disable core dumps

It makes sense to disable any core dumps on Linux by default for all your systems. This is because the files take up disk space and may contain sensitive data. So if you don’t need the core dumps for troubleshooting purposes, disabling them is a safe option.

Option 1: ulimit via the configuration file

To disable core dumps we need to set a ulimit value. This is done via the /etc/security/limits.conf file and defines some shell specific restrictions.

Good to know is that there are soft and hard limits. A hard limit is something that never can be overridden, while a soft limit might only be applicable for specific users. If we would like to ensure that no process can create a core dump, we can set them both to zero. Although it may look like a boolean (0 = False, 1 = True), it actually indicates the allowed size.

The asterisk sign means it applies to all users. The second column states if we want to use a hard or soft limit, followed by the columns stating the setting and the value.

Option 2: configure ulimit via profile

The values for ulimit can also be set via /etc/profile or a custom file in the /etc/profile.d directory. The latter is preferred when it is available. For example by creating a file named /etc/profile.d/disable-coredumps.sh.

echo “ulimit -c 0 > /dev/null 2>&1” > /etc/profile.d/disable-coredumps.sh

This command adds the setting to a new file and sets both the soft and hard limit to zero. Each user gets this value when logging in.

Besides ulimit settings, there are also kernel settings to consider. So choosing one of the options is the first step.

Option 3: disable via systemd

When using systemd and the systemd-coredump service, change the coredump.conf file. This file is most likely located at /usr/lib/sysctl.d/50-coredump.conf. As systemd has a set of files, ensure to check the others like:

Set the Storage setting to ‘none’. Then configure ProcessSizeMax to limited the maximum size to zero.

Typically it is sufficient to just reload the systemd configuration.

If this still creates a core dump, then reboot the system.

Disable setuid processes dumping their memory

Processes with elevated permissions (or the setuid bit), might be still able to perform a core dump, depending on your other settings. As these processes usually have more access, they might contain more sensitive data segments in memory. So time to change this as well. The behavior can be altered with a sysctl key, or directly via the /proc file system. For permanent settings, the sysctl command and configuration is typically used. A setting is called a ‘key’, which has a related value attached to it (also known as a key-value pair).

To disable program with the setuid bit to dump, set the fs.suid_dumpable to zero.

Reload the sysctl configuration with the -p flag to activate any changes you made.

Just want to test without making permanent changes? Use sysctl -w followed by the key=value.

Tip: Using sysctl you can tune your system and is a good way to harden the Linux kernel.

Enable core dumps

The primary reason to allow core dumps is for troubleshooting purposes. The dumped memory of the process can be used for debugging issues, usually by more experienced developers. A software vendor may ask to enable core dumps. Usually to discover why a process crashed in the first place and find the related routine that caused it.

Enabling core dumps on Linux is similar to disabling them, except that a few specific details should be configured. For example, if you only need details from a particular program, you can use soft limits. This is done by using -S which indicates that it is a soft limit. The -c denotes the size of a core dump.

Next step is to only allow ‘my-program-to-troubleshoot’ to create a core dump.

ulimit -S -c unlimited my-program-to-troubleshoot

If you want to allow all processes to use core dumps, use the line above without the program, or set a system limit in /etc/security/limits.conf

Troubleshoot setuid binaries

Binaries that have a setuid bit set, can run with root permissions. This special type of access needs to be restricted as much as possible. Also for the creation of core dumps, it needs to be configured properly. This is done with the sysctl fs.suid_dumpable key.

• 0 – disabled
• 1 – enabled
• 2 – enabled with restrictions

So if you like to troubleshoot programs with a setuid bit set, you can temporarily change the fs.suid_dumpable to 1 or 2. Setting it to 2 is preferred as this makes the core dumps only readable to the root user. This is a good alternative for systems with sensitive data. Setting the option to 1 is better suited for personal development systems.

Create normal dump files

One of the big mysteries with Linux systems is where the core dumps are located. Linux has a trick in place to capture core dumps. This particular setting is done via the sysctl kernel.core_pattern setting or /proc/sys/kernel/core_pattern. Most systems will have a pipe ( | ) in this setting to indicate that a program needs to take care of the generated data. So if you wonder where your core dump goes, follow the pipe!

Core dumps on Ubuntu systems are typically going to Apport. For Red Hat based systems it may be redirected to Automatic Bug Reporting Tool (ABRT).

You can temporarily change this setting, by echoing “core” to that file, or use the sysctl utility.

An important note is that this change might not be enough. It depends also on your fs.suid_dumpable setting. A warning will be logged to your kernel logger if that is the case.

Sep 06 15:51:18 hardening kernel: Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.

When needed set your core_pattern to a full path, optionally with variables defining who was running it, the PID, etc.

sysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p

In this example, our dumps will contain the user id, program name, and process id.

Systemd core dumps

When using a modern Linux distribution you will most likely have systemd enabled. You might need to override settings via /etc/sysctl.d/50-coredump.conf and define how and where you want to store your core dumps.

Using systemd-coredump

Your kernel.core_pattern may be defined to use the systemd-coredump utility. The default path where core dumps are stored is then in /var/lib/systemd/coredump.

Testing your configuration

Most other tutorials just give you the settings to be configured. But how would you know things work as expected? You will need to test it!

Create a core dump

Option 1: Create an unstable program

Let’s create a simple program. Its primary goal is to crash when being executed and then optionally create a core dump. Install gcc on your system and create a file crash.c in your home directory.

This program will start the main function and return an integer value (number). However, it is dividing 1 by zero, which is not allowed and will crash. The next step is compiling our little buggy program.

Our unstable little program

Even the compiler shows our program contains a serious issue and displays a warning about it. Now let’s run it and see if this is the case.

From this single line, we can actually learn a few things. First of all that it quit with an exception, specifically referring to floating points. This is a decimal number format for programs, so it may indicate that something happened while doing some math. Another conclusion is that the core is dumped due to the (core dumped) addition at the end. If core dumps were disabled, this would not appear.

Great, so with this crash above we have now a dumped file, right? Not exactly. Depending on your Linux distribution things might not as simple as it looks. Each distribution deals differently with core dumps and the default settings. Most recent Linux distributions also use systemd now and the rules have slightly been changed with that as well. Depending on your configuration, you might need to search for your core dumps. So here are some tips to ensure everything is configured correctly.

Option 2: kill a running process

Instead of using a test program, you can also terminate an existing process. This is done by using the SIGSEGV, which is short for segmentation violation and also known as a segmentation fault.

If you replace PID with “$$” the current program (most likely your shell) will crash. Everything for science, right?

Option 3: using gdb

If you have the developer debugging tool gdb installed, then attach to a process of choice using its process ID (PID).

Then when at the gdb prompt, generate the core dump by invoking the generate-core-file instruction. After using this command, it should return you output.

Check ulimit settings

The ulimit settings define what may happen when a program crashes. So it is safe to first check this, for both root and a normal non-privileged user.

Check hard limit for core dumps:

Check soft limit as well:

Check the core pattern

Use the /proc file system to gather the value and change it temporarily during testing. If you prefer using sysctl, then query the kernel.core_pattern key.

It might show something like this:

In this case, a crash will be piped to the apport utility. So this means that crashes are going to be analyzed by Apport. Normally crashes are found in /var/crash, but may also be in /var/spool or /var/lib/systemd/coredump on other Linux distributions.

Check the journal (systemd)

In our case journalctl shows our crash, so that’s a start.

Sep 06 15:19:23 hardening kernel: traps: crash[22832] trap divide error ip:4004e5 sp:7fff4c2fc650 error:0 in crash[400000+1000]

After checking all these settings you should be able to create a nice core dump.

Conclusion

Core dumps can be useful for troubleshooting, but a disaster for leaking sensitive data. Disable core dumps when possible, and only enable them when really needed. In such case check if the files are stored safely, so normal users can’t see the data. And independently of what choice you made, always test if your configuration does work exactly as you expect it to work.

Do you have other tips regarding core dumps? Share them in the comments!

Keep learning

So you are interested in Linux security? Join the Linux Security Expert training program, a practical and lab-based training ground. For those who want to become (or stay) a Linux security expert.

Run automated security scans and increase your defenses. Lynis is an open source security tool to perform in-depth audits. It helps with system hardening, vulnerability discovery, and compliance.

Continue reading

No related posts found.

9 comments

I want to disable core dumps completely.
I followed the steps provided by you and did the changes.

1. appended in the /etc/security/limits.conf
* hard core 0

2. In /etc/sysctl.conf changed
fs.suid_dumpable=0

3. Executed “sysctl -p”

4. There are no extra files in /etc/security/limits.d/*conf
that overwrites the /etc/security/limits.conf entry.

5. “ulimit -H -c” gives 0
“ulimit -S -c” gives 0

6. core pattern is “cat /proc/sys/kernel/core_pattern”
|/opt/sonus/platform/core.sh %e %p %h %t
where core.sh is a script we have to simplify and write the cores

SO HERE using crash program I am still getting the core dump

7. But if core pattern is “cat /proc/sys/kernel/core_pattern” simply a file say
/opt/sonus/platform/core.%e %p %h %t
then cores are disabled

So when I am using pipe why am I getting core dumps even after disabling them.

Источник