Linux process core dumped

HowTo: Debug Crashed Linux Application Core Files Like A Pro

C ore dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Turn On Core File Creation Support

By default most Linux distributions turn off core file creation (at least this is true for RHEL, CentOS, Fedora and Suse Linux). You need to use the ulimit command to configure core files.

See The Current Core File Limits

Type the following command:
# ulimit -c
Sample outputs:

The output 0 (zero) means core file is not created.

Change Core File Limits

In this example, set the size limit of core files to 75000 bytes:
# ulimit -c 75000

HowTo: Enable Core File Dumps For Application Crashes And Segmentation Faults

Edit /etc/profile file and find line that read as follows to make persistent configuration:

Update it as follows:

Save and close the file. Edit /etc/sysctl.conf, enter:
# vi /etc/sysctl.conf
Append the following lines:

Save and close the file. Where,

1. kernel.core_uses_pid = 1 – Appends the coring processes PID to the core file name.
2. fs.suid_dumpable = 2 – Make sure you get core dumps for setuid programs.
3. kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t – When the application terminates abnormally, a core file should appear in the /tmp. The kernel.core_pattern sysctl controls exact location of core file. You can define the core file name with the following template whih can contain % specifiers which are substituted by the following values when a core file is created:
   • %% – A single % character
   • %p – PID of dumped process
   • %u – real UID of dumped process
   • %g – real GID of dumped process
   • %s – number of signal causing dump
   • %t – time of dump (seconds since 0:00h, 1 Jan 1970)
   • %h – hostname (same as ’nodename’ returned by uname(2))
   • %e – executable filename

Finally, enable debugging for all apps, enter (Redhat and friends specific):
# echo «DAEMON_COREFILE_LIMIT=’unlimited'» >> /etc/sysconfig/init
Reload the settings in /etc/sysctl.conf by running the following command:
# sysctl -p

How Do I Enable Core Dumping For Specific Deamon?

To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/lighttped and add line as follows:

Please note that DAEMON_COREFILE_LIMIT is Redhat specific, for all other distro add configuration as follows:

Save and close the file. Restart / reload lighttpd:
# /etc/init.d/lighttpd restart
# su — lighttpd
$ ulimit -c
Sample outputs:

Now, you can send core files to vendor or software writes.

How Do I Read Core Files?

You need use the gdb command as follows:
$ gdb /path/to/application /path/to/corefile
See the gdb command man page for more information.

strace command

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers. See how to use the strace command under Linux to debug the problems.

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

Recommended readings:

• Debugging Tip: Trace the Process and See What It is Doing with strace
• The Art of Debugging with GDB, DDD, and Eclipse
• man pages core(5), strace, and bash

Stay tunned for gdb tutorial which will explains how to use generated core file to track down problem.

Category	List of Unix and Linux commands
Documentation	help • mandb • man • pinfo
Disk space analyzers	df • duf • ncdu • pydf
File Management	cat • cp • less • mkdir • more • tree
Firewall	Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04
Linux Desktop Apps	Skype • Spotify • VLC 3
Modern utilities	bat • exa
Network Utilities	NetHogs • dig • host • ip • nmap
OpenVPN	CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04
Package Manager	apk • apt
Processes Management	bg • chroot • cron • disown • fg • glances • gtop • jobs • killall • kill • pidof • pstree • pwdx • time • vtop
Searching	ag • grep • whereis • which
Shell builtins	compgen • echo • printf
Text processing	cut • rev
User Information	groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w
WireGuard VPN	Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04

Comments on this entry are closed.

Great article, bravo 🙂

pity there’s actually nothing about debugging..

This part explained how to configure core dumps. I’m working on follow up article using gdb; so stay tuned.

zahn pan tussi great ho topho kabul

Your site is awesome especially for newbies and intermediates. Very clear and concise.

“strace command” link at the end of the article is not correct and pointing to the current web page only.

Thanks for the heads up!

@storm, thanks for the kind words!

Really nice info

Thanks for this article

Nice article, just maybe make the subject line relevant to the article…

“How to enable core dumps” instead of “Debug apps”…esp since there is virtually nothing about debugging in this article.

Thanks for nice material, but i want to know how to analyze core dump
thanks again

thanks for nice article, but how can we analyze core dump ?
thanks again

you send the linux commnad on my Email-ID

Please, explain more clear wich line add into /etc/sysconfig/daemon-file file, and which line into /etc/init.d/lighttped file; in the part about Specific Deamon?

Anyboby ask me how do i read core files .
how to use gdb command .

Just wondering why setting the following to “2” and not “1”?…

i deleted the latest core file by mistake. now the core file is not getting created and it seems some functions like NTP is not working.

How to solve this issue?

Anyone knows how to create Core Dump of entire System?

Thanks for the information. Life becomes easy with people like you.

Misleading title. This is not about debugging, much less, like a pro. It’s so frustrating when articles exist only to lure users that may click on ads.

Do people really pay that much attention to titles? I came heree for the content, which was how to enable core dumps on linux. To that end, it worked like a charm.

I’m returning to *nix programming after a long absence, and was trying to understand the use of DAEMON_COREFILE_LIMIT. What I haven’t seen mentioned is that this environment variable is used by the shell script function ‘daemon()’ which is included in the group of functions in etc/init.d/functions. If you start your daemon by calling daemon(), that variable will get used in a call to ‘ulimit’ by the shell. Otherwise, it will have no effect. I haven’t verified this yet (getting access to our servers is a slow process), but it seems that our custom shell script to start the daemon can do the same thing by calling ulimit directly.

root@duty-05-ubuntu:/var# echo “DAEMON_COREFILE_LIMIT=’unlimited’” >> /etc/sysconfig/init
bash: /etc/sysconfig/init: No such file or directory
root@duty-05-ubuntu:/var#

Article does not tell how to actually does what the title says, and sadly google shows this as first page for actual topic in title.

If I do like kill -9 named Can I get core dump?

You have an error in your write up. Please correct.

You wrote, “To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/lighttped and add line as follows:”

It should read instead, “To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/sysconfig/lighttped and add line as follows:”

Источник

Core dump

A core dump is a file containing a process’s address space (memory) when the process terminates unexpectedly. Core dumps may be produced on-demand (such as by a debugger), or automatically upon termination. Core dumps are triggered by the kernel in response to program crashes, and may be passed to a helper program (such as systemd-coredump) for further processing. A core dump is not typically used by an average user, but may be passed on to developers upon request where it can be invaluable as a post-mortem snapshot of the program’s state at the time of the crash, especially if the fault is hard to reliably reproduce.

Contents

Disabling automatic core dumps

Users may wish to disable automatic core dumps for a number of reasons:

• Performance: generating core dumps for memory-heavy processes can waste system resources and delay the cleanup of memory.
• Disk space: core dumps of memory-heavy processes may consume disk space equal to, if not greater, than the process’s memory footprint if not compressed.
• Security: core dumps, although typically readable only by root, may contain sensitive data (such as passwords or cryptographic keys), which are written to disk following a crash.

Using sysctl

sysctl can be used to set the kernel.core_pattern to nothing to disable core dump handling. Create this file[1]:

To apply the setting immediately, use sysctl :

Using systemd

systemd’s default behavior is defined in /usr/lib/sysctl.d/50-coredump.conf , which sets kernel.core_pattern to call systemd-coredump . It generates core dumps for all processes in /var/lib/systemd/coredump . systemd-coredump behavior can be overridden by creating a configuration snippet in the /etc/systemd/coredump.conf.d/ directory with the following content[2][3]:

Then reload systemd’s configuration.

This method alone is usually sufficient to disable userspace core dumps, so long as no other programs enable automatic core dumps on the system, but the coredump is still generated in memory and systemd-coredump run.

Using PAM limits

The maximum core dump size for users logged in via PAM is enforced by limits.conf. Setting it to zero disables core dumps entirely. [4]

Using ulimit

Command-line shells such as bash or zsh provide a builtin ulimit command which can be used to report or set resource limits of the shell and the processes started by the shell. See bash(1) § SHELL BUILTIN COMMANDS or zshbuiltins(1) for details.

To disable core dumps in the current shell:

Making a core dump

To generate a core dump of an arbitrary process, first install the gdb package. Then find the PID of the running process, for example with pgrep:

Attach to the process:

Then at the (gdb) prompt:

Now you have a coredump file called core.2071 .

Where do they go?

The kernel.core_pattern sysctl decides where automatic core dumps go. By default, core dumps are sent to systemd-coredump which can be configured in /etc/systemd/coredump.conf . By default, all core dumps are stored in /var/lib/systemd/coredump (due to Storage=external ) and they are compressed with zstd (due to Compress=yes ). Additionally, various size limits for the storage can be configured.

To retrieve a core dump from the journal, see coredumpctl(1) .

Examining a core dump

Use coredumpctl to find the corresponding dump:

You need to uniquely identify the relevant dump. This is possible by specifying a PID , name of the executable, path to the executable or a journalctl predicate (see coredumpctl(1) and journalctl(1) for details). To see details of the core dumps:

Pay attention to «Signal» row, that helps to identify crash cause. For deeper analysis you can examine the backtrace using gdb:

When gdb is started, use the bt command to print the backtrace:

See Debugging/Getting traces if debugging symbols are requested, but not found.

Источник