Debugging applications in linux

Debugging

This article or section needs expansion.

This page is mainly about how to gather more information in connection with bug reports. Even though the word «debug» is used, it is not intended as a guide for how to debug programs while developing.

Contents

When an application fails

Run it from the commandline

This article or section is a candidate for merging with General troubleshooting.

This article or section needs language, wiki syntax or style improvements. See Help:Style for reference.

If an application suddenly crashes, try running it from a terminal emulator. Type in the name of the application in lowercase letters. If you do not know the name of the executable, only the name of the package, the following command can find the name of the executable. Replace packagename with the name of the package:

Check availability of a core dump

A core dump is a file containing a process’s address space (memory) when the process terminates unexpectedly. If the application is compiled in a debug-friendly way, the «core» file can be used to find out where things went wrong.

The location of core dumps may vary depending on the operating system configuration. See core dump to find whether generation of core dump files is enabled on your system and where do they go.

Segmentation faults

There are several techniques that can be used to figure out what went wrong. Put your detective hat on.

gdb is an ancient and well tested application for debugging applications. See Debugging/Getting traces#Getting the trace for more instructions how to use it to obtain a trace. While running from gdb you might have to wait for the segfault. Afterwards, post the trace to a pastebin service and include the URL in your bug report.

Improved gdb output

This article or section is a candidate for merging with Debugging/Getting traces.

First recompile the application in question with debug options. Make sure that debug and !strip are in the options array in the PKGBUILD, for example:

Then install the package and run it again with gdb, as above.

If you have a «core» file, it can be used together with gdb to get a backtrace:

Valgrind

Assuming you have an unstripped binary without inlined functions, it is usually a good idea to also run that program through valgrind . valgrind is a tool that emulates a CPU and usually shows where things go wrong or provide additional info in addition to gdb.

it will provide a lot of helpful debug output if there is a crash. Consider -v and —leak-check=full to get even more info.

and run the output through kcachegrind to graphically explore the functions the program uses. If a program hangs, this makes it easier to pinpoint the location of the error.

Missing files or libraries

Strace

strace finds out in detail what an application is actually doing. If an application tries to open a file that is not there, it can be discovered by strace.

For finding which files a program named appname tries to open:

Save the output, post it to a pastebin service and keep the URL in handy.

LD_DEBUG

Setting LD_DEBUG=files gives another overview of what files an application is looking for. For an application named appname:

The output will end up in appname.log .

For more information, see ld-linux(8) .

Readelf

If you get no such file or directory when running an application, try the following command:

(replace /usr/bin/appname with the location of your executable)

Make sure the interpreter in question (like /lib/ld-linux-x86-64.so.2 ) actually exists. Install ld-lsb if need be.

If it is not written in C or C++, but perhaps in Python

Use file on the executable to get more information (replace appname» with your executable):

If it says ELF it is a binary executable and is usually written in C or C++. If it says Python script you know you are dealing with an application written in Python.

If it is a shell script, open up the shell script in a text editor and see (usually at the bottom of the file) if you can find the name of the real application (ELF file). You can then temporarily put «gdb» right in the shellscript, before the name of the executable, for debugging purposes. See the sections about gdb further up. Prefix the command with gdb —args if the executable in question needs arguments as well.

For pure shell scripts, you can also use bash -x script_name or bash -xv script_name .

For Python applications, the output will often say which file and line number the crash occurred at. If you are proficient with Python, you can try to fix this and include the fix in the bug report.

Report the bug

Please report a bug at https://bugs.archlinux.org and possibly also directly to the developers of the application in question, then include a link in the Arch Linux bug report. This helps us all.

However, if you think there is something wrong with the application itself, and not with how it is packaged, report the bug directly to upstream (which means the developers of the application). Normally, software streams from developers, through packagers/maintainers and down to users. Upstream means the other way, so for this case: directly to the developers of an application.

Источник

Linux Applications Debugging Techniques/The debugger

Contents

Preparations [ edit | edit source ]

Someday some hard to reproduce issue will be found on a production machine. Typically, such a machine is difficult to access, has no development environment and nothing can be installed on it. At most it will have gdb, but very likely not. Typically also, the heaviest user will be the one to find the issue and typically still, the heaviest user is the one being the bigger — money-wise. And that issue has to be root cause diagnosed and fixed.

Unless the application has been prepared for this forensic gathering moment, not much can be done to diagnose where the issue is. Thus, preparations should start with compilation:

• Have a «symbol server» and make sure it is reacheable. Compile on the symbol server.
• Compile the release with debugging symbols. Strip them if you do not want to ship them but keep them.
• Ship gdbserver with the application for remote debugging.

These preparations will allow one to:

• Debug the application running on any machine, including machines where there is no gdb installed.
• Debug from any machine that has network visibility to the symbol server.

Also, think beforehand how would you debug on the machine:

• Embed a breakpoint in the code, at hard of reach places of interest, then
  • Start the application
  • Attach to it with the debugger
  • Wait until the breakpoint is hit

The «symbol server» [ edit | edit source ]

One way to easily reach the right code from within the debugger is to build the binaries within an auto-mounted folder, each build in its own sub-folder. The same auto-mount share should be accessible from the machine you are debugging on.

Debian [ edit | edit source ]

• Install autofs

• In /etc/auto.master uncomment the line:

• In /etc/exports export the folder:

• As root: restart autofs & nfs and export the build share:

Redhat [ edit | edit source ]

• Export the folder: edit /etc/exports
• As root (RedHat): service autofs start

Finally, build the binaries within the automounted directory on the build machine (here the build machine is bear):

Notice the filename path that is resolved with the symbol information:

If the symbols have been stripped from the binaries, point gdb to the folders where the symbols are with the debug-file-directory directive.

References [ edit | edit source ]

Source Code [ edit | edit source ]

To point the debugger to the source files:

Remote debugging [ edit | edit source ]

• On the machine where the application runs (appmachine):
  • If gdbserver is not present, copy it over.
  • Start the application.
  • Start gdbserver: gdbserver gdbmachine:2345 --attach program
• On gdbmachine:
  • At the gdb prompt, enter: target remote appmachine:2345

Sometimes you may have to tunnel over ssh:

• On gdbmachine:
  • ssh -L 5432:appmachine:2345 user@appmachine
  • At the gdb prompt: target remote localhost:5432

References [ edit | edit source ]

Attaching to a process [ edit | edit source ]

Find out the PID of the process, then:

Debugging programs that spawn multiple children [ edit | edit source ]

• set detach-on-fork off
• see «all-stop» vs «non-stop» modes in the GDB documentation and their related settings

Embedding breakpoints in the source [ edit | edit source ]

On x86 platforms:

Or a more elaborate one:

This will break into the debugger on hard to reach conditions:

References [ edit | edit source ]

Data breakpoints (watchpoints) [ edit | edit source ]

Watchpoints can be implemented either in software either in hardware if the CPU supports it. Typically on an Intel processor there are eight debug registers out of which only four can be used for hardware breakpoints and this limits the number of watchpoints system wide.

Breakpoints conditional on caller [ edit | edit source ]

This requires gdb 7.9 or later, configured with python support:

The text user interface [ edit | edit source ]

GDB features a text user interface for code, disassembler and registers. For instance:

• Ctrl-x 1 will show the code pane
• Ctrl-x a will hide the TUI panes

None of the GUI interfaces to gdb (Qt Creator stands out for being intuitive and easy to use) can offer access to all of the gdb functionality.

curses gdb offers an improved TUI. A comprehensive list of debugger GUIs is available here.

Reverse debugging [ edit | edit source ]

As an example, reverse debugging is a functionality no GUI offers access to:

Register watch [ edit | edit source ]

You can watch registers. Note this will force the debugger to single step the debugged program and it will run very slowly:

References [ edit | edit source ]

.gdbinit [ edit | edit source ]

As a note, in upcoming gdb releases, .gdbinit will be replaced by gdb-gdb.gdb:

C++ support [ edit | edit source ]

Canned gdb macros [ edit | edit source ]

Mangling [ edit | edit source ]

gdb might need a bit of guidance with C++11 binaries:

Templates [ edit | edit source ]

One can eventually use templight to debug and profile templates.

Источник

How to Debug application in linux

Recently I was at some interview for Linux SYS Admin position and I’ve got the question «How do you debug some application in Linux».

I am not new in Linux word and I actually stuck with this question.

I am aware about «top», «ps» command, but may be there are some other advanced commands.

I suggest that there is no one specific answer, but I think there is a couple of ways to do it.

Can any one provide some way ‘How to debug some application’?.

1 Answer 1

I would check the logs. I ll explain my self:

Top and ps provide a brief overview about the system resources the application is using. This is of course useful but, as I get it, debugging an application includes getting to know the function of this application (e.g what tasks it performs, and generally any information it provides).

The first thing I would check are application logs. And by logs I mean every type of logs (application specific — or system logs).

Gdb and strace in my opinion are low level tools, and are useful when an application has unexpected behaviour, receives some sort of errors, of if you want to reverse-engineer its function, check the system-calls hierarchy etc. But this is probably not the case.

Any serious application you will use as a sysadmin will provide sufficient amount of logs, either by setting a debug flag on, or by default.

So in sort, I would answer : By checking logs (syslogs and application logs)

Источник

Debug Applications with gdb Command In Linux

gdb is the short form of GNU Debugger. A debugger is a tool used to search and find and get detailed information about bugs in application binaries. gdb is popular in the Linux community which is used by most of the IDE, Programming tools event in Android IDE’s. In this tutorial, we will look at how to start and use the basic features of gdb .

Example Code

During this tutorial, we will use the following simple application code which is written in C Programming language. This code just calculates factorial.

Install gdb For Ubuntu, Debian, Mint, Kali

We can install gdb for Ubuntu, Debian, Mint and Kali with the following lines.

Install gdb For CentOS, RHEL, Fedora

Installation for gdb in RPM based distros like CentOS, RHEL and Fedora

Compile Application

We will start by compiling our example application. We will use GCC or GNU Compiler Collection which is a defacto compiler for the Linux environment. gcc can be installed with the following line if not installed.

Now we will compile our application just providing the source code file name which is poftut.c in this case.

Compile Application with Debug Info

We can debug all ready compiled applications but there is some useful option that can be used to provide more information about the application while debugging. We can enable debugging information which can be used to match debugging code with source code and provides more detailed information. We can use -g option while compiling with gcc like below.

Setup Debugging By Running Executable

We can start applications for debugging with different methods but the most basic and practical way is starting with the application name. We will start poftut app like below.

We can see gdb interactive shell is started after loading app and related debugging information.

Start Debugging

We can start debugging the given application by using run command. But keep in mind that this will start process run which will end in this case without a problem. While running the standard output will be put to the screen. and the last thing is exit status which is normal in this case.

Inspect Crashes

One of the use case for gbd is inspecting crashes to find the cause. The application can be exited by crashing and in this case crash information is get by gdb . We can list crash information with backtrace command like below.

Breakpoints

In big applications, we generally need to inspect the flow of the application. While inspecting we need to look specific situation which is important for us. We can set breakpoints that will stop the execution of the application. We can check the current register and memory values easily. In this example, we set a break for the line printf .

List Breakpoints

We can list existing breakpoints and their hit count with the info break command like below.

List Register Values

We can list current register values with info registers command like below. Keep in mind that register values are very volatile and changes in each step in general.

Stepping

During debugging we generally want to go one step further which will change current memory and register values. We can use step command for this.

We can also spefiy the step count which is 3 in this case.

Источник