Memory leaks с linux

Kernel Memory Leak Detector¶

Kmemleak provides a way of detecting possible kernel memory leaks in a way similar to a tracing garbage collector, with the difference that the orphan objects are not freed but only reported via /sys/kernel/debug/kmemleak. A similar method is used by the Valgrind tool ( memcheck —leak-check ) to detect the memory leaks in user-space applications.

Usage¶

CONFIG_DEBUG_KMEMLEAK in “Kernel hacking” has to be enabled. A kernel thread scans the memory every 10 minutes (by default) and prints the number of new unreferenced objects found. If the debugfs isn’t already mounted, mount with:

To display the details of all the possible scanned memory leaks:

To trigger an intermediate memory scan:

To clear the list of all current possible memory leaks:

New leaks will then come up upon reading /sys/kernel/debug/kmemleak again.

Note that the orphan objects are listed in the order they were allocated and one object at the beginning of the list may cause other subsequent objects to be reported as orphan.

Memory scanning parameters can be modified at run-time by writing to the /sys/kernel/debug/kmemleak file. The following parameters are supported:

disable kmemleak (irreversible)

enable the task stacks scanning (default)

disable the tasks stacks scanning

start the automatic memory scanning thread (default)

stop the automatic memory scanning thread

set the automatic memory scanning period in seconds (default 600, 0 to stop the automatic scanning)

trigger a memory scan

clear list of current memory leak suspects, done by marking all current reported unreferenced objects grey, or free all kmemleak objects if kmemleak has been disabled.

dump information about the object found at

Kmemleak can also be disabled at boot-time by passing kmemleak=off on the kernel command line.

Memory may be allocated or freed before kmemleak is initialised and these actions are stored in an early log buffer. The size of this buffer is configured via the CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE option.

If CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF are enabled, the kmemleak is disabled by default. Passing kmemleak=on on the kernel command line enables the function.

If you are getting errors like “Error while writing to stdout” or “write_loop: Invalid argument”, make sure kmemleak is properly enabled.

Basic Algorithm¶

The memory allocations via kmalloc() , vmalloc() , kmem_cache_alloc() and friends are traced and the pointers, together with additional information like size and stack trace, are stored in a rbtree. The corresponding freeing function calls are tracked and the pointers removed from the kmemleak data structures.

An allocated block of memory is considered orphan if no pointer to its start address or to any location inside the block can be found by scanning the memory (including saved registers). This means that there might be no way for the kernel to pass the address of the allocated block to a freeing function and therefore the block is considered a memory leak.

The scanning algorithm steps:

mark all objects as white (remaining white objects will later be considered orphan)

scan the memory starting with the data section and stacks, checking the values against the addresses stored in the rbtree. If a pointer to a white object is found, the object is added to the gray list

scan the gray objects for matching addresses (some white objects can become gray and added at the end of the gray list) until the gray set is finished

the remaining white objects are considered orphan and reported via /sys/kernel/debug/kmemleak

Some allocated memory blocks have pointers stored in the kernel’s internal data structures and they cannot be detected as orphans. To avoid this, kmemleak can also store the number of values pointing to an address inside the block address range that need to be found so that the block is not considered a leak. One example is __vmalloc().

Testing specific sections with kmemleak¶

Upon initial bootup your /sys/kernel/debug/kmemleak output page may be quite extensive. This can also be the case if you have very buggy code when doing development. To work around these situations you can use the ‘clear’ command to clear all reported unreferenced objects from the /sys/kernel/debug/kmemleak output. By issuing a ‘scan’ after a ‘clear’ you can find new unreferenced objects; this should help with testing specific sections of code.

To test a critical section on demand with a clean kmemleak do:

Then as usual to get your report with:

Freeing kmemleak internal objects¶

To allow access to previously found memory leaks after kmemleak has been disabled by the user or due to an fatal error, internal kmemleak objects won’t be freed when kmemleak is disabled, and those objects may occupy a large part of physical memory.

In this situation, you may reclaim memory with:

Kmemleak API¶

See the include/linux/kmemleak.h header for the functions prototype.

kmemleak_init — initialize kmemleak

kmemleak_alloc — notify of a memory block allocation

kmemleak_alloc_percpu — notify of a percpu memory block allocation

kmemleak_vmalloc — notify of a vmalloc() memory allocation

kmemleak_free — notify of a memory block freeing

kmemleak_free_part — notify of a partial memory block freeing

kmemleak_free_percpu — notify of a percpu memory block freeing

kmemleak_update_trace — update object allocation stack trace

kmemleak_not_leak — mark an object as not a leak

kmemleak_ignore — do not scan or report an object as leak

kmemleak_scan_area — add scan areas inside a memory block

kmemleak_no_scan — do not scan a memory block

kmemleak_erase — erase an old value in a pointer variable

kmemleak_alloc_recursive — as kmemleak_alloc but checks the recursiveness

kmemleak_free_recursive — as kmemleak_free but checks the recursiveness

The following functions take a physical address as the object pointer and only perform the corresponding action if the address has a lowmem mapping:

Dealing with false positives/negatives¶

The false negatives are real memory leaks (orphan objects) but not reported by kmemleak because values found during the memory scanning point to such objects. To reduce the number of false negatives, kmemleak provides the kmemleak_ignore, kmemleak_scan_area, kmemleak_no_scan and kmemleak_erase functions (see above). The task stacks also increase the amount of false negatives and their scanning is not enabled by default.

The false positives are objects wrongly reported as being memory leaks (orphan). For objects known not to be leaks, kmemleak provides the kmemleak_not_leak function. The kmemleak_ignore could also be used if the memory block is known not to contain other pointers and it will no longer be scanned.

Some of the reported leaks are only transient, especially on SMP systems, because of pointers temporarily stored in CPU registers or stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 1000) representing the minimum age of an object to be reported as a memory leak.

Limitations and Drawbacks¶

The main drawback is the reduced performance of memory allocation and freeing. To avoid other penalties, the memory scanning is only performed when the /sys/kernel/debug/kmemleak file is read. Anyway, this tool is intended for debugging purposes where the performance might not be the most important requirement.

To keep the algorithm simple, kmemleak scans for values pointing to any address inside a block’s address range. This may lead to an increased number of false negatives. However, it is likely that a real memory leak will eventually become visible.

Another source of false negatives is the data stored in non-pointer values. In a future version, kmemleak could only scan the pointer members in the allocated structures. This feature would solve many of the false negative cases described above.

The tool can report false positives. These are cases where an allocated block doesn’t need to be freed (some cases in the init_call functions), the pointer is calculated by other methods than the usual container_of macro or the pointer is stored in a location not scanned by kmemleak.

Page allocations and ioremap are not tracked.

Testing with kmemleak-test¶

To check if you have all set up to use kmemleak, you can use the kmemleak-test module, a module that deliberately leaks memory. Set CONFIG_DEBUG_KMEMLEAK_TEST as module (it can’t be used as built-in) and boot the kernel with kmemleak enabled. Load the module and perform a scan with:

Note that the you may not get results instantly or on the first scanning. When kmemleak gets results, it’ll log kmemleak: of leaks> new suspected memory leaks . Then read the file to see then:

Removing the module with rmmod kmemleak_test should also trigger some kmemleak results.

© Copyright The kernel development community.

Источник

5 useful tools to detect memory leaks with examples

Table of Contents

In this tutorial I will share different methods and tools to detect and find memory leaks with different processes in Linux. As a developer we often face scenarios when proess such as httpd apache, java starts consuming high amount of memory leading to OOM (Out Of memory) situations.

So it is always healthy to keep monitoring the memory usage of critical process. I work for an application which is memory intensive so it is my job to make sure other processes are not eating up the memory unnecessarily. In this process I use different tools in real time environments to detect memory leaks and then report it to the responsible developers.

What is Memory Leak?

• Memory is allocated on demand—using malloc() or one of its variants—and memory is freed when it’s no longer needed.
• A memory leak occurs when memory is allocated but not freed when it is no longer needed.
• Leaks can obviously be caused by a malloc() without a corresponding free() , but leaks can also be inadvertently caused if a pointer to dynamically allocated memory is deleted, lost, or overwritten.
• Buffer overruns—caused by writing past the end of a block of allocated memory—frequently corrupt memory.
• Memory leakage is by no means unique to embedded systems, but it becomes an issue partly because targets don’t have much memory in the first place and partly because they often run for long periods of time without rebooting, allowing the leaks to become a large puddle.
• Regardless of the root cause, memory management errors can have unexpected, even devastating effects on application and system behavior.
• With dwindling available memory, processes and entire systems can grind to a halt, while corrupted memory often leads to spurious crashes.

Before you go ahead I would recommend you to also read about Linux memory management so that you are familiar with the different terminologies used in Linux kernel in terms of memory.

1. Memwatch

• MEMWATCH, written by Johan Lindh, is an open-source memory error-detection tool for C.
• It can be downloaded from https://sourceforge.net/projects/memwatch
• By simply adding a header file to your code and defining MEMWATCH in your gcc command, you can track memory leaks and corruptions in a program.
• MEMWATCH supports ANSI C; provides a log of the results; and detects double frees, erroneous frees, unfreed memory, overflow and underflow, and so on.

I have downloaded and extracted memwatch on my Linux server as you can check in the screenshot:

Next before we compile the software, we must comment the below line from test.c which is a part of memwatch archive.

Next compile the software:

Next we will create a dummy C program memory.c and add the memwatch.h include on line 3 so that MEMWATCH can be enabled. Also, two compile-time flags -DMEMWATCH and -DMW_STDIO need to be added to the compile statement for each source file in the program.

The code shown in allocates two 512-byte blocks of memory (lines 10 and 11), and then the pointer to the first block is set to the second block (line 13). As a result, the address of the second block is lost, and a memory leak occurs.

Now compile the memwatch.c file, which is part of the MEMWATCH package with the sample source code ( memory1.c ). The following is a sample makefile for building memory1.c . memory1 is the executable produced by this makefile :

Next we execute the program memory1 which captures two memory-management anomalies

MEMWATCH creates a log called memwatch.log . as you can see below, which is created by running the memory1 program.

memwatch log file with memory leak information

MEMWATCH tells you which line has the problem. For a free of an already freed pointer, it identifies that condition. The same goes for unfreed memory. The section at the end of the log displays statistics, including how much memory was leaked, how much was used, and the total amount allocated.

In the above figure you can see that the memory management errors occur on line 15, which shows that there is a double free of memory. The next error is a memory leak of 512 bytes, and that memory is allocated in line 11.

2. Valgrind

• Valgrind is an Intel x86-specific tool that emulates an x86-class CPU to watch all memory accesses directly and analyze data flow
• One advantage is that you don’t have to recompile the programs and libraries that you want to check, although it works better if they have been compiled with the -g option so that they include debug symbol tables.
• It works by running the program in an emulated environment and trapping execution at various points.
• This leads to the big downside of Valgrind, which is that the program runs at a fraction of normal speed, which makes it less useful in testing anything with real-time constraints.

Valgrind can detect problems such as:

• Use of uninitialized memory
• Reading and writing memory after it has been freed
• Reading and writing from memory past the allocated size
• Reading and writing inappropriate areas on the stack
• Memory leaks
• Passing of uninitialized and/or unaddressable memory
• Mismatched use of malloc /new/new() versus free/delete/delete()

Valgrind is available in most Linux distributions so you can directly go ahead and install the tool

Next you can execute valgrind with the process for which you wish to check for memory leak. For example I wish to check memory leak for amsHelper process which is executed with -f option. Press Ctrl+C to stop monitoring

To save the output to a log file and to collect more details on the leak use —leak-check-full along with —log-file=/path/og/log/file . Press Ctrl+C to stop monitoring

Now you can check the content of /tmp/mem-leak-amsHelper.log . When it finds a problem, the Valgrind output has the following format:

3. Memleax

One of the drawbacks of Valgrind is that you cannot check memory leak of an existing process which is where memleax comes for the rescue. I have had instances where the memory leak was very sporadic for amsHelper process so at times when I see this process reserving memory I wanted to debug that specific PID instead of creating a new one for analysis.

memleax debugs memory leak of a running process by attaching it. It hooks the target process’s invocation of memory allocation and free, and reports the memory blocks which live long enough as memory leak, in real time. The default expire threshold is 10 seconds, however you should always set it by -e option according to your scenarios.

You can download memleax from the official github repository
Next memleax expects few dependencies which you must install before installing the memleax rpm

So I have manually copied these rpms from my official repository as this is a private network I could not use yum or dnf

Now since I have installed both the dependencies, I will go ahead and install memleax rpm:

Next you need the PID of the process which you wish to monitor. You can get the PID of your process from ps -ef output

Now here we wish to check the memory leak of 45256 PID

You may get output similar to above in case of a memory leak in the application process. Press Ctrl+C to stop monitoring

4. Collecting core dump

It helps for the developer at times we can share the core dump of the process which is leaking memory. In Red Hat/CentOS you can collect core dump using abrt and abrt-addon-ccpp
Before you start make sure the system is set up to generate application cores by removing the core limits:

Next install these rpms in your environment

Ensure the ccpp hooks are installed:

Ensure that this service is up and the ccpp hook to capture core dumps is enabled:

Enable the hooks

To get a list of crashes on the command line, issue the following command:

But since there are no crash the output would be empty. Next get the PID for which you wish to collect core dump, here for example I will collect for PID 45256

Next we have to send SIGABRT i.e. -6 kill signal to this PID to generate the core dump

Next you can check the list of available dumps, now you can see a new entry for this PID. This dump will contain all the information required to analyse the leak for this process

5. How to identify memory leak using default Linux tools

We discussed about third party tools which can be used to detect memory leak with more information in the code which can help the developer analyse and fix the bug. But if our requirement is just to look out for process which is reserving memory for no reason then we will have to rely on system tools such as sar , vmstat , pmap , meminfo etc

So let’s learn about using these tools to identify a possible memory leak scenario. Before you start you must be familiar with below areas

1. How to check the actual memory consumed by individual process
2. How much memory reservation is normal for your application process

If you have answers to above questions then it will be easier to analyse the problem.

For example in my case I know the memory usage of amsHelper should not be more than few MB but in case of memory leak the memory reservation goes way higher than few MB. if you have read my earlier article where I explained different tools to check actual memory usage, you would know that Pss gives us an actual idea of the memory consumed by the process.

pmap would give you more detailed output of memory consumed by individual address segments and libraries of the process as you can see below

Alternatively you can get the same information with more details using smaps of the respective process. Here I have written a small script to combine the memory and get the total but you can also remove the pipe and break down the command to get more details

So you can put a cron job or create a daemon to timely monitor the memory consumption of your application using these tools to figure out if they are consuming too much memory over time.

Conclusion

In this tutorial I shared different commands, tools and methods to detect and monitor memory leak across different types of applications such as C or C++ programs, Linux applications etc. The selection of tools would vary based on your requirement. There are many other tools such as YAMD, Electric fence, gdb core dump etc which can help you capture memory leak.

Lastly I hope the steps from the article to check and monitor memory leak on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

References

Related Posts

Didn’t find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Источник