Linux what is using memory cache

Tutorial: Beginners guide on linux memory management

Table of Contents

Linux memory management is a very vast topic and it is not possible to cover all the areas in single article. I will try to give you an overview on major areas and will help you understand important terminologies related to memory management in Linux.

Overview on Linux Memory Management

• The central part of the computer is CPU and RAM is the front end portal to CPU
• Everything that is going to CPU will go through RAM
• For example, if we have a process which is loading, the process will first be loading in RAM and the CPU will get process data from RAM
• But to make it faster, the CPU has level one, level two, level three cache. That is like RAM, but on the CPU
• There’s very small amounts of cache on the CPU, because it’s very expensive, and it’s also not very useful for all the instructions.
• So the process information will be copied from RAM to CPU and the CPU will build its cache.
• Here cache plays an important part of memory management on Linux as well.

• If a user is requesting information from hard disk, it is copied to RAM, and the user will be served from RAM.
• And while the information is copied from hard disk, it is placed in what you call the page cache.
• So the page cache stores recently requested data to make it faster if the same data is needed again.
• And if the user starts modifying data, it will go to RAM as well, and from RAM it will be copied to the hard disk, but that will only happen if the data has been sitting in RAM long enough.
• Data won’t be written immediately from RAM to hard disk, but to optimize write to the hard disk, Linux works with the concept of dirty cache.
• It tries to buffer as much as data in order to create an efficient write request.
• So as you can see in this small diagram, everything that happens on your computer goes through RAM.
• So using RAM on a Linux computer is essential for the well working of the Linux operating system.

Now we will discuss the individual part of Linux memory management and understand different terminologies related to this flow.

Understanding Virtual Memory

When analysing Linux memory usage, you should know how Linux uses Virtual and Resident Memory. Virtual Memory on Linux is to be taken literally: it is a non-existing amount of memory that the Linux kernel can be referred to.

Currently my RHEL 7 Linux has 128GB of Total Physical Memory

But as you see this node also has 32TB of Virtual Memory

• Now the idea is that if in Linux, when a process is loading, this process needs memory pointers, and these memory pointers don’t really have to refer to actual physical RAM, and that is why we are using virtual memory.
• Virtual Memory is used by the Linux kernel to allow programs to make a memory reservation.
• After making this reservation, no other application can reserve the same memory.
• Making the reservation is a matter of setting pointers and nothing else. It doesn’t mean that the memory reservation is also actually going to be used.
• When a program has to use the memory it has reserved, it is going to issue a malloc system call, which means that the memory is actually going to be allocated. At that moment, we’re talking about resident memory.

What is memory over-allocation (over-commitment) and OOM?

• In top command we can see the virt column and the res column.
• The VIRT is for Virtual Memory. That’s the amount of kilobytes that currently the process has allocated.
• And RES is for Resident, that is what it is really using.
• I have sorted the output to show processes with most memory consumption.

• We can see in the VIRT section that huge amount of memory is allocated to the process such as for java 22.4GB of virtual memory is allocated while only 4.8GB is used
• If you add all of these VIRT memory to one another, you are getting far beyond the total of 128GB of physical RAM that is available in this system.
• That is what we call memory over-allocation.
• To tune the behavior of over committing memory, you can write to the /proc/sys/vm/overcommit_memory parameter. This parameter can have some values.
• The default value is 0, which means that the kernel checks if it still has memory available before granting it. If that doesn’t give you the performance you need,
• The value 1, means that the system thinks there is enough memory in all cases. This is good for performance of memory-intensive tasks but may result in processes getting killed automatically.
• The value of 2, means that the kernel fails the memory request if there is not enough memory available.

Let us understand in laymen terms,

• This means that when any application or process requests for memory, kernel will always honour that request and «give«.
• Please NOTE, that kernel gives certain amount of memory but this memory is not marked as «used«.
• Instead this is considered as «virtual memory» and only when the application or process tries to write some data into the memory, kernel will mark that section of memory as «used«.
• So if suddenly one or some process would start using more resident memory (RES), the kernel needs to honour that request, and the result is that you can reach a situation where there is no more memory available.
• That is called OOM (Out Of Memory) situation when your system is running low on memory, and it is getting out of memory
• In such case the out of memory killer becomes active, and it will kill the process that least requires to have the memory, and that is kind of a random situation, so you’ll get a random process killed if your system is running out of memory.

Источник

How to Clear Memory Cache in Linux

Sometimes the system goes out of memory due to huge RAM is used by cached objects. In that cases, either you need to increase physical memory in the system or add more swap space. You can also instruct kernel to clear RAM memory cache on system by adding a number in /proc/sys/vm/drop_caches file.

It is safe but not recommended to clear the memory cache on a Linux system. Clearing the Memory cache in Linux systems slows down the system performance as reading files from memory is much faster than persistent disk. Since it discards cached objects from memory, it may cost a significant amount of I/O and CPU to recreate the dropped objects. This tutorial will help you to clear the memory cache on Linux/Unix system via the command line.

How to Clear Memory Cache on Linux

There are three options available to clear the memory cache in Linux. Choose one of the below options to flush the Linux system cache memory as per your requirements.

• Clear PageCache, dentries and inodes in cache memory. In short it will clear all the memory cache:
• Clear dentries and inodes only in cache memory
• Clear page cache only in cache memory

Here the first command sync is used to synchronize all the in-memory cache files to the persistent storage. The next command is separated with a “;”. Once the first command is completed, the next command will be triggered to clear cache memory.

Scheduleng the Clear Memory Cache with Crontab

You can also schedule a corn job to clear the cache on a regular basis. Schedule the following in system crontab to automatically flush cache memory at a regular interval.

Open a terminal and execute ‘crontab -e’ command to edit crontab:

Append below entry to the file:

The above cron will execute on every hour and flushes the memory cache on your system.

On the production servers, it is not recommended to schedule a clear cache command. It can lead to data corruption or data loss. So beware before running the above command in a production environment.

How to find Cached Memory in Linux

Use free command to find out cache memory uses by Linux system. The output of the free command is like below

Here the last column is showing cached memory (12953 MB) on Linux system. The -m option is used to show output MB’s.

Источник

Understanding memory information on Linux systems

Every operating system needs memory to store program code segments and data. This is also true for Linux systems. The problem: there is a lot of information available regarding memory usage and its behavior. Let’s discover how Linux manages its memory and how we can gather memory information.

After reading this guide, you will be able to:

• Show the total amount of memory
• Display all memory details
• Understand the details listed in /proc/meminfo
• Use tools like dmesg, dmidecode, free, and vmstat

Table of Contents

Linux memory information

Random access memory

When we talk about memory in this article, we usually mean random access memory (RAM). This is the memory which can be used for both showing and storing data. Typically we will find in this type of memory the programs that are running on the system, including the Linux kernel itself. Besides the program code, memory also stores a lot of data. A good example is when you are running a MySQL database server. The program itself is relatively small, the data itself is huge. So we will also have a look at tuning programs and their memory usage, as this is typically a problem with memory-hungry programs.

Determine the amount of RAM

The first step is to discover the amount of RAM we have in the system. There are a few ways on how to achieve this, starting from the data stored in dmesg .

The output may look something like this:

This information shows the number of memory available in kilobytes. The first value shows what is currently available, the second value displays the total memory in the system. These values are usually very close. This indicates that most of the memory can be used and is a good thing. The small portion “missing” is used by the initial loading of the kernel. If there is a big gap, then this might be caused by the kernel and how many memory it can allocate. Especially with 32 bits versions of Linux, this number is limited.

Details and information about RAM modules

The next step is learning more about the RAM modules itself. You will need the dmidecode utility for this, which is available for most Linux distributions. To gather memory information, tell the dmidecode to only show information for device type 17.

Depending on your hardware it may be able to extract the specifics of your modules and show detailed information. You may need to run this as root user. Normal users won’t have the right permissions to read all information.

In this output above you can see the details of the first memory module. We see it is a chip of 4 GB and configured at a 1600 MHz speed. This is a great way to determine the memory available in a Linux system, together with detailed output. Unfortunately, the command does not always play well with virtual systems.

No data is displayed on our virtual test system 🙁

Let’s move on to the next set of utilities and gather details regarding memory usage.

Available memory

After the Linux kernel is booted, it is time to start programs. The kernel itself is not responsible for the programs. Instead, it delegates this responsibility to a service manager like init or systemd. This process is the first to be started and will get process ID 1. Its duty is to start other services and programs during the lifetime of the system. Each program will consume some amount of memory, depending on the program size and the related data. Let’s have a look at some ways to see available memory on Linux and retrieving related details.

See available memory with free command

The first command to obtain available memory information is the perfectly named tool free .

This utility shows two different types of memory: normal memory and swap memory. Swap is a type of memory that you want to avoid needing as much as possible. If it would be used, then it means your normal memory is full. The system will then leverage the swap memory to temporarily store data, at the cost of disk operations. As they are much slower than normal RAM, your system will be impacted. In this screenshot, we see the swap is not used, which is good.

The free utility retrieves this memory information from a file named /proc/meminfo. Let’s have a look at that as well.

Details from /proc/meminfo

The next step to obtain everything available regarding memory is found in the procfs tree, usually mounted under /proc. This file is very extensive, so have a look at it on your system:

A partial output listing showing how memory is used

Memory management under Linux is extensive and changed over time to what it is now. This results in a delicate system that optimizes memory usage as much as possible. Let’s get into some of these fields and understand better how Linux does its job.

Cached, SwapCached

The system does a lot of repetition, including reading the same files or data. Everything that goes into memory and is no longer needed, will be kept for a little bit longer. If you then request the same data while it is in memory, you will get it almost instantly. This is what happens when you run a find command on a particular directory the first time, which usually takes a while. Run it again and it will be much quicker.

Active, Inactive

A page cache optimizes access to files. These buffers can be recently used (=active), or not (=inactive).

Active is the total of Active(anon) and Active(file). Similarly, Inactive is the total of Inactive(anon) + Inactive(file).

SwapTotal, SwapFree

These provide insights in the configured swap memory and how much is left. Ideally, the SwapFree value is equal to SwapTotal, meaning no swap is in use at that time. Swapping is disk intensive.

Dirty

The Dirty field refers to data that is stored in memory and still needs to be written to the disk.

You can test this easily by writing to a temporary file and compare the value before and after.

cat /proc/meminfo | grep Dirty && dd if=/dev/zero of=/tmp/testfile.txt bs=1M count=10 && cat /proc/meminfo | grep Dirty

Note: if you repeat this command, you will see the effect of smart memory management. In that case, the value before and after will most likely be the same, as some data was cached and directly returned as a finished action. You can counter this by retrieving random data from /dev/random.

Slab, SReclaimable, SUnreclaim

The kernel does a lot of repetition during the time it is running. Some objects, like asking for the specific inode of a file may be performed thousand times a day. In such case, it would be wise to store it in a quick reference list, or cache. Slab is the combination of caches for kernel objects, to optimize those activities that happen the most.

The Slab field is the total of SReclaimable and SUnreclaim.

Slab: 32272 kB
SReclaimable: 18144 kB
SUnreclaim: 14128 kB

NFS_Unstable

For systems that use NFS this is a good measurement to see how much data is not committed to the storage yet. For systems without NFS, this value can be ignored and is usually just zero.

More fields

If you compared these fields with your own system, you will discover there are more fields. Depending on your workload, you will have to discover what fields make sense to monitor. What does generally work well during troubleshooting, is comparing similar systems and check for the differences in /proc/meminfo. It may give a good indication of where memory is used and what keeps the system busy.

Using vmstat utility

Another nice utility that is often available is vmstat. With -s we can query memory statistics.

We can also query the previous mentioned slabs.

Monitoring memory usage in Linux

If you have a monitoring system in place, then two key attributes from /proc/meminfo should be monitored.

By monitoring these two values you may discover memory leaks and badly optimized systems. You may also pick up on a misbehaving process now and then.

For environments that have many similar systems with a typical workload (e.g. lots of web servers doing the same), then it would make sense to monitor more of the keys in /proc/meminfo. Storing the data a few times per day may give you the ability to compare systems and find exceptional events.

Frequently Asked Questions

How can I find out the total physical memory (RAM)?

Use the free command to show the total amount of memory and how it is assigned.

What Linux tool can I use to see the details of the memory modules?

Use the hwinfo tool to gather details regarding the memory. If that is not available, then consider using the output from dmesg.

How can I see which processes consume the most memory?

Use the ps command to show memory usage and do a reverse sort.

Why are the buffer and cache use so much memory on Linux?

Linux considers unused memory to be wasted memory. So it will use as much memory as possible to speed up the performance on the system. The related caches and buffers contain typically data related to the file system. That is also why a second run of the find command in the same directory runs much quicker. The kernel will reassign memory to processes when needed.

Where does the kernel store the details regarding virtual memory management?

Most of the related settings, like how to act during an Out-of-Memory event, will be stored in /proc/sys/vm.

Conclusion

Linux memory management is an extensive subject and there is a lot to learn. Make sure to understand the basics, like how to obtain memory information, including that of RAM and swap. This is of great help during troubleshooting and to know what programs need to do their job.

Did you learn something from this article? Great! Share it on your favorite website or with others. If you have a nice tool for memory analysis or got a question, you are welcome to use the comments!

Keep learning

So you are interested in Linux security? Join the Linux Security Expert training program, a practical and lab-based training ground. For those who want to become (or stay) a Linux security expert.

Run automated security scans and increase your defenses. Lynis is an open source security tool to perform in-depth audits. It helps with system hardening, vulnerability discovery, and compliance.

Источник