Enable huge pages linux

G HugePages

This chapter provides an overview of Hugepages and guides Linux system administrators to configure HugePages on Linux.

G.1 Overview of HugePages

HugePages is a feature integrated into the Linux kernel 2.6. Enabling HugePages makes it possible for the operating system to support memory pages greater than the default (usually 4 KB). Using very large page sizes can improve system performance by reducing the amount of system resources required to access page table entries. HugePages is useful for both 32-bit and 64-bit configurations. HugePage sizes vary from 2 MB to 256 MB, depending on the kernel version and the hardware architecture. For Oracle Databases, using HugePages reduces the operating system maintenance of page states, and increases Translation Lookaside Buffer (TLB) hit ratio.

Transparent Hugepages is currently not an alternative to manually configure HugePages.

This section includes the following topics:

G.1.1 Tuning SGA With HugePages

Without HugePages, the operating system keeps each 4 KB of memory as a page. When it allocates pages to the database System Global Area (SGA), the operating system kernel must continually update its page table with the page lifecycle (dirty, free, mapped to a process, and so on) for each 4 KB page allocated to the SGA.

With HugePages, the operating system page table (virtual memory to physical memory mapping) is smaller, because each page table entry is pointing to pages from 2 MB to 256 MB.

Also, the kernel has fewer pages whose lifecycle must be monitored. For example, if you use HugePages with 64-bit hardware, and you want to map 256 MB of memory, you may need one page table entry (PTE). If you do not use HugePages, and you want to map 256 MB of memory, then you must have 256 MB * 1024 KB/4 KB = 65536 PTEs.

HugePages provides the following advantages:

Increased performance through increased TLB hits

Pages are locked in memory and never swapped out, which provides RAM for shared memory structures such as SGA

Contiguous pages are preallocated and cannot be used for anything else but for System V shared memory (for example, SGA)

Less bookkeeping work for the kernel for that part of virtual memory because of larger page sizes

G.1.2 Configuring HugePages on Linux

Complete the following steps to configure HugePages on the computer:

Run the following command to determine if the kernel supports HugePages:

Some Linux systems do not support HugePages by default. For such systems, build the Linux kernel using the CONFIG_HUGETLBFS and CONFIG_HUGETLB_PAGE configuration options. CONFIG_HUGETLBFS is located under File Systems and CONFIG_HUGETLB_PAGE is selected when you select CONFIG_HUGETLBFS .

Edit the memlock setting in the /etc/security/limits.conf file. The memlock setting is specified in KB, and the maximum locked memory limit should be set to at least 90 percent of the current RAM when HugePages memory is enabled and at least 3145728 KB (3 GB) when HugePages memory is disabled. For example, if you have 64 GB RAM installed, then add the following entries to increase the maximum locked-in-memory address space:

You can also set the memlock value higher than your SGA requirements.

Log in as oracle user again and run the ulimit -l command to verify the new memlock setting:

Run the following command to display the value of Hugepagesize variable:

Complete the following procedure to create a script that computes recommended values for hugepages configuration for the current shared memory segments:

Create a text file named hugepages_settings.sh .

Add the following content in the file:

Run the following command to change the permission of the file:

Run the hugepages_settings.sh script to compute the values for hugepages configuration:

Before running this script, ensure that all the applications that use hugepages run.

Set the following kernel parameter, where value is the HugePages value that you determined in step 7:

To ensure that HugePages is allocated after system restarts, add the following entry to the /etc/sysctl.conf file, where value is the HugePages value that you determined in step 7:

Run the following command to check the available hugepages :

Restart the instance.

Run the following command to check the available hugepages (1 or 2 pages free):

If you cannot set your HugePages allocation using nr_hugepages , then your available memory may be fragmented. Restart your server for the Hugepages allocation to take effect.

G.1.3 Restrictions for HugePages Configurations

HugePages has the following limitations:

You must unset both the MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters. For example, to unset the parameters for the database instance, use the command ALTER SYSTEM RESET .

Automatic Memory Management (AMM) and HugePages are not compatible. When you use AMM, the entire SGA memory is allocated by creating files under /dev/shm . When Oracle Database allocates SGA with AMM, HugePages are not reserved. To use HugePages on Oracle Database 12 c , You must disable AMM.

If you are using VLM in a 32-bit environment, then you cannot use HugePages for the Database Buffer cache. You can use HugePages for other parts of the SGA, such as shared_pool , large_pool , and so on. Memory allocation for VLM (buffer cache) is done using shared memory file systems ( ramfs/tmpfs/shmfs ). Memory file systems do not reserve or use HugePages.

HugePages are not subject to allocation or release after system startup, unless a system administrator changes the HugePages configuration, either by modifying the number of pages available, or by modifying the pool size. If the space required is not reserved in memory during system startup, then HugePages allocation fails.

Ensure that HugePages is configured properly as the system may run out of memory if excess HugePages is not used by the application.

If there is insufficient HugePages when an instance starts and the initialization parameter use_large_pages is set to only , then the database fails to start and an alert log message provides the necessary information on Hugepages.

G.1.4 Disabling Transparent HugePages

Transparent HugePages memory is enabled by default with Red Hat Enterprise Linux 6, SUSE 11, and Oracle Linux 6 with earlier releases of Oracle Linux Unbreakable Enterprise Kernel 2 (UEK2) kernels. Transparent HugePages memory is disabled by default in later releases of UEK2 kernels.

Transparent HugePages can cause memory allocation delays at runtime. To avoid performance issues, Oracle recommends that you disable Transparent HugePages on all Oracle Database servers. Oracle recommends that you instead use standard HugePages for enhanced performance.

Transparent HugePages memory differs from standard HugePages memory because the kernel khugepaged thread allocates memory dynamically during runtime. Standard HugePages memory is pre-allocated at startup, and does not change during runtime.

To check if Transparent HugePages is enabled run one of the following commands as the root user:

Red Hat Enterprise Linux kernels:

The following is a sample output that shows Transparent HugePages is being used as the [always] flag is enabled.

If Transparent HugePages is removed from the kernel then the /sys/kernel/mm/transparent_hugepage or /sys/kernel/mm/redhat_transparent_hugepage files do not exist.

To disable Transparent HugePages perform the following steps:

Источник

• Hugepages

When a process uses some memory, the CPU is marking the RAM as used by that process. For efficiency, the CPU allocate RAM by chunks of 4K bytes (it’s the default value on many platforms). Those chunks are named pages. Those pages can be swapped to disk, etc.

Since the process address space are virtual, the CPU and the operating system have to remember which page belong to which process, and where it is stored. Obviously, the more pages you have, the more time it takes to find where the memory is mapped. When a process uses 1GB of memory, that’s 262144 entries to look up (1GB / 4K). If one Page Table Entry consume 8bytes, that’s 2MB (262144 * 8) to look-up.

Most current CPU architectures support bigger pages (so the CPU/OS have less entries to look-up), those are named Huge pages (on Linux), Super Pages (on BSD) or Large Pages (on Windows), but it all the same thing.

Read the documentation for more information about hugetlbpage.

Enabling HugeTlbPage

Currently, there is no standard way to enable HugeTLBfs, mainly because the FHS has no provision for such kind of virtual file system, see 572733 . (Fedora mounts it in /dev/hugepages/, so don’t be surprised if you find some example on the web that use this location)

Linux support «Huge page tables» (HugeTlb) is available in Debian since DebianLenny (actually, since 2.6.23). A good introduction to large pages is available from ibm.com.

Create a group for users of hugepages, and retrieve it’s GID (is this example, 2021) then add yourself to the group.
Note: this should not be needed for libvirt (see /etc/libvirt/qemu.conf)

Edit /etc/sysctl.conf and add this text to specify the number of pages you want to reserve (see pages-size)

Create a mount point for the file system

Add this line in /etc/fstab (The mode of 1770 allows anyone in the group to create files but not unlink or rename each other’s files. 1 )

Reboot (This is the most reliable method of allocating huge pages before the memory gets fragmented. You don’t necessarily have to reboot. You can try to run sysctl -p to apply the changes. if grep "Huge" /proc/meminfo don’t show all the pages, you can try to free the cache with sync ; echo 3 > /proc/sys/vm/drop_caches (where «3» stands for «purge pagecache, dentries and inodes») then try sysctl -p again. 2 )

limits.conf

You should configure the amount of memory a user can lock, so an application can’t crash your operating system by locking all the memory. Note that any page can be locked in RAM, not just huge pages. You should allow the process to lock a little bit more memory that just the the space for hugepages.

See Limits (ulimit -l and memlock in /etc/security/limits.conf).

Multiple huge page size support

Some architectures (like ia64) can have multiple and/or configuration «huge» pages size.

boot parameters and mount options in hugetlbpage.txt in documentations.

arm64

The Debian arm64 kernel (running with a 4KB standard PAGE_SIZE) supports 2MB and 1GB HugeTLB page sizes. One has to pre-allocate 1GB HugeTLB pages on boot by specifying arguments on the kernel command line, the following will pre-allocate 10 x 1GB huge pages:

If one elects to build their own Debian arm64 kernel with CONFIG_ARM64_64K_PAGES=y, then only 512MB HugeTLB (and THP) pages are available. These are available at run time.

x86_64

Depending on the processor, there are at least two different huge page sizes on the x86_64 architecture: 2MB and 1GB. If the CPU supports 2MB pages, it has the PSE cpuinfo flag, for 1GB it has the PDPE1GB flag. /proc/cpuinfo shows whether the two flags are set.

If this commands returns a non-empty string, 2MB pages are supported.

If this commands returns a non-empty string, 1GB pages are supported.

Before they are actually both available, they may have to be activated at boot time. The following kernel boot parameters enable 1GB pages and create a pool of one 1GB page:

After boot, the huge page pools look like this:

Getting informations

You can get a list of available huge page sizes with hugeadm:

hugeadm also displays the number of allocated huge pages per available size:

An alternative to retrieve the current available/used page for the default huge page size is /proc/meminfo:

(read Documentation/vm/hugetlbpage.txt for more information about it)

Standard Debian Kernel have HUGETLB enabled (What about Lenny ? Xen ?):

Various runtime settings (see documentation)

Huge pages sizes

Tools

libhugetlbfs is a library which provides easy access to huge pages of memory. The library also comes with several userspace tools to help with huge page usability, environment setup, and control.
http://libhugetlbfs.ozlabs.org/
533708 — ITP: libhugetlbfs — Initial package request
http://www.ibm.com/developerworks/wikis/display/LinuxP/libhuge+short+and+simple

Hugepage enabled applications

An application can allocate/use HugeTlbPage through two different means:

mmap system call require a mounted hugetlbfs, with appropriate permissions.

Shared memory segment (shmat/shmget system calls or mmap with MAP_HUGETLB), must be member of a group, configured in /proc/sys/vm/hugetlb_shm_group.

MySQL

Java (Sun, OpenJDK)

Sun and OpenJDK can use large pages.

sun-java6-bin — Sun Java(TM) Runtime Environment (JRE) 6 (architecture dependent files)

openjdk-6-jre-headless — OpenJDK Java runtime, using Hotspot JIT (headless)

Basically, it seems that one have to launch java with -XX:+UseLargePages. The maximum size of the Java heap (-Xmx) should fit in your reserved Huge pages ; same for ulimit -l and/or memlock in /etc/security/limits.conf.

Not in group defined in /proc/sys/vm/hugetlb_shm_group

Memcached

memcached can use huge pages, read the manpage :

Try to use large memory pages (if available). Increasing the memory page size could reduce the number of TLB misses and improve the performance. In order to get large pages from the OS, memcached will allocate the total item-cache in one large chunk. Only available if supported on your OS.

PosgreSQL

PostgreSQL gains huge page support in version 9.4. For previous versions, see hugetlblib above.

Virtualisation

Some consideration about hugepages an virtualisation.

1. Before enabling hugepages in a virtual machine, you should make sure the that your virtualization tool can handle it.
2. Whether a virtualization tools supports hugepages for it’s client or for itself are probably two different aspects.

It is unclear which version of Xen supports huge pages, and how it is used.

Источник

Configuring HugePages for Oracle on Linux (x86-64)

Introduction

For large SGA sizes, HugePages can give substantial benefits in virtual memory management. Without HugePages, the memory of the SGA is divided into 4K pages, which have to be managed by the Linux kernel. Using HugePages, the page size is increased to 2MB (configurable to 1G if supported by the hardware), thereby reducing the total number of pages to be managed by the kernel and therefore reducing the amount of memory required to hold the page table in memory. In addition to these changes, the memory associated with HugePages can not be swapped out, which forces the SGA to stay memory resident. The savings in memory and the effort of page management make HugePages pretty much mandatory for Oracle 11g systems running on x86-64 architectures.

Just because you have a large SGA, it doesn’t automatically mean you will have a problem if you don’t use HugePages. It is typically the combination of a large SGA and lots database connections that leads to problems. To determine how much memory you are currently using to support the page table, run the following command at a time when the server is under normal/heavy load.

Automatic Memory Management (AMM) is not compatible with Linux HugePages, so apart from ASM instances and small unimportant databases, you will probably have no need for AMM on a real database running on Linux. Instead, Automatic Shared Memory Management and Automatic PGA Management should be used as they are compatible with HugePages.

Configuring HugePages

Run the following command to determine the current HugePage usage. The default HugePage size is 2MB on Oracle Linux 5.x and as you can see from the output below, by default no HugePages are defined.

Depending on the size of your SGA, you may wish to increase the value of Hugepagesize to 1G.

Create a file called «hugepages_setting.sh» with the following contents.

Thanks to Bjoern Rost for pointing out the issue when using the script against UEK3 and the suggested fix. I’ve subsequently added support for 3.10 and 4.1. There is a newer version of this script available from MOS (Doc ID 401749.1) which includes these kernel versions also.

Make the file executable.

Make sure all the Oracle services are running as normal on the server, then run the script and make a note of the recommended «vm.nr_hugepages» value.

Edit the «/etc/sysctl.conf» file as the «root» user, adding the following entry, adjusted based on your output from the script. You should set the value greater than or equal to the value displayed by the script. You only need 1 or 2 spare pages.

One person reported also needing the hugetlb_shm_group setting on Oracle Linux 6.5. I did not and it is listed as a requirement for SUSE only. If you want to set it, get the ID of the dba group.

Use the resulting group ID in the «/etc/sysctl.conf» file.

Run the following command as the «root» user.

Alternatively, edit the «/etc/grub.conf» file, adding «hugepages=306» to the end of the kernel line for the default kernel and reboot.

You can now see the HugePages have been created, but are currently not being used.

Add the following entries into the «/etc/security/limits.conf» script or «/etc/security/limits.d/99-grid-oracle-limits.conf» script, where the setting is at least the size of the HugePages allocation in KB (HugePages * Hugepagesize). In this case the value is 306*2048=626688.

If you prefer, you can set these parameters to a value just below the size of physical memory of the server. This way you can forget about it, unless you add more physical memory.

Check the MEMORY_TARGET parameters are not set for the database and SGA_TARGET and PGA_AGGREGATE_TARGET parameters are being used instead.

Restart the server and restart the database services as required.

Check the HugePages information again.

You can see the HugePages are now being used.

Remember, if you increase your memory allocation or add new instances, you need to retest the required number of HugePages, or risk Oracle running without them.

Force Oracle to use HugePages (USE_LARGE_PAGES)

Sizing the number of HugePages correctly is important because prior to 11.2.0.3, if the whole SGA doesn’t fit into the available HugePages, the instance will start up without using any. From 11.2.0.3 onward, the SGA can run partly in HugePages and partly not, so the impact of this issue is not so great. Incorrect sizing may not be obvious to spot. Later releases of the database display a «Large Pages Information» section in the alert log during startup.

If you are running Oracle 11.2.0.2 or later, you can set the USE_LARGE_PAGES initialization parameter to «only» so the database fails to start if it is not backed by hugepages. You can read more about this here.

On startup the «Large Page Information» in the alert log reflects the use of this parameter.

Attempting to start the database when there aren’t enough HugePages to hold the SGA will now return the following error.

The «Large Pages Information» section of the alert log output describes the startup failure and the appropriate action to take.

Disabling Transparent HugePages (RHEL6/OL6 and RHEL7/OL7)

Starting from RHEL6/OL6, Transparent HugePages are implemented and enabled by default. They are meant to improve memory management by allowing HugePages to be allocated dynamically by the «khugepaged» kernel thread, rather than at boot time like conventional HugePages. That sounds like a good idea, but unfortunately Transparent HugePages don’t play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note.

The following examples use the base path of «/sys/kernel/mm/transparent_hugepage/» which is used by OL6/OL7. For RHEL6/RHEL7 use «/sys/kernel/mm/redhat_transparent_hugepage/» as the base path.

You can check the current setting using the following command, which is displaying the default value of «enabled=[always]».

For Oracle Linux 6 the preferred method to disable Transparent HugePages is to add «transparent_hugepage=never» to the kernel boot line in the «/boot/grub/grub.conf» file.

Oracle Linux 7 is similar, but uses GRUB2 so you need to edit the «/boot/grub2/grub.cfg» file using the grubby command.

The server must be rebooted for this to take effect.

Alternatively, add the following lines into the «/etc/rc.local» file and reboot the server.

Whichever method you choose, remember to check the change has work after reboot.

In OL7/RHEL7 you also need to consider the «tuned profile». The following script shows how to create and enable an amended version of the currently active tuned profile.

Thanks to Mor for pointing this out and directing me to the notes here and here.

With Transparent HugePages disabled, you should proceed to configure conventional HugePages, as described above.

Configuring 1G Hugepagesize

As mentioned by Eugene in the comments, Oracle currently don’t recommend using 1G Hugepagesize. You can read more about this in MOS Doc ID 1607545.1. With that in mind, the rest of this section should probably be considered more of an academic exercise.

Check if your current hardware can support a Hugepagesize of 1G. If the following command produces any output, it can.

Thanks to Kevin Closson for pointing out the hardware support requirement.

Edit the «/etc/grub.conf» file, adding the following entries on to the kernel line of the default grub entry. Adjust the «hugepages» entry to the desired number of 1G pages. Notice this includes the disabling of Transparent HugePages, which is not mandatory, but a good idea.

Check the current HugePages setup.

Reboot and check the HugePages setup again.

Источник