Linux audit rate limit exceeded

Why do I see «audit: backlog limit exceeded» errors in my EC2 Linux instance’s screenshot and system logs, and what can I do to avoid this?

Last updated: 2020-09-18

I see «audit callbacks suppressed» and «audit: backlog limit exceeded» error messages in my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance’s screenshot and system logs. Why am I receiving these messages, and how can I prevent them from reoccurring?

Short description

The audit backlog buffer in a Linux system is a kernel level socket buffer queue that the operating system uses to maintain or log audit events. When a new audit event triggers, the system logs the event and adds it to the audit backlog buffer queue.

The backlog_limit parameter value is the number of audit backlog buffers. The parameter is set to 320 by default, as shown in the following example:

Audit events logged beyond the default number of 320 cause the following errors on the instance:

An audit buffer queue at or exceeding capacity might also cause the instance to hang or remain in an unresponsive state.

To avoid backlog limit exceeded errors, increase the backlog_limit parameter value. Large servers have a larger number of audit logs triggered, so increasing buffer space helps avoid error messages.

Note: Increasing the audit buffer consumes more of the instance’s memory. How large you make the backlog_limit parameter depends on the total memory of the instance. If the system has enough memory, you can try doubling the existing backlog_limit parameter value.

The following is a calculation of the memory required for the auditd backlog. Use this calculation to determine how large you can make the backlog queue without causing memory stress on your instance.

One audit buffer = 8970 Bytes
Default number of audit buffers (backlog_limit parameter) = 320
320 * 8970 = 2870400 Bytes, or 2.7 MiB

The size of the audit buffer is defined by the MAX_AUDIT_MESSAGE_LENGTH parameter. For more information, see MAX_AUDIT_MESSAGE_LENGTH in the Linux audit library on github.com.

Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, stop and start the instance. Then, perform the following steps to change the audit buffer value.

Resolution

Note: In this example, we’re changing the backlog_limit parameter value to 8192 buffers. 8192 buffers equals 70 MiB of memory based on the preceding calculation. You can use any value based on your memory calculation.

1. Access the instance using SSH.

2. Verify the current audit buffer size.

Note: The backlog_limit parameter is listed as -b. For more information, see auditctl(8) on the auditctl-man-page

Amazon Linux 1 and other operating systems that don’t have systemd:

Amazon Linux 2 and other operating systems that use systemd:

3. Access the audit.rules file using an editor, such as the vi editor:

Amazon Linux 1 and other operating systems that don’t use systemd:

Amazon Linux 2 and other operating systems that use systemd:

4. Edit the -b parameter to a larger value. The following example changes the -b value to 8192.

5. Restart the auditd service. The new backlog_limit value takes effect. The value also updates in auditctl -s, as shown in the following example:

Note: If your instance is inaccessible and you see backlog limit exceeded messages in the system log, stop and start the instance. Then, perform the preceding steps to change the audit buffer value.

Did this article help?

Do you need billing or technical support?

Источник

CentOS

The Community ENTerprise Operating System

kernel: audit: printk limit exceeded .

kernel: audit: printk limit exceeded .

Post by _ck_ » 2015/06/06 11:53:17

What does this mean in /var/log/messages and does it need to be fixed?

kernel: audit: printk limit exceeded

Also, is this something new with systemd in the latest centos7, I seem to get audit messages every time cron does something every minute.

Can I quiet it down to show only when there is a problem, or does the fact it shows messages indicate a problem — doesn’t look like it?

Re: kernel: audit: printk limit exceeded .

Post by TrevorH » 2015/06/06 12:06:27

Re: kernel: audit: printk limit exceeded .

Post by _ck_ » 2015/06/06 12:14:25

Should I have auditd running? Is there a performance hit?

Can I knock down the verbosity to errors/warnings instead of every activity?

What’s strange is I don’t remember seeing all these constant messages in /var/log/messages until one of the more recent centos7 updates.

Did something change or I did I accidentally turnoff auditd when it was running before?

Aha, I see a /var/log/audit/ directory and it has several rotated audit.log

There seems to be one each day except not today.

Maybe I accidentally killed auditd somehow. Nope it is running.

Источник

Linux audit rate limit exceeded

The Linux audit framework as shipped with this version of SUSE Linux Enterprise Server provides a CAPP-compliant (Controlled Access Protection Profiles) auditing system that reliably collects information about any security-relevant event. The audit records can be examined to determine whether any violation of the security policies has been committed, and by whom.

Providing an audit framework is an important requirement for a CC-CAPP/EAL (Common Criteria-Controlled Access Protection Profiles/Evaluation Assurance Level) certification. Common Criteria (CC) for Information Technology Security Information is an international standard for independent security evaluations. Common Criteria helps customers judge the security level of any IT product they intend to deploy in mission-critical setups.

Common Criteria security evaluations have two sets of evaluation requirements, functional and assurance requirements. Functional requirements describe the security attributes of the product under evaluation and are summarized under the Controlled Access Protection Profiles (CAPP). Assurance requirements are summarized under the Evaluation Assurance Level (EAL). EAL describes any activities that must take place for the evaluators to be confident that security attributes are present, effective, and implemented. Examples for activities of this kind include documenting the developers’ search for security vulnerabilities, the patch process, and testing.

This guide provides a basic understanding of how audit works and how it can be set up. For more information about Common Criteria itself, refer to the Common Criteria Web site.

Linux audit helps make your system more secure by providing you with a means to analyze what is happening on your system in great detail. It does not, however, provide additional security itself—it does not protect your system from code malfunctions or any kind of exploits. Instead, audit is useful for tracking these issues and helps you take additional security measures, like AppArmor , to prevent them.

Audit consists of several components, each contributing crucial functionality to the overall framework. The audit kernel module intercepts the system calls and records the relevant events. The auditd daemon writes the audit reports to disk. Various command line utilities take care of displaying, querying, and archiving the audit trail.

Audit enables you to do the following:

Audit maps processes to the user ID that started them. This makes it possible for the administrator or security officer to exactly trace which user owns which process and is potentially doing malicious operations on the system.

Important: Renaming User IDs

Audit does not handle the renaming of UIDs. Therefore avoid renaming UIDs (for example, changing tux from uid=1001 to uid=2000 ) and obsolete UIDs rather than renaming them. Otherwise you would need to change auditctl data (audit rules) and would have problems retrieving old data correctly.

Linux audit provides tools that write the audit reports to disk and translate them into human readable format.

Review Particular Audit Events

Audit provides a utility that allows you to filter the audit reports for certain events of interest. You can filter for:

Remote Host Name

Remote Host Address

System Call Arguments

Success or Failure

Audit provides the means to filter the audit reports for events of interest and to tune audit to record only selected events. You can create your own set of rules and have the audit daemon record only those of interest to you.

Guarantee the Availability of the Report Data

Audit reports are owned by root and therefore only removable by root . Unauthorized users cannot remove the audit logs.

Prevent Audit Data Loss

If the kernel runs out of memory, the audit daemon’s backlog is exceeded, or its rate limit is exceeded, audit can trigger a shutdown of the system to keep events from escaping audit’s control. This shutdown would be an immediate halt of the system triggered by the audit kernel component without synchronizing the latest logs to disk. The default configuration is to log a warning to syslog rather than to halt the system.

If the system runs out of disk space when logging, the audit system can be configured to perform clean shutdown. The default configuration tells the audit daemon to stop logging when it runs out of disk space.

31.1 Introducing the Components of Linux Audit #Edit source

The following figure illustrates how the various components of audit interact with each other:

FigureВ 31.1: Introducing the Components of Linux Audit #

Straight arrows represent the data flow between components while dashed arrows represent lines of control between components.

The audit daemon is responsible for writing the audit messages that were generated through the audit kernel interface and triggered by application and system activity to disk. The way the audit daemon is started is controlled by systemd . The audit system functions (when started) are controlled by /etc/audit/auditd.conf . For more information about auditd and its configuration, refer to Section 31.2, “Configuring the Audit Daemon”.

The auditctl utility controls the audit system. It controls the log generation parameters and kernel settings of the audit interface and the rule sets that determine which events are tracked. For more information about auditctl , refer to Section 31.3, “Controlling the Audit System Using auditctl ”.

The file /etc/audit/audit.rules contains a sequence of auditctl commands that are loaded at system boot time immediately after the audit daemon is started. For more information about audit rules, refer to Section 31.4, “Passing Parameters to the Audit System”.

The aureport utility allows you to create custom reports from the audit event log. This report generation can easily be scripted, and the output can be used by various other applications, for example, to plot these results. For more information about aureport , refer to Section 31.5, “Understanding the Audit Logs and Generating Reports”.

The ausearch utility can search the audit log file for certain events using various keys or other characteristics of the logged format. For more information about ausearch , refer to Section 31.6, “Querying the Audit Daemon Logs with ausearch ”.

The audit dispatcher daemon ( audispd ) can be used to relay event notifications to other applications instead of (or in addition to) writing them to disk in the audit log. For more information about audispd , refer to Section 31.9, “Relaying Audit Event Notifications”.

The autrace utility traces individual processes in a fashion similar to strace . The output of autrace is logged to the audit log. For more information about autrace , refer to Section 31.7, “Analyzing Processes with autrace ”.

Prints a list of the last logged-in users, similarly to last . aulast searches back through the audit logs (or the given audit log file) and displays a list of all users logged in and out based on the range of time in the audit logs.

Prints the last login for all users of a machine similar to the way lastlog does. The login name, port, and last login time will be printed.

31.2 Configuring the Audit Daemon #Edit source

Before you can actually start generating audit logs and processing them, configure the audit daemon itself. The /etc/audit/auditd.conf configuration file determines how the audit system functions when the daemon has been started. For most use cases, the default settings shipped with SUSE Linux Enterprise Server should suffice. For CAPP environments, most of these parameters need tweaking. The following list briefly introduces the parameters available:

Depending on whether you want your environment to satisfy the requirements of CAPP, you need to be extra restrictive when configuring the audit daemon. Where you need to use particular settings to meet the CAPP requirements, a “ CAPP Environment ” note tells you how to adjust the configuration.

log_file specifies the location where the audit logs should be stored. log_format determines how the audit information is written to disk and log_group defines the group that owns the log files. Possible values for log_format are raw (messages are stored exactly as the kernel sends them) or nolog (messages are discarded and not written to disk). The data sent to the audit dispatcher is not affected if you use the nolog mode. The default setting is raw and you should keep it if you want to be able to create reports and queries against the audit logs using the aureport and ausearch tools. The value for log_group can either be specified literally or using the group’s ID.

Note: CAPP Environment

In a CAPP environment, have the audit log reside on its own partition. By doing so, you can be sure that the space detection of the audit daemon is accurate and that you do not have other processes consuming this space.

Determine how much of a priority boost the audit daemon should get. Possible values are 0 to 20. The resulting nice value calculates like this: 0 — priority_boost

Specifies whether, how, and how often the audit logs should be written to disk. Valid values for flush are none , incremental , data , and sync . none tells the audit daemon not to make any special effort to write the audit data to disk. incremental tells the audit daemon to explicitly flush the data to disk. A frequency must be specified if incremental is used. A freq value of 20 tells the audit daemon to request that the kernel flush the data to disk after every 20 records. The data option keeps the data portion of the disk file synchronized at all times while the sync option takes care of both metadata and data.

Note: CAPP Environment

In a CAPP environment, make sure that the audit trail is always fully up to date and complete. Therefore, use sync or data with the flush parameter.

Specify the number of log files to keep if you have given rotate as the max_log_file_action . Possible values range from 0 to 99 . A value less than 2 means that the log files are not rotated. As you increase the number of files to rotate, you increase the amount of work required of the audit daemon. While doing this rotation, auditd cannot always service new data arriving from the kernel as quickly, which can result in a backlog condition (triggering auditd to react according to the failure flag, described in Section 31.3, “Controlling the Audit System Using auditctl ”). In this situation, increasing the backlog limit is recommended. Do so by changing the value of the -b parameter in the /etc/audit/audit.rules file.

disp_qos and dispatcher

The dispatcher is started by the audit daemon during its start. The audit daemon relays the audit messages to the application specified in dispatcher . This application must be a highly trusted one, because it needs to run as root . disp_qos determines whether you allow for lossy or lossless communication between the audit daemon and the dispatcher.

If you select lossy , the audit daemon might discard some audit messages when the message queue is full. These events still get written to disk if log_format is set to raw , but they might not get through to the dispatcher. If you select lossless the audit logging to disk is blocked until there is an empty spot in the message queue. The default value is lossy .

name_format and name

name_format controls how computer names are resolved. Possible values are none (no name will be used), hostname (value returned by gethostname), fqd (fully qualified host name as received through a DNS lookup), numeric (IP address) and user . user is a custom string that needs to be defined with the name parameter.

max_log_file and max_log_file_action

max_log_file takes a numerical value that specifies the maximum file size in megabytes that the log file can reach before a configurable action is triggered. The action to be taken is specified in max_log_file_action . Possible values for max_log_file_action are ignore , syslog , suspend , rotate , and keep_logs . ignore tells the audit daemon to do nothing when the size limit is reached, syslog tells it to issue a warning and send it to syslog, and suspend causes the audit daemon to stop writing logs to disk, leaving the daemon itself still alive. rotate triggers log rotation using the num_logs setting. keep_logs also triggers log rotation, but does not use the num_log setting, so always keeps all logs.

Note: CAPP Environment

To keep a complete audit trail in CAPP environments, the keep_logs option should be used. If using a separate partition to hold your audit logs, adjust max_log_file and num_logs to use the entire space available on that partition. Note that the more files that need to be rotated, the longer it takes to get back to receiving audit events.

space_left takes a numerical value in megabytes of remaining disk space that triggers a configurable action by the audit daemon. The action is specified in space_left_action . Possible values for this parameter are ignore , syslog , email , exec , suspend , single , and halt . ignore tells the audit daemon to ignore the warning and do nothing, syslog has it issue a warning to syslog, and email sends an e-mail to the account specified under action_mail_acct . exec plus a path to a script executes the given script. Note that it is not possible to pass parameters to the script. suspend tells the audit daemon to stop writing to disk but remain alive while single triggers the system to be brought down to single user mode. halt triggers a full shutdown of the system.

Note: CAPP Environment

Make sure that space_left is set to a value that gives the administrator enough time to react to the alert and allows it to free enough disk space for the audit daemon to continue to work. Freeing disk space would involve calling aureport -t and archiving the oldest logs on a separate archiving partition or resource. The actual value for space_left depends on the size of your deployment. Set space_left_action to email .

Specify an e-mail address or alias to which any alert messages should be sent. The default setting is root , but you can enter any local or remote account as long as e-mail and the network are properly configured on your system and /usr/lib/sendmail exists.

admin_space_left and admin_space_left_action

admin_space_left takes a numerical value in megabytes of remaining disk space. The system is already running low on disk space when this limit is reached and the administrator has one last chance to react to this alert and free disk space for the audit logs. The value of admin_space_left should be lower than the value for space_left . The possible values for admin_space_left_action are the same as for space_left_action .

Note: CAPP Environment

Set admin_space_left to a value that would allow the administrator’s actions to be recorded. The action should be set to single .

Specify which action to take when the system runs out of disk space for the audit logs. Valid values are ignore , syslog , rotate , exec , suspend , single , and halt . For an explanation of these values refer to space_left and space_left_action .

Note: CAPP Environment

As the disk_full_action is triggered when there is absolutely no more room for any audit logs, you should bring the system down to single-user mode ( single ) or shut it down completely ( halt ).

Specify which action to take when the audit daemon encounters any kind of disk error while writing the logs to disk or rotating the logs. The possible value are the same as for space_left_action .

Note: CAPP Environment

Use syslog , single , or halt depending on your site’s policies regarding the handling of any kind of hardware failure.

The audit daemon can receive audit events from other audit daemons. The TCP parameters let you control incoming connections. Specify a port between 1 and 65535 with tcp_listen_port on which the auditd will listen. tcp_listen_queue lets you configure a maximum value for pending connections. Make sure not to set a value too small, since the number of pending connections may be high under certain circumstances, such as after a power outage. tcp_client_ports defines which client ports are allowed. Either specify a single port or a port range with numbers separated by a dash (for example 1-1023 for all privileged ports).

Specifying a single allowed client port may make it difficult for the client to restart their audit subsystem, as it will be unable to re-create a connection with the same host addresses and ports until the connection closure TIME_WAIT state times out. If a client does not respond anymore, auditd complains. Specify the number of seconds after which this will happen with tcp_client_max_idle . Keep in mind that this setting is valid for all clients and therefore should be higher than any individual client heartbeat setting, preferably by a factor of two. tcp_max_per_addr is a numeric value representing how many concurrent connections from one IP address are allowed.

We recommend using privileged ports for client and server to prevent non-root (CAP_NET_BIND_SERVICE) programs from binding to those ports.

When the daemon configuration in /etc/audit/auditd.conf is complete, the next step is to focus on controlling the amount of auditing the daemon does, and to assign sufficient resources and limits to the daemon so it can operate smoothly.

31.3 Controlling the Audit System Using auditctl #Edit source

auditctl is responsible for controlling the status and some basic system parameters of the audit daemon. It controls the amount of auditing performed on the system. Using audit rules, auditctl controls which components of your system are subjected to the audit and to what extent they are audited. Audit rules can be passed to the audit daemon on the auditctl command line or by composing a rule set and instructing the audit daemon to process this file. By default, the auditd daemon is configured to check for audit rules under /etc/audit/audit.rules . For more details on audit rules, refer to Section 31.4, “Passing Parameters to the Audit System”.

The main auditctl commands to control basic audit system parameters are:

auditctl -e to enable or disable audit

auditctl -f to control the failure flag

auditctl -r to control the rate limit for audit messages

auditctl -b to control the backlog limit

auditctl -s to query the current status of the audit daemon

Before running auditctl -S on your system, add -F arch=b64 to prevent the architecture mismatch warning.

The -e , -f , -r , and -b options can also be specified in the audit.rules file to avoid having to enter them each time the audit daemon is started.

Any time you query the status of the audit daemon with auditctl -s or change the status flag with auditctl -e FLAG , a status message (including information on each of the above-mentioned parameters) is printed. The following example highlights the typical audit status message.

Источник