Scom monitoring linux log files

Linux Log file monitoring in System Center Operations Manager

This version of Operations Manager has reached the end of support, we recommend you to upgrade to Operations Manager 2019.

System Center Operations Manager now has enhanced log file monitoring capabilities for Linux servers by using the newest version of the agent that uses Fluentd. This update provides the following improvements over previous log file monitoring:

• Wild card characters in log file name and path.
• New match patterns for customizable log search like simple match, exclusive match, correlated match, repeated correlation and exclusive correlation.
• Support for generic Fluentd plugins published by the fluentd community.

Basic operation

The basic operation of log file monitoring in Linux includes the following steps:

1. Record is written to a log on a Linux agent.
2. Fluentd collects the record and creates an event on pattern match.
3. Event is sent to OMED service on management server.
4. Rules and monitors in a custom management pack collect events and create alerts in Operations Manager.

Overview of configuration

The following steps are required to enable log file monitoring on Linux agents. Each of these steps is described in detail in the following sections.

1. Import the latest Linux management pack.
2. Install the latest version of the Linux agent on each Linux computer to be monitored.
3. Create Fluentd configuration file to collect logs.
4. Copy configuration file to Linux agents.
5. Create rules and monitors using the sample management pack to collect events from the log and create alerts.

Install the latest version of the Linux agent

The latest version of the Linux agent supports Fluentd, which is required for enhanced log file monitoring. You can get details and the installation process for the new agent at Install agent on UNIX and Linux from command line.

Configure Linux Log File monitoring

The Linux Management pack bundle has the latest Operations Manager agent (with Fluentd). To configure Linux log file monitoring, users should perform the following:

1. Import the latest Linux Management pack using the standard process for installing a management pack.
2. Install the new Linux agent on the Linux servers, this can be done through discovery wizard or manually.
3. Enable the OMED service on each management server in the resource pool managing the Linux agents.

The OMED service collects events from Fluentd and converts them to Operations Manager events. Users should import a custom management pack which can generate alerts based on the events received from the Linux servers.

You enable the OMED service either from the Operations console or manually on the management server or gateway server.

From Operations console

1. From the Operations Console, go to Monitoring >Operations Manager >Management Server >Management Servers State.
2. Select the management server in the Management Servers state pane.
3. In the Tasks pane, select Health Service Tasks >Enable System Center OMED Server.

Manually

1. Click Start, in the Start Search box, type services.msc , and then press Enter.
2. In the details pane, right-click the service System Center Operations Manager External DataSource Service, and then click Properties.
3. On the General tab, in Startup type , click Automatic, and then click OK.
4. In the details pane, right-click the service and then click Start.

Create FluentD configuration file

You configure Fluentd operation with a configuration file. For log monitoring, you need to create a configuration file that includes such information as source log file name and path and filters to define which data to collect.

The master Fluentd configuration file omsagent.conf is located in /etc/opt/microsoft/omsagent/scom/conf/. You can add log file monitoring configuration directly to this file, but you should create a separate configuration file to better manage the different settings. You then use an @include directive in the master file to include your custom file.

For example, if you created logmonitoring.conf in /etc/opt/microsoft/omsagent/scom/conf/omsagent.d, you would add one of the following lines to fluent.conf:

You can get details on Fluentd configuration files at Fluentd Configuration file syntax. The following sections describe settings in different directives of the configuration file unique to log file monitoring. Each includes sample settings that you can paste into a configuration file and modify for your requirements.

A complete sample configuration file for log monitoring is available for you to review and evaluate before creating your own.

Source

The Source directive defines the source of the data you’re collecting. This is where you define the details of your log file. Fluentd picks up each record written to the source and submits an event for it into Fluentd’s routing engine. You need to specify a tag here in this directive. The tag is a string that is used as the directions for Fluentd’s internal routing engine to correlate different directives.

This example shows syslog records collected and tagged for processing by Operations Manager.

Match

The match directive defines how to process events collected from the source with matching tags. Only events with a tag matching the pattern will be sent to the output destination. When multiple patterns are listed inside one match tag, events can match any of the listed patterns. The type parameter that specifies which plugin to use for these events.

This example processes events with tags matching scom.log.** and scom.alert (** matches zero or more tag parts). It specifies the out_scom plugin which allows the events to be collected by the Operations Manager management pack.

To disable Server Auth on the Linux machines that are using Fluentd communication, add a parameter enable_server_auth false to the SCOM out plugin for Fluentd, such as the following:

Filter

The filter directive has same syntax as match but allows for more complex filtering of which data to process. Collected events must match the criteria of all filters to be added to the output.

There are six filter plugins for log file monitoring described here. Use one or more of these filters to define the events that you want to collect from your log file.

Simple match: filter_scom_simple_match

Takes up to 20 input patterns. Sends an event to Operations Manager whenever any pattern is matched.

Exclusive match: filter_scom_excl_match

Takes two input patterns. Sends an event to Operations Manager when a single record matches pattern 1 but does not match pattern 2.

Repeated correlation: filter_scom_repeated_cor

Takes three inputs: a patterns, a time interval, and a number of occurrences. When a match is found for the first pattern, a timer starts. An event is sent to Operations Manager if the pattern is matched the specified number of times before the timer ends.

Correlated match: filter_scom_cor_match

Takes three inputs: two patterns and a time interval. When a match is found for the first pattern, a timer starts. An event is sent to Operations Manager if there is a match for the second pattern before the timer ends.

Exclusive correlation: filter_scom_excl_correlation

Takes three inputs: two patterns and a time interval. When a match is found for the first pattern, a timer starts. An event is sent to Operations Manager if there is no match for the second pattern before the timer ends.

Operations Manager converter: filter_scom_converter

Sends an event to Operations Manager for all records it receives. Sends the specified event ID and description as part of the event.

Copy configuration file to agent

The fluentd configuration file must be copied to /etc/opt/microsoft/omsagent/scom/conf/omsagent.d on all Linux computers you want to monitor. You must also add an @include directive in the master configuration file as described above.

Create rules and monitors

The Linux MP does not provide modules to collect events from FluentD. The Linux MP is bundled with the Linux agent. It is the fluentd module in the Linux agent and the OMED service on the management and gateway server that provides the capabilities for enhanced log file monitoring.

You need to create your own management pack with custom rules and monitors that use the module Microsoft.Linux.OMED.EventDataSource which collects the events from Fluentd.

The following table lists the parameters of Microsoft.Linux.OMED.EventDataSource.

Parameter	Type	Description
ComputerName	String	Required. Specifies the name of the Linux computer for which events to be read. The ComputerName parameter is most commonly passed to the module by using the $Target notation, although it can be specified as any string. This module attempts to read events generated by the given Linux computer.
ManagedEntityId	String	Required. Specifies the managed entity ID of monitored entity. The ManagedEntityId parameter is most commonly passed to module by using $Target\Id$.
EventNumber	Integer	Optional. Indicates the event number of the event to retrieve. If this option is omitted, the module returns all events generated for that computer and managed entity.

Overview of configuration

The following steps are required to enable log file monitoring on Linux agents. Each of these steps is described in detail in the following sections.

1. Import the latest Linux management pack.
2. Install the latest version of the Linux agent on each Linux computer to be monitored.
3. Create Fluentd configuration file to collect logs.
4. Copy configuration file to Linux agents.
5. Create rules and monitors using the sample management pack to collect events from the log and create alerts.

Install the latest version of the Linux agent

The latest version of the Linux agent supports Fluentd, which is required for enhanced log file monitoring. You can get details and the installation process for the new agent at Install agent on UNIX and Linux from command line.

Configure Linux Log File monitoring

The Linux Management pack bundle has the latest Operations Manager agent (with Fluentd). To configure Linux log file monitoring, users should perform the following:

1. Import the latest Linux Management pack using the standard process for installing a management pack.
2. Install the new Linux agent on the Linux servers, this can be done through discovery wizard or manually.
3. Enable the OMED service on each management server in the resource pool managing the Linux agents.

The OMED service collects events from Fluentd and converts them to Operations Manager events. Users should import a custom management pack which can generate alerts based on the events received from the Linux servers.

You enable the OMED service either from the Operations console or manually on the management server or gateway server.

From Operations console

1. From the Operations Console, go to Monitoring >Operations Manager >Management Server >Management Servers State.
2. Select the management server in the Management Servers state pane.
3. In the Tasks pane, select Health Service Tasks >Enable System Center OMED Server.

Manually

1. Click Start, in the Start Search box, type services.msc , and then press Enter.
2. In the details pane, right-click the service System Center Operations Manager External DataSource Service, and then click Properties.
3. On the General tab, in Startup type , click Automatic, and then click OK.
4. In the details pane, right-click the service and then click Start.

Create FluentD configuration file

You configure Fluentd operation with a configuration file. For log monitoring, you need to create a configuration file that includes such information as source log file name and path and filters to define which data to collect.

The master Fluentd configuration file omsagent.conf is located in /etc/opt/microsoft/omsagent/scom/conf/. You can add log file monitoring configuration directly to this file, but you should create a separate configuration file to better manage the different settings. You then use an @include directive in the master file to include your custom file.

For example, if you created logmonitoring.conf in /etc/opt/microsoft/omsagent/scom/conf/omsagent.d, you would add one of the following lines to fluent.conf:

You can get details on Fluentd configuration files at Fluentd Configuration file syntax. The following sections describe settings in different directives of the configuration file unique to log file monitoring. Each includes sample settings that you can paste into a configuration file and modify for your requirements.

A complete sample configuration file for log monitoring is available for you to review and evaluate before creating your own.

Source

The Source directive defines the source of the data you’re collecting. This is where you define the details of your log file. Fluentd picks up each record written to the source and submits an event for it into Fluentd’s routing engine. You need to specify a tag here in this directive. The tag is a string that is used as the directions for Fluentd’s internal routing engine to correlate different directives.

This example shows syslog records collected and tagged for processing by Operations Manager.

Match

The match directive defines how to process events collected from the source with matching tags. Only events with a tag matching the pattern will be sent to the output destination. When multiple patterns are listed inside one match tag, events can match any of the listed patterns. The type parameter that specifies which plugin to use for these events.

This example processes events with tags matching scom.log.** and scom.alert (** matches zero or more tag parts). It specifies the out_scom plugin which allows the events to be collected by the Operations Manager management pack.

To disable Server Auth on the Linux machines that are using Fluentd communication, add a parameter enable_server_auth false to the SCOM out plugin for Fluentd, such as the following:

Filter

The filter directive has same syntax as match but allows for more complex filtering of which data to process. Collected events must match the criteria of all filters to be added to the output.

There are six filter plugins for log file monitoring described here. Use one or more of these filters to define the events that you want to collect from your log file.

Simple match: filter_scom_simple_match

Takes up to 20 input patterns. Sends an event to Operations Manager whenever any pattern is matched.

Exclusive match: filter_scom_excl_match

Takes two input patterns. Sends an event to Operations Manager when a single record matches pattern 1 but does not match pattern 2.

Repeated correlation: filter_scom_repeated_cor

Takes three inputs: a patterns, a time interval, and a number of occurrences. When a match is found for the first pattern, a timer starts. An event is sent to Operations Manager if the pattern is matched the specified number of times before the timer ends.

Correlated match: filter_scom_cor_match

Takes three inputs: two patterns and a time interval. When a match is found for the first pattern, a timer starts. An event is sent to Operations Manager if there is a match for the second pattern before the timer ends.

Exclusive correlation: filter_scom_excl_correlation

Takes three inputs: two patterns and a time interval. When a match is found for the first pattern, a timer starts. An event is sent to Operations Manager if there is no match for the second pattern before the timer ends.

Operations Manager converter: filter_scom_converter

Sends an event to Operations Manager for all records it receives. Sends the specified event ID and description as part of the event.

Copy configuration file to agent

The fluentd configuration file must be copied to /etc/opt/microsoft/omsagent/scom/conf/omsagent.d on all Linux computers you want to monitor. You must also add an @include directive in the master configuration file as described above.

Create rules and monitors

The Linux MP does not provide modules to collect events from FluentD. The Linux MP is bundled with the Linux agent. It is the fluentd module in the Linux agent and the OMED service on the management and gateway server that provides the capabilities for enhanced log file monitoring.

You need to create your own management pack with custom rules and monitors that use the module Microsoft.Linux.OMED.EventDataSource which collects the events from Fluentd.

The following table lists the parameters of Microsoft.Linux.OMED.EventDataSource.

Parameter	Type	Description
ComputerName	String	Required. Specifies the name of the Linux computer for which events to be read. The ComputerName parameter is most commonly passed to the module by using the $Target notation, although it can be specified as any string. This module attempts to read events generated by the given Linux computer.
ManagedEntityId	String	Required. Specifies the managed entity ID of monitored entity. The ManagedEntityId parameter is most commonly passed to module by using $Target\Id$.
EventNumber	Integer	Optional. Indicates the event number of the event to retrieve. If this option is omitted, the module returns all events generated for that computer and managed entity.

Overview of configuration

log file monitoring requires the following steps, detailed information for these is provided in the following sections:

1. Import the latest Linux management pack.
2. Install the latest version of the Linux agent on each Linux computer to be monitored.
3. Install latest OMSAgent on each Linux computer to be monitored.
4. Create Fluentd configuration file to collect logs.
5. Copy configuration file to Linux agents.
6. Create rules and monitors using the sample management pack to collect events from the log and create alerts.

Install the log monitoring management pack

In Operations Manager 2019, install Microsoft.Linux.Log.Monitoring management pack to enable Linux log file monitoring.

If you have the OMS agent configured, and you try to uninstall UNIX and LINUX agent from the console, then OMS component will not be uninstalled from the agent.

Configure Linux log file monitoring

To configure Linux log file monitoring, do the following:

Import the latest Linux management pack using the standard process for installing a management pack.

Install the new Linux agent on the Linux servers manually or by using Discovery wizard.

Install latest OMSAgent on each Linux computer that you want to monitor.

Источник