Adam the Automator

How To Set Up Windows Event Log Forwarding In Windows Server 2016

Jeff Christman

Read more posts by this author.

Event log management is a critical skill to learn in all Windows environments. Activity is being recorded to Windows event logs every second and it acts as not only a security tool but also as a vital troubleshooting aid.

Once a server environment goes past a few servers though, managing individual server event logs becomes unwieldy at best. Luckily, you have a feature called Windows Event Forwarding (WEF) to make it easier.

Table of Contents

Windows Event Log Forwarding Overview

WEF is a service that allows you to forward events from multiple Windows servers and collect them in one spot. The service has two main components; a forwarder and a collector. A collector is a service running on Windows server that collects all events sent to it from an event log forwarder.

The “link” between the forwarding server and a collector is known as a subscription.

Collectors serve as subscription managers that accept events and allow you to specify which event log alerts to collect from endpoints.

WEF Project Overview

This is a Project article where we cover how to build a project or implement a solution. Each section hereafter will be cumulative steps that build upon the previous.

For this project, you’re going to learn how to set up a basic WEF implementation. You’ll learn how to set up both a collector and how to forward events to a collector with a subscription.

You’ll learn how to:

1. Set up and configure an event log collector on a Windows Server instance. This will be the Windows Server that all of the event log forwarders will send events to.
2. Create a GPO which, when applied, will point applicable Windows Server instances to the collector to send events to.
3. Configuring the types of events to send to the collector.

You will learn how to work through each step in the remainder of this article.

Environment and Knowledge Requirements

Before you get too far, let’s first ensure my environment is the same as yours. Please be sure you have the following items in place before starting:

• (2) Windows Server instances – You can use any Window Server instance of 2012 R2 or higher. In this article, I’ll be using Windows Server 2016.
• Active Directory
• GPO – A familiarity with Group Policy Objects will be required.
• WinRM- WinRM needs to be running on all clients. Not configured just running.

Configuring the Event Log Collector

The first task to perform is configuring one of your Windows Server instances as the collector. Recall that the collector is the one that receives incoming event logs from the forwarder.

Enabling WinRM on the Collector

Windows Server instances that forward events to the collector do so over PowerShell Remoting or WinRM. You’ll first have to ensure WinRM is available on your collector. If the collector is running Windows Server 2012 R2 and above, WinRM is enabled by default, but the Windows Firewall may be interfering.

Run the the Enable-PSRemoting PowerShell cmdlet with no parameters on the collector. Even if PowerShell Remoting is already enabled, it will skip the necessary steps.

To be sure, you can also run Invoke-Command -ComputerName -ScriptBlock <1>from a remote computer. If you don’t receive an error, PowerShell Remoting is working.

Starting the Subscription Collector Service

Now that PowerShell Remoting is enabled and listening, start the subscription collector service. The subscription collector service needs to also start up automatically when Windows Server boots up.

On the collector, open Event Viewer click on Subscriptions. The first time you open the Subscriptions option, Windows will ask if you want to start the Windows Event Log Collector Service and configured to start automatically. Click Yes to accept.

You can see an example of the message below.

Congratulations! You now have a collector configured. It’s now time set up a GPO which will instruct Windows Server instances to forward events to the collector.

Setting up the Forwarders’ GPO

The next step is to configure one or more Windows servers to begin forwarding event logs to the collector. The easiest way to do so is by creating a GPO. This GPO can then be applied to one or more OUs which contain the servers to send events from.

You’ll learn the basics of setting up the necessary settings in a GPO in this Project article. But if you’d like to a complete rundown with all the available options, check out the Microsoft documentation.

Allowing the Network Service to Read Event Logs

WEF uses the Network Service account to read and send events from a forwarder to a collector. By default, the Network Service account does not have access to do this. You’ll first need to set this ACL to allow it.

Note: Many of the event logs in Windows Server already provide the Network Service account access to the common event logs like Application and System. But the account is not given access to the Security event log and other custom event logs.

To allow the Network Service account to read event logs on event log forwarders, use a GPO. In this article, you’ll learn how to allow the Network Service account access to the Security event log. Other event logs will follow the same process.

1. Begin by opening up a command prompt and running wevtutil gl security . This will provide various information about the Security event log. But the piece to pay attention to is the channelAccess SDDL.

You can see below an example of the SDDL you’ll need for the Security event log. The channelAccess line represents the permissions set on the event log. Copy the SDDL highlighted below and save it somewhere for later to add to a GPO.

2. Create a GPO via the Group Policy Management Console. Inside of the GPO, navigate to Computer Configuration → Policies → Administrative Templates → Windows Components → Event Forwarding → Configure target subscription manager.

3. Set the value for the target subscription manager to the WinRM endpoint on the collector. You will set the Server to be in the format:

Note the Refresh interval at the end of the collector endpoint. The Refresh interval indicates how often clients should check in to see if new subscriptions are available.

4. Next, find the SDDL you copied earlier from running wevtutil gl security and paste it into the setting Computer Configuration → Policies → Administrative Templates → Windows Components → Event Log Service → Security → Configure log access.

Note that this SDDL will take precedence over all other permissions that have been configured for the event log.

You can see an example of what your GPO will look like below for the Security event log.

5. Once the GPO is created, you’ll then either link this GPO to an existing OU containing the Windows servers to send event logs from or create a new OU and link the GPO. Any AD computer account you add to this OU will now set up a subscription to the collector.

Setting up a Subscription

While configuring WEF to collect all events for all Windows servers in an Active Directory domain may seem like a good idea, it’s not. You must be selective and only forward events that are important to you. Filtering out the noise from what matters is where WEF demonstrates its true value.

Let’s work through setting up a subscription for the Security Event log.

Since you’ve already created the GPO and linked it to an Active Directory OU containing the Windows servers you’d like to send events from, the event sources are already set up

1. On the collector, open the Windows Event Viewer and right-click on Subscriptions, then create subscription.

2. As shown below, select the Source computer initiated option and then click Select Computer Groups. This is where you will select which computers you’d like to forward events from.

Pro Tip: Selecting AD Groups. Ex: “Domain Controllers” will auto-populate any computers within the group. No need to select individual computers every time you add a new server.

3. Next select the events to forward. Opening up the query filter as you can see below, select Security to forward events to the collector from the Security event log.

4. Once the Security log is selected, you can filter down even more by entering the event ID, keywords, users and computers as shown below.

5. Click OK to exit from the Query Filter.

6. Click Advanced in the Subscription Properties window. Now select Minimize Latency. This setting will ensure the collector will receive events as soon as possible and also to help it catch up if it gets behind.

Verifying the WEF Configuration

Once WEF is set up, you should now check to see if the forwarders actually checked in by checking the Source Computers column on the main Subscriptions page.

You can also check the Event Forwarding Plugin Operational log under Applications and Services on the client to make sure everything is working. This is where you’ll see descriptive errors if something has gone awry with Kerberos or firewalls.

All that is left to to is find a low-value client, clear the Security log and see if you get an alert.

Your Takeaways

In this Project, you learned how to set up a basic WEF subscription. You:

• Set up an event collector
• Created a GPO to create a subscription on various Windows Server forwarders
• Configured a WEF subscription to only send specific events
• Ensured the WEF subscription sent events as fast as possible

WEF is a bit tricky to configure initially, but once up and running, you should have little problems and minimal maintenance headaches.

Forwarded Events log is unavailable and EventCollector log is not displayed after you uninstall Windows Management Framework 3.0 in Windows 7 SP1 and Windows Server 2008 R2 SP1

Symptoms

Assume that you install Windows Management Framework 3.0 on a computer that is running Windows 7 Service Pack 1 (SP1) or Windows Server 2008 R2 SP1. After you uninstall the Windows Management Framework 3.0 package and then restart the computer, you encounter the following issues in Event Viewer:

When you click Windows Logs, and then click Forwarded Events, all the Forwarded Events logs become unavailable.

When you click Applications and Services Logs, click Microsoft, and then click Windows, EventCollector is not displayed.

Note For more information about Windows Management Framework 3.0, click the following article number to view the article in the Microsoft Knowledge Base:

2506143 Description of Windows Management Framework 3.0 for Windows 7 SP1 and Windows Server 2008 R2 SP1

Cause

These issues occur because the registry keys that are related to the Forwarded Events log are removed when you uninstall of the Windows Management Framework 3.0 package.

Note For information about the removed keys, refer to the Registry Key section in this article.

Resolution

Hotfix information

To resolve this issue, install the update that is described in update 2823180. For more information about how to obtain this update, click the following article number to view the article in the Microsoft Knowledge Base:

2823180 Update is available for Windows Management Framework 3.0 in Windows 7 SP1, Windows Server 2008 R2 SP1, or Windows Server 2008 SP2

Prerequisites

To apply this hotfix, you must be running Windows 7 SP1 or Windows Server 2008 R2 SP1. Additionally, Windows Management Framework 3.0 must be installed or previously installed in Windows.

For more information about how to obtain a Windows 7 or Windows Server 2008 R2 service pack, click the following article number to view the article in the Microsoft Knowledge Base:

976932 Information about Service Pack 1 for Windows 7 and for Windows Server 2008 R2

Registry information

After you uninstall the Windows Management Framework 3.0 package, the following registry keys are deleted from your computer:

GeekSilver’s Blog

Create Eventlog forwarding and WinRM

• 15/01/2016 – 12:54 pm
• Posted in Windows 2012
• Tagged event forwarding, Windows 2012, winrm

okay, I’m glad I can back and write something newish. This post is all about creating event log forwarding , centralized event log and WinRM.

Overview

Why you need centralized event log solution

Windows Event log has always been first line of defense and reflect what happened to your computers. It will be your company’s frontier defense line against PTH, or any hack attacks. If any events happened to IT people’s laptop that has privilege account logged in before, it will be great early alert for IT Admin to take action against this account or focus and track it down.

In the ideal world, we would have all events from everyone and understand what exactly happened. But the reality is no one is able to handle that amount of work and whether this can be efficient enough to provide useful information is another question.

If collector servers or clients are offline, the related events will be holding and submit to server once client/server comes back online.

so this is big Yes to nice to have, but how?

Who we are collecting

Because we only monitor very critical and abnormal events (like security logs get wipe out), the chance it happens should be very minimum so we don’t need big space for log collector. We can collect event logs from Laptops, Servers, Desktops which is assigned by computer groups. We can deploy GPO to enable computers to look for collector for subscriptions. Each computer can submit to multiple collectors at same time.

What we collecting

We only collect critical events like security logs get wipe out, local administrator account get logged in laptop or local administrator group membership has been changed, service get installed at beginning level. Those events will be absolutely critical. We can control which event we want easily for each subscriptions.

What we do with those logs

SCOM can be used to monitor those security logs and alert to related teams for further investigation. SIEM can be used to collect logs from log collect server and log server can overwrite old logs to save disk space.

Enough to say, let’s take some action here.

I’m going to build 1 collector server to collect one client log. Yes, you can use multiple collectors as active active solution just in case one of collectors is down.

In this lab, I’m going to use HTTPS as protocol rather than HTTP.

Configuration

Tasks on Collect server

We have quite few things to do on the collect server. The first step is to enable Winrm on the server.

WinRM configuration

WinRM is acting as proxy and interface on the server and passing the request to event log service in the background. Hence we must enable WinRM.

One of precondition to enable WinRM is to enable firewall service. Because when you run winRM qc, following things happened.

The above command will perform the following steps:

• Start the WinRM service.
• Set the WinRM service type to auto start.
• Create an HTTP listener on port 5985 to accept requests on any IP address.
• Enable firewall exception for WS-Management traffic (for http only)

In old WinRM, it’s using port 80/443. From WinRM 2.0, it starts to use 5985/5986.

Hence, yes, windows Firewall must be on.

Next, we need to create a new Rule as we are going to use HTTPS 5986.

So You must create Inbound Rules to allow TCP 5986 to work.

If you enable windows firewall, you might want to open following ports as well.

Remote Desktop – User Mode (both TCP/UDP)

File and Printer Sharing (Echo Request – ICMPv4-In)

Then, you can run Winrm qc

Winrm qc is Winrm quick config to configures this machine to accept WS-Management request from other machine. (think about Web Proxy)

By default, WinRM can be used for different Resource URIs. It can be used by WMI, IPMI, WinRM Configuration and of course, Eventlog URI.(think about Web proxy acting as front listener and pass information to Exchange or other servers behind firewall).

When client hit on listener, depends on the path of files client API is access, different URIs will respond.

After you run winrm qc, (you also need to start WinRM service on all clients, just need to start service, no need to create listener). you can use following command to test.

You can run Winrm id

This information to prove WinRM is starting correctly. Also it tells you which URI responsible for security profiles.

For detecting client firewall and server whether they can reach to each other, following command can be used.

Winrm id -r:dest_server

Winrm id -r:source_server

now, we need to check whether listener is present.

Winrm e winrm/config/listener

Great, now we have a listener which accept request.

But notice it is HTTP protocol, there is no HTTPS?

In terms of getting HTTPS, you would need to have a Web Server certificate. A standard web server certificate will suffice there is no need to create a template for it. Just make sure you put FQDN in common name and DNS name as well. nothing special.

Once the certificate in place, you need to run mmc->Add Certificate snap-in ->Computer account

Double click the certificate (you generated from CA), go to Details and select Thumbprint

Now, you need to high light all details of certificate thumbprint and Ctrl+C to copy the content

winrm create winrm/config/Listener?Address=*+Transport=HTTPS @

Modify above commandline with new content from thumbprint and also ServerFQDN, run it in Server command windows.

Use Winrm e winrm/config/listener to double check

You should see HTTPS appears as well.

Notice you got hostname, IP, and Certificate Thumbprint here.

If you somehow want to delete and reset everything because you did something wrong, use following command.

winrm invoke Restore winrm/Config @<>

Be aware this reset winrm configuration. so if you have other important thing on WINRM, you need to be more specific

Now, Winrm is ready to use on Server.

Setup SPN for your server

WinRM is using kerberos as authentication by default, hence SPN is required.

after finishing WinRM, you can double check whether SPN is registered by running

setspn -l servername

then, you are looking for WSMAN/servername and WSMAN/ServerFQDN

If you can’t find it, you must use setspn to create one.

Eventlog configuration

Next step is configure Event forwarding subscription.

go to services.msc to make sure Windows event Collector service is running

Remember client will reach server to download subscription to find out what they need to upload.

First, we need to create subscription, open event viewer

Notice I select Source computer initiated.

the reason I select event 999 is I can only create my event between 1-1000. so 999 is selected here.

select HTTPS and Minmize Latency for the lab fact

click OK, OKAY, then it’s finished.

GPO configuration

Now, we need to create GPO.

There are two basic items you must put into GPO.

first one is the link lead client to server.

This is where you configure the link for client seeking collector server. As you can see from the picture, I have setup two servers and one for http, one for HTTPS. Client is able to send events to both servers.

Be aware the format of link has to be Server=http://serverFQDN:5985/wsman/SubscriptionManager/WEC,Refresh=10

The refresh here means how often client contacts server for subscription information. 10 means 10 minutes.

If you want to refresh client to download latest subscription, best way to do is run gupdate /force

The second part of GPO item is security for event log Service.

Event log service on client must allow Network Service to access and transfer events to collector Server. hence, you need to grant permission for it.

The way you do is as following:

log on to client and run following command line

wevtutil gl security

noticing everything after ChannelAccess:, which is start with O:BAG:SYD:xxxxx this is the one we after.

now, let’s read this line, it doesn’t contain (A;;0x1;;;NS). If it doesn’t, you need to add this one at the end of this line.

now, put it into GPO.

and push the policy to client.

Client Configuration

Client configuration is relatively easy. Just make sure WinRM service is running(don’t need to be configured). Group policy has been pushed and applied.

Now, we do can use command line to manually create event to verify whether collector has got it.

eventcreate /T Error /ID 999 /L application /D “Test0001”

run this command in CMD window, it will create event in the application.

Troubleshooting

The main troubleshooting log is from server and client end.

Event logs

check Forwarded Events from Windows Logs of Server to see forwarded events

Check Applications and Services->Microsoft->Windows->EventCollector

Check Applications and Services->Microsoft->Windows->Eventlog-ForwardingPlugin

Check Applications and Services->Microsoft->Windows->Windows Remote Management

Errors I have encountered

Access denied, error code 5

I had a terrible experience on my first run which I spent days trying to resolve it.

If you can recall that network service is used from client to communicate to server, well, network service will act as computer object over the network. so from server point of view, this is request from Client computer account. My server somehow get default setting in security to block all computer account access.

by default, there should be a group called “Everyone”. but it’s missing. After I added authenticated users group into this security, everything works.

Encountered an internal error in SSL library

this is one of silly mistake I made in my life. After successful test with HTTP, I switched to HTTPS to make it work. but clearly, I forget to change port from 5985 to 5986. For trying to fix that, I even created a whole new template of cerificate….

If you replacing certificate, you need to reboot your server. Restart WINRM service is not enough.