Deploying SCOM 2016 Agents to Domain controllers – some assembly required


 

image

 

Something that a fellow PFE (Brian Barrington) called to my attention, with SCOM 2016 agents, when installed on a Domain Controller:  the agent just sits there and does not communicate.

 

The reason?  Local System is denied by HSLOCKDOWN.

HSLockdown is a tool that grants or denies a particular RunAs account access to the SCOM agent Healthservice.  It is documented here.

 

When we deploy a SCOM 2016 agent to a domain controller – you might see it goes into a heartbeat failed state immediately, and on the agent – you might see the following events in the OperationsManager log:

 

Log Name:      Operations Manager
Source:        HealthService
Event ID:      7017
Task Category: Health Service
Level:         Error
Computer:      DC1.opsmgr.net
Description:
The health service blocked access to the windows credential NT AUTHORITY\SYSTEM because it is not authorized on management group SCOM.  You can run the HSLockdown tool to change which credentials are authorized.

 
Followed eventually by a BUNCH of this:

 

Log Name:      Operations Manager
Source:        HealthService
Event ID:      1102
Task Category: Health Service
Level:         Error
Computer:      DC1.opsmgr.net
Description:
Rule/Monitor “Microsoft.SystemCenter.WMIService.ServiceMonitor” running for instance “DC1.opsmgr.net” with id:”{00A920EF-0147-3FCC-A5DC-CEC1CA93AFED}” cannot be initialized and will not be loaded. Management group “SCOM”

If you open an Elevated command prompt, and browse to the SCOM agent folder – you can run HSLOCKDOWN /L to list the configuration:

 

image

 

There it is.  NT Authority\SYSTEM is denied.

 

I’ll be researching why this change was made – this did not happen by default in SCOM 2012R2. 

In the meantime – the resolution is simple.

 

On domain controllers – simply run the following command in the agent path where HSLOCKDOWN.EXE exists:

 

HSLockdown.exe <YouManagementGroupName> /R “NT AUTHORITY\SYSTEM”

This will remove the explicit deny for Local System.  Restart the SCOM Microsoft Monitoring Agent Service (Healthservice)

.

Here is an example (my management group name is “SCOM”)

image


Comments (10)

  1. Ian Smith says:

    I’m glad we brought this back in 2016! Granting SCOM admin effective domain admin rights was great from an agent setup perspective but horrible from a security point of view.

  2. Steve says:

    So, what if my AD group wants to leave the block in place and run the agent as a domain account? We are currently using the System account for all of our agents, but after showing this post to our AD admins, they are rethinking that. I did a test agent install from my current 2012 R2 system. When I got to the last page of the agent install wizard, I changed the Agent Action Account from Local System to a domain account that has admin rights on the target server. The agent was successfully installed, and the Default Action Account run as profile for the agent correctly reflects the domain account I specified. However, the Microsoft Monitoring Agent service on the target server was installed to run as the Local System account! How do I resolve this? Do I have to just manually change the service account? Will that break anything else?

    1. Kevin Holman says:

      Then it is totally fine if using a domain account for monitoring AD… to leave the default HSLOCKDOWN configuration blocking local system.

      Now – on your specific question – the SCOM “Healthservice” will ALWAYS run as Local System in the services applet. That’s just how it works. The Healthservice doesnt do anything other than handle the communications, and spawn the worker processes. However – the MonitoringHost.exe processes spawned will always run as the default agent action account – or specific RunAs accounts. This is what matters. Don’t change the account that the service executes as – that stays as local system and isnt a security issue. It is the default agent action account that matters. If you do choose to run as a specific domain account, but you grant that account domain admin for it to work – I dont see how that is any more secure than just using Local System on the domain controller.

  3. Steve says:

    Okay, thanks for the fast reply! 🙂

    1. Steve says:

      Actually, I just checked on my test agent install system, and there are three MonitoringHost.exe processes running. Two of them are running as the System account and one is running as the default agent action account as you described. Does this sound normal?

      1. Kevin Holman says:

        Yes – the ones that run under system are internal only and do not run MP workflows, unless there is a workflow that has been associated with the “Privileged Monitoring” profile…. which is specifically used for workflows that MUST run as local system…. of course those would fail to run if local system is denied via HSLOCKDOWN and we would see this in the OpsMgr event logs on the agent.

        1. Steve says:

          Thanks again for the information! 🙂

  4. Steve says:

    Another question (sorry). Is there a general set of workflows that use the privileged monitoring account? I found this article:

    https://support.microsoft.com/en-us/kb/946428

    It says that the account is used to process Health Service configuration. Is this all the account is used for? If not, is there a list somewhere of all the workflows that use this account? Can individual management packs use the account for random workflows? How would one go about finding those, if so? Thanks again for all of your insight Kevin.

  5. René Eigenmann says:

    Thanks for this Information

  6. Saeed says:

    Awesome man, you save me a lot of time 🙂 Thanks

Skip to main content