Deploying SCOM 2016 Agents to Domain controllers – some assembly required


 

image

 

Something that a fellow PFE (Brian Barrington) called to my attention, with SCOM 2016 agents, when installed on a Domain Controller:  the agent just sits there and does not communicate.

 

The reason?  Local System is denied by HSLOCKDOWN.

HSLockdown is a tool that grants or denies a particular RunAs account access to the SCOM agent Healthservice.  It is documented here.

 

When we deploy a SCOM 2016 agent to a domain controller – you might see it goes into a heartbeat failed state immediately, and on the agent – you might see the following events in the OperationsManager log:

 

Log Name:      Operations Manager
Source:        HealthService
Event ID:      7017
Task Category: Health Service
Level:         Error
Computer:      DC1.opsmgr.net
Description:
The health service blocked access to the windows credential NT AUTHORITY\SYSTEM because it is not authorized on management group SCOM.  You can run the HSLockdown tool to change which credentials are authorized.

 
Followed eventually by a BUNCH of this:

 

Log Name:      Operations Manager
Source:        HealthService
Event ID:      1102
Task Category: Health Service
Level:         Error
Computer:      DC1.opsmgr.net
Description:
Rule/Monitor “Microsoft.SystemCenter.WMIService.ServiceMonitor” running for instance “DC1.opsmgr.net” with id:”{00A920EF-0147-3FCC-A5DC-CEC1CA93AFED}” cannot be initialized and will not be loaded. Management group “SCOM”

If you open an Elevated command prompt, and browse to the SCOM agent folder – you can run HSLOCKDOWN /L to list the configuration:

 

image

 

There it is.  NT Authority\SYSTEM is denied.

 

I’ll be researching why this change was made – this did not happen by default in SCOM 2012R2. 

In the meantime – the resolution is simple.

 

On domain controllers – simply run the following command in the agent path where HSLOCKDOWN.EXE exists:

 

HSLockdown.exe <YouManagementGroupName> /R “NT AUTHORITY\SYSTEM”

This will remove the explicit deny for Local System.  Restart the SCOM Microsoft Monitoring Agent Service (Healthservice)

.

Here is an example (my management group name is “SCOM”)

image


Comments (15)

  1. Ian Smith says:

    I’m glad we brought this back in 2016! Granting SCOM admin effective domain admin rights was great from an agent setup perspective but horrible from a security point of view.

  2. Steve says:

    So, what if my AD group wants to leave the block in place and run the agent as a domain account? We are currently using the System account for all of our agents, but after showing this post to our AD admins, they are rethinking that. I did a test agent install from my current 2012 R2 system. When I got to the last page of the agent install wizard, I changed the Agent Action Account from Local System to a domain account that has admin rights on the target server. The agent was successfully installed, and the Default Action Account run as profile for the agent correctly reflects the domain account I specified. However, the Microsoft Monitoring Agent service on the target server was installed to run as the Local System account! How do I resolve this? Do I have to just manually change the service account? Will that break anything else?

    1. Kevin Holman says:

      Then it is totally fine if using a domain account for monitoring AD… to leave the default HSLOCKDOWN configuration blocking local system.

      Now – on your specific question – the SCOM “Healthservice” will ALWAYS run as Local System in the services applet. That’s just how it works. The Healthservice doesnt do anything other than handle the communications, and spawn the worker processes. However – the MonitoringHost.exe processes spawned will always run as the default agent action account – or specific RunAs accounts. This is what matters. Don’t change the account that the service executes as – that stays as local system and isnt a security issue. It is the default agent action account that matters. If you do choose to run as a specific domain account, but you grant that account domain admin for it to work – I dont see how that is any more secure than just using Local System on the domain controller.

  3. Steve says:

    Okay, thanks for the fast reply! 🙂

    1. Steve says:

      Actually, I just checked on my test agent install system, and there are three MonitoringHost.exe processes running. Two of them are running as the System account and one is running as the default agent action account as you described. Does this sound normal?

      1. Kevin Holman says:

        Yes – the ones that run under system are internal only and do not run MP workflows, unless there is a workflow that has been associated with the “Privileged Monitoring” profile…. which is specifically used for workflows that MUST run as local system…. of course those would fail to run if local system is denied via HSLOCKDOWN and we would see this in the OpsMgr event logs on the agent.

        1. Steve says:

          Thanks again for the information! 🙂

  4. Steve says:

    Another question (sorry). Is there a general set of workflows that use the privileged monitoring account? I found this article:

    https://support.microsoft.com/en-us/kb/946428

    It says that the account is used to process Health Service configuration. Is this all the account is used for? If not, is there a list somewhere of all the workflows that use this account? Can individual management packs use the account for random workflows? How would one go about finding those, if so? Thanks again for all of your insight Kevin.

  5. René Eigenmann says:

    Thanks for this Information

  6. Saeed says:

    Awesome man, you save me a lot of time 🙂 Thanks

    1. Raj says:

      1.Can SCOM 2012 R2 agent still work and reporting to upgraded SCOM 2016 Management Server group from SCOM 2012 R2 Management group until all the agents are upgraded to SCOM 2016 agent? 2) Can SCOM 2016 agent reporting to SCOM 2012 R2 management group?

      1. Kevin Holman says:

        Q: Can SCOM 2012 R2 agent still work and reporting to upgraded SCOM 2016 Management Server group from SCOM 2012 R2 Management group until all the agents are upgraded to SCOM 2016 agent?

        A: Yes, of course, that is a natural result of upgrading a SCOM e2012 environment. There will be a period of time in the upgrade process where agents are still on SCOM 2012R2 version. We support a SCOM 2012R2 agent reporting to SCOM 2016 as long as it is in the context be being upgraded.

        Q: Can SCOM 2016 agent reporting to SCOM 2012 R2 management group?

        A: Yes. This is would be very common when doing side by side upgrade migrations.

  7. Hello Kevin,

    Thank you for the KB, I did a tool to run the HSLOCKDOWN commands.
    Can you please give your feedback

    https://waleedmostafa.wordpress.com/2017/03/01/scom-2016-agent-hslockdown-tool/

    Thank you in Advance.

  8. Afternoon, how does that apply/work if you are using a low priv account (per the mp guide), with the AD Mgmt Pack to monitor yor environment? Do you need to specify the mgmt group and domain\service account? Would I use the R of A switch to ensure it has no issues related to hslockdown? We are currently 50% 2016 AD/DNS servers, 50% 2012R2 AD/DNS servers? Does this also apply to other servers (2012r2 in particluar) that use the deafult system account/Action account for monitoring/performance? I have run this on numerous of our 212r2 boxes just to check the status and many of them have NT Authority\System denied/not allowed all though they clearly are running correctly?
    Thanks agian,
    Tony…

  9. Thanks for solution, Kevin! But when I do it on DC-server, my SCOM management server became GRAY 🙁
    DC-server is fine(green agent). Searches on the Internet did not yield results, maybe you know how fix it?

Skip to main content