I`ve saw some posts about this topic, however none of them really tell you how to achieve this behavior or if it even really works. So I`ve made a lab scenario where I wanted to set up my authentication not via Kerberos, but via certificates however without a PKI. This can be done by generating your own self signed certificates for all SCOM members- MS, GW, agents.. however without PKI templates or inf requests this is tricky.
Before beginning to explain this, I just want to point out that this is unsupported (even if it works)and should be used only as a last resort or where a PKI investment is not worth like very small environments or so.
On the MS you`re going to create a self signed certificate with [View:~/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-01-65/self_2D00_signed_5F00_cert_5F00_SCOM.txt.ps1:550:0] the following script (self_signed_cert_SCOM) - it`s basically a powershell script where you specify the FQDN of the server and it will store a certificate in the computer personal store.
After this, use momcertimport against this certificate you just created which now lies in the personal store- it will write the certificate serial number in reverse order under HKLM\Software\Microsoft\Microsoft OperationsManager\3.0\Machine Settings
Double Click the Key ChannelCertificateSerialNumber and delete half of the serial number.
Export the MS certificate from the store with the private key and copy it in the trusted root store on the Gateway or the agent with who you want to build the connection to.
On the GW/Agent follow the same steps as per MS, just copy the MS certificate in the trusted root store, other steps are identical.
After this, make a cache flush (delete Health Service State folder) from GW/Agent and MS and check if the certificates have been loaded successfully. This is event ID 20053.
PS: If you have CN and DNS entries for the same hostname so double resolution for a server- just request the certificate with the DNS resolution.. this is the primary resolution used by the Healthservice.