Setting up and Troubleshooting interop or Universal connector with HPOVO

Article
07/03/2012

In this blog, i have not just concentrated on resolving the issue but also to give you a fair idea on where to look if the issue occurs. Basically the locations of logs and some possible solutions of it.

For setting up the connector, check the link

After we confirmed that our connector service and Provider are in place. Good to see that.

Look into the log scxcimd.log at /var/opt/Microsoft/scx/log

I see below alarming entries. So we are not able to validate the wsman account.

06/28/2012-02:13:03 SEVERE cimserver: Provider is not a CMPI style provider. library /opt/microsoft/scx/lib/providers/libSCXInteropProviderModule.so, error ProviderLoadFailure: (/opt/microsoft/scx/lib/providers/libSCXInteropProviderModule.so:SCX_ConnectorProvider):Cannot load library, error: /opt/microsoft/scx/lib/providers/libSCXInteropProviderModule.so: wrong ELF class: ELFCLASS64

You can also see this is scinteropd.log

<SOAP-ENV:Fault xmlns:SOAP-ENV="https://www.w3.org/2003/05/soap-envelope"><SOAP-ENV:Code><SOAP-ENV:Value>SOAP-ENV:Receiver</SOAP-ENV:Value><SOAP-ENV:Subcode><SOAP-ENV:Value>wsman:InternalError</SOAP-ENV:Value></SOAP-ENV:Subcode></SOAP-ENV:Code><SOAP-ENV:Reason><SOAP-ENV:Text xml:lang="en-US">CIM_ERR_FAILED: ProviderLoadFailure: (/opt/microsoft/scx/lib/providers/libSCXInteropProviderModule.so:SCX_ConnectorProvider):Cannot load library, error: /opt/microsoft/scx/lib/providers/libSCXInteropProviderModule.so: wrong ELF class: ELFCLASS64</SOAP-ENV:Text></SOAP-ENV:Reason></SOAP-ENV:Fault>

DEBUG Microsoft.SystemCenter.Interop.NoProviderAvailableException: Could not connect to any providers to retrieve update events

 Event Type: Error
Event Source: OpsMgr Connector for HP OpenView Operations
Event Category: None
Event ID: 0
Date: 6/26/2012
Time: 10:07:49 PM
User: <MS Name>

Description:
Service cannot be started. System.ApplicationException: Could not find configuration for 'OpsMgr Connector for HP OpenView Operations'
 at Microsoft.SystemCenter.Interop.Configuration.Config.Load(String key)
 at Microsoft.SystemCenter.Interop.Configuration.Config..ctor(String configName)
 at Microsoft.SystemCenter.Interop.ConnectorService.OnStart(String[] args)
 at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

For more information, see Help and Support Center at https://go.microsoft.com/fwlink/events.asp.

It only means that some of the library files are not loaded. It doesn’t look strange to me. Just to let you know, this happens when we have x64 bit agent installed on x86 machine and vice versa. And that was found true. We definitely did that.

To check what bit (32 or 64) agent installed, run below command,

file /opt/microsoft/scx/bin/scxcimserver

To check bit information if provider, run below

File /opt/microsoft/scx/lib/providers/libSCXInteropProviderModule.so

Both output should match to come to conclusion.

When you enable the verbose logging on the UNIX machine with below command.

scxadmin -log-set all verbose

We were able to see below messages in cimserver.trc

1340870984s-99035us: SSL [30728:4148087648:TLS.cpp:196]: In SSLSocket::incompleteSecureReadOccurred : err = 5 error:00000000:lib(0):func(0):reason(0)

1340870984s-99044us: Http [30728:4148087648:HTTPConnection.cpp:2075]: Total bytesRead = 0; Bytes read this iteration = 0

1340870984s-99054us: XmlIO [30728:4148087648:HTTPConnection.cpp:2107]:

Possibility:

1) The version of OpenSSL on the RHL 5.8 box is slightly incompatible

2) Network issue Or the certificate, however with the current data, I don’t see it in log.

So next step is, install x86 agent on x86 hardware and so on. This should fix the issue. As soon as done, we didn’t see any such entries. That was good.

So here is the steps of install on the provider part.

1) Have scx-1.0.4-265.rhel.5.x64.rpm installed which is a provider component, (If x64)

2) Next install MSFTInteropOvo.xxxx (Only if we are going for interop connector)

Note: Provider component 1^st, Connector on MS 2^nd.

We did it,

We still can’t validate the wsman credential. OK no problem, let’s check if this command works

"winrm enumerate https://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_Processor__cimnamespace=root/scx -auth:basic -remote: https://<Linux server>:1270 -username:<Username> -password:<Password> -encoding:utf-8 -skipCAcheck -skipCNcheck"

And it fails with “Access denied” error. Though I have used root account.

Immediately I think of this link https://support.microsoft.com/kb/2653882 but this is something related to Solaris, here I have Linux 5.8.

I check below file if this exists, may be something wrong with PAM.

# cat /etc/pam.d/scx

cat: /etc/pam.d/scx: No such file or directory

So the PAM doesn’t exist at all here. This is the file, that is created while the install goes on, there should be scx file that should be created automatically. SO we can make use of sshd. So try this

cd /etc/pam.d
	cp sshd scx

So SCX file should look like this, when you run command “cat scx”

Now try running your winrm command from above. And it goes success!!!

But still we are not able to validate the WSMAN credentials in the connector configuration wizard.

I can definitely think of checking my certificate, so I run winRM command without –skipCNcheck and –skipCAcheck, it works fine, so its not a certificate issue. If there is a failure, you can take a look at

https://support.microsoft.com/kb/2651766

in the application event log, I see below event

 Event Type: Error
Event Source: OpsMgr Connector for HP OpenView Operations
Event Category: None
Event ID: 1500
Date: 6/28/2012
Time: 12:37:51 PM
User: N/A
Computer: <MS Name>

Description:
2012-06-28 12:37:51.511 [Microsoft.SystemCenter.Interop.Framework.Modules.WSManModule] ERROR No providers could be contacted. Could not get update events. Cannot create WSManConfig for any providers

We couldn’t make the wsman validation work, so I decided to manually import the certificate. By using below command by going into the directory

%Program Files%\System center Operations Manager 2007 R2 Connectors\opsMgr Connector For HP OpenView Operations

"Scicert <Provider machine> <User account> <Password> “OpsMgr Connector for HP Openview Operations” silent"

This popped some error

“Connector certificate import operation returned an error code. Certificate installation was not complete. Please refer to connector Deployment Guide for further details”

Then I found, I had incorrect Operations manager server in the server names. So make sure that you have correct server names here. So this issue can happen when you have incorrect names.

As soon as I corrected the operations manager server name, and enter on “Configure” it returned success!!.

In case you are seeing below error in the event log.

OK, now i was able to get my wsman credential validated.

At some point in the event, i still see below event of access denied.

 Event Type: Error
Event Source: OpsMgr Connector for HP OpenView Operations
Event Category: None
Event ID: 1151
Date: 6/28/2012
Time: 12:12:42 PM
User: N/A
Computer: Machine name

 Description:

 2012-06-28 12:12:42.779 [Microsoft.SystemCenter.Interop.Framework.Modules.OpsMgrModule] ERROR Not authorized to connect to server: <Machine name>. The user <User account> does not have sufficient permission to perform the operation.

Remember going through Kevin’s blog for checking the rights of the account. https://blogs.technet.com/b/kevinholman/archive/2010/09/23/installing-the-opsmgr-r2-universal-connector.aspx

That means, this is the account that runs the connector service. And needs to have below permission.

After all, I got this after successful validation

Then I tried forwarding the alerts to the connector and it was success with ticket ID field populated.

Note:-

Another one I thought of bringing to notice.

While Integration between SCOM 2007 and HP Operations Manager (OM) on Red Hat Linux.

In the log /var/opt/microsoft/scx/log/scxcimd.log:

06/22/2012-15:52:46 INFO cimserver: Authentication failed for user=<opc_adm>

06/22/2012-15:53:16 INFO cimserver: Authentication failed for user=<opc_adm>

Remember, this account is an HPOVO administrator by default. So you may need to check if below steps are correct.

1) Verify is password entered is correct

2) Verify if you are able to log into HPOVO with opc_adm account

If you are not sure about the password and wondering what password you have entered and how to re-enter the password for account “ops_adm” ,

To know what password you have entered, try below

vi /etc/init.d/scx-scinteropd

Change the log level to 51

-D51 (instead of -D1 or whatever it is now)

that will verify the password is being decrypted successfully

Once this is done, restart the provider service. Then check the log again. You will see below information along with the password entered. Check scinteropd.log

05-29-2012 17:32:46 - INFO: Debug Level = '51'

05-29-2012 17:32:46 - INFO: Instance name = 'SciInst'.

05-29-2012 17:32:46 - INFO: Operator name = 'opc_adm'.

05-29-2012 17:32:46 - INFO: DELETE FROM LOG - Password = <Password>

05-29-2012 17:32:46 - ERROR: Cannot connect to the MSI!

05-29-2012 17:32:46 - INFO: opc_connect() - Invalid password!

OK, here is how you can update your password of ops_adm. Run below

“/etc/opt/microsoft/scx/conf/scicrypt.bin <New or correct Password>” --> ENTER

This way, you have updated or changed the password. And access denied or password issues should be resolved,

If this doesn’t, There are two chances,

1) ops_adm is not able to connect to HPOVO API called “opc_connect()”

2) there are chances that MSI "Message Stream Interface" - MSI has been corrupt

At this point, you may need to engage HPOVO expert to find. But you know where you are with the issue.

Setting up and Troubleshooting interop or Universal connector with HPOVO

Additional resources