System Center Operations Manager 1801 has enhanced log file monitoring capabilities for Linux Servers.
- Operations Manager now supports Fluentd, an Open source Data collector.
- Customers can also leverage Fluentd capabilities and plugins published by the Fluentd community to get enhanced customizable log file monitoring.
- The existing OMI based monitoring for currently supported Linux workloads will continue to work as it is today.
With this release we have added support for the following log file monitoring capabilities
- Support for wildcard characters in log file name and path.
- Support for new match patterns for customizable log search like simple match, exclusive match, correlated match, repeated correlation and exclusive correlation. We have released 6 new filter plugins for customizable log search.
- Support for generic Fluentd plugins published by the fluentd community. System Center Operations Manager 1801 would include a convertor plugin which would convert the fluentd data from generic plugins to the format specific for SCOM log file monitoring.
Below are few architectural changes in the SCOM Management server and the SCOM Linux agent to support Fluentd.
The new Linux SCOM agent would include a Fluentd agent (as shown in the above picture (1)).
Users would define the log file names, match pattern and the event to be generated on pattern match along with the event description in the Fluentd Configuration file.
On match of a log record, Fluentd would send the event to the System Center Operations Manager External Datasource service on the SCOM Management Server / Gateway (2).This is a Windows REST based service which would receive the event and send it to a dedicated custom Event log channel Microsoft.Linux.OMED.EventDataSource (3).
User would need to import a management pack (4) which would look for events in this custom event channel and generate alerts accordingly
On Linux Server:
On SCOM Management Server:
User needs to follow the below steps on the Management Server
User would need to import the latest Linux Management pack (shipped with the SCOM 1801 binaries) and install the new SCOM agent on the Linux Servers.
Users can install the agent either manually or through discovery wizard (recommended). For detailed steps, refer here.
Author Fluentd configuration file and place it on the Linux Servers
Customers need to author a Fluentd configuration file and can use any of the existing enterprise tools like Chef/Puppet to place the configuration file to the Linux server.
Recommended practice is to copy the configuration into /etc/opt/microsoft/omsagent/scom/conf/omsagent.d directory on all Linux servers and include the configuration file directory as @include directive in the master configuration file /etc/opt/microsoft/omsagent/scom/conf/omsagent.conf
The Fluentd configuration file is where the user should define the input, output and the behavior (match processing) of Fluentd. This is done by defining the following in the configuration file:
Fluentd’s input sources are defined in the source directive using desired input plugins. Users would need to define the log file names along with the file path here in this directive. Wild card characters are support both in file name and path.
Filter directive is the chained processing pipeline. Users would need to define the match pattern and the events that are to be generated on a match here in this section. We have released the following filter plugins with this release
Users define the output processing in Match directive. We have released “out_scom” match plugin which would send the events generated by Fluentd to the System Center Operations Manager External Datasource service on the SCOM Management Server/Gateway.
For more detailed instructions on how to author a Fluentd configuration file, refer here.
On SCOM Management server: Import Management pack and enable OMED Service
On Management Server User needs to do the following:
1) Start OMED service (refer here).
2) Import Management pack for log file monitoring.
User can import the sample Management pack (reference here ), save this as an xml file and import it in SCOM console. This Management pack has a rule that looks for all events from the new data source Microsoft.Linux.OMED.EventDataSource and generates alerts accordingly. The Alert severity and priority are set in the management pack. The Alert description is obtained from the event description which would be defined by the user in the Fluentd configuration file.
If users are interested to generate alerts only for specific events generated, they could author their own custom management pack using VSAE.
User would like to monitor the following scenarios
1) Apache http server URL monitoring
Scenario: Monitor a web URL hosted on Apache http server and generate alerts on SCOM Management server if the URL has any issues.
Log to be monitored: User monitors Apache http server access.log for error code. If the log receives any code other than 200 (success code) an event will be sent to SCOM Management Server.
2) Authentication failure
Scenario: If a user tries to access a server more than 5 times with an incorrect password, an alert would be sent to the SCOM server alerting an unauthorized user trying to intrude.
Log to be monitored: User monitors Linux Server auth.log for authentication failure error messages. If the messages exceeds 5 times in 10 seconds and event will be sent to SCOM Management server.
Sample Configuration File:
The OMEDService on SCOM Management server would receive an event on match of a log record along with the log record context. User would need to import a management pack on SCOM server which would generate alert when there is an event received from Linux Server.
Events on the SCOM Management Server:
Generated Alert on the Management Server:
The Alert context will contain the log record which will have more details on the error code received while trying to access the URL.
Other Sample User Scenarios:
For more detailed steps look at the online documentation.
We’d love to hear your feedback on this new feature. Feel free to send your feedback to firstname.lastname@example.org.