Syslog collection in Operations Management Suite
Summary: Learn how to configure and use the OMS Agent for Linux for syslog collection.
Good morning everyone, Anurag Gupta from the OMS team here, and today I want to talk about syslog collection with Operations Management Suite.
One of the core features with Linux management in Operations Management Suite (OMS) is collection of Syslog events. Linux syslog events can range from kernel logs to application data and audit logs. In addition to Linux servers, many network devices, firewalls, and applications allow native syslog forwarding.
With the OMS Agent for Linux, you can bring all of these forwarded syslog messages into the powerful OMS Log Analytics platform to quickly search through meaningful logs or alert on certain syslog events.
This post gives an overview of syslog collection, steps to enable high-volume syslog collection, and useful scenarios with syslogs.
Overview
The OMS Agent for Linux, which is based on Fluentd, includes native support for syslog protocol messages. When the OMS Agent for Linux is installed, it automatically configures other syslog applications (rsyslog and syslog-ng) to write all warning and above events to its listener on port 25224.
If you are already forwarding events to rsyslog or syslog-ng on a single Linux server by simply installing the OMS Agent for Linux, all events that are warning or higher are automatically routed to the OMS service. If you want to enable additional log levels or facilitate collection, refer to Configuring syslog collection from the OMS portal .
Configuring the OMS Agent for Linux for direct forwarding
Additionally, instead of forwarding syslog messages to rsyslog or syslog-ng, syslog messages can be routed directly to the OMS Agent for Linux.
To enable direct collection, edit the configuration file located at /etc/opt/microsoft/omsagent/conf/omsagent.conf.
1. Add a new source in the configuration file to ingest the incoming syslog data.
a. Replace <TAGNAME> with a custom name for this new incoming data stream e.g. netdevice.
b. Replace the <UDP or TCP> with the protocol_type that’s required.
c. Change port settings and bind address as needed.
<source>
type syslog
port 25226
bind 0.0.0.0
tag oms.<TAGNAME>
protocol_type <UDP OR TCP>
</source>
2. Add a new filter in the configuration file to format the incoming syslog data.
a. Replace <TAGNAME> with the same custom name that you used in Step 1.
<filter oms.<TAGNAME>.**>
type filter_syslog
</filter>
3. Restart omsagent for new settings to take effect.
service omsagent restart or systemctl restart omsagent
Configuring the OMS Agent for Linux for high-volume syslog collection
By default, the OMS Agent for Linux can handle a throughput of 500 messages/second without change in the configuration. The following configuration changes throughput to collect 2000 messages/second.
Note: With high-volume syslog collection, we recommend the TCP protocol to avoid message loss.
1. Enable new OMS Syslog endpoint by using the previous instructions.
2. Edit the OMS Agent for Linux configuration file.
a. Add more output threads
b. Change buffer size
c. Reduce time for retries
<match oms.** docker.**>
type out_oms
log_level info
num_threads 10
buffer_chunk_limit 5m
buffer_type file
buffer_path /var/opt/microsoft/omsagent/state/out_oms*.buffer
buffer_queue_limit 30
flush_interval 20s
retry_limit 10
# Do not wait more than 5 minutes before sending again
max_retry_wait 300s
retry_wait 2s
</match>
Syslog scenarios
Alert on firewall blocks
Because many firewall systems allow native syslog forwarding, it’s simple to send these events to OMS through the OMS Agent for Linux. After these logs are in OMS, we can use the power of OMS Log Analytic alerts to mark when specific firewall events are detected and route these events to required owners via email or a Webhook.
As a sample case, if we are using the Universal Firewall on systems that are based on Debian, we can alert every time we receive a “UFW BLOCK” event. We can quickly search for interesting events in Log Analytics and then select the Alert option on the top bar.
For a full list of OMS Log Analytic alert features, check out the Enable Microsoft Operations Management Suite email alerts blog post.
Using Minify to detect patterns
Because syslog contains a multitude of types, the ability to read and gain insight about those logs in a scalable manner becomes a challenge. Minify, which is one of the latest features of OMS Log Analytics, can help you quickly find meaningful messages and cluster syslog events.
In Search, click the Minify tab to start to bundle messages and give you the ability to quickly see commonality among events.
For a detailed view of Minify functionality, see the Introducing Minify! blog post.
That is all I have for you today.
Get a free Microsoft Operations Management Suite (#MSOMS) subscription so that you can test the new alerting features. You can also get a free subscription for Microsoft Azure.
I invite you to follow me on Twitter and the Microsoft OMS Facebook site. If you want to learn more about Windows PowerShell, visit the Hey, Scripting Guy Blog. If you have any questions, send email to me at scripter@microsoft.com. I wish you a wonderful day, and I’ll see you tomorrow.
Anurag Gupta
Microsoft Operations Management Team