Friday, March 11, 2011

Sending email from syslog events

Historically, I have been monitoring my F5 devices using syslog and Zenoss. Syslog on the F5 devices forwards to my Zenoss system, which then parses the alerts by severity, source, and regex string comparison (node up/node down).

My Zenoss system died, so I started looking for other methods of performing the same tasks. Initially I looked at Nagios because we are using it to monitor our apps, sadly there doesnt appear to be native syslog monitoring, which means that solution is extra difficult.

I found the site, http://www.johnandcailin.com/blog/john/how-setup-real-time-email-notification-critical-syslog-events, that described how to make syslog email on its own. Below are the key steps needed to make this work


configuring syslog to write to a named-pipe

first, create a named-pipe for critical messages, for example:
# mkdir /etc/syslog.pipes
# mknod /etc/syslog.pipes/criticalMessages p
# chmod 600 /etc/syslog.pipes/criticalMessages
next, configure syslog to log all critical messages written to the local0 facility to this pipe. add the following statement to your syslog.conf file.
local0.crit   |/etc/syslog.pipes/criticalMessages

sending out messages

the final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i've included an example below, let's call it /usr/bin/syslogMailer:
#!/bin/bash

# syslogMailer: a script to read stdin and turn each line into an alert
# email typically this is used to read a named-pipe written to by syslog
#
#   example usage: syslogMailer < /etc/syslog.pipes/criticalMessages
#

alertRecipient="fireman@example.com"      # the mail recipient for alerts
TMOUT=1                                   # don't wait > 1 second for input

# process each line of input and produce an alert email
while read line
do
   # remove any repeated messages
   echo ${line} | grep "message repeated" > /dev/null 2>&1
   if test $? -eq 1
   then
      # send the alert
      echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
   fi
done

cron

# m h  dom mon dow   command
0-59/5 * * * * /usr/bin/syslogMailer < /etc/syslog.pipes/criticalMessages > /dev/null 2>&1

No comments: