Friday, March 11, 2011

Sending email from syslog events

Historically, I have been monitoring my F5 devices using syslog and Zenoss. Syslog on the F5 devices forwards to my Zenoss system, which then parses the alerts by severity, source, and regex string comparison (node up/node down).

My Zenoss system died, so I started looking for other methods of performing the same tasks. Initially I looked at Nagios because we are using it to monitor our apps, sadly there doesnt appear to be native syslog monitoring, which means that solution is extra difficult.

I found the site, http://www.johnandcailin.com/blog/john/how-setup-real-time-email-notification-critical-syslog-events, that described how to make syslog email on its own. Below are the key steps needed to make this work


configuring syslog to write to a named-pipe

first, create a named-pipe for critical messages, for example:
# mkdir /etc/syslog.pipes
# mknod /etc/syslog.pipes/criticalMessages p
# chmod 600 /etc/syslog.pipes/criticalMessages
next, configure syslog to log all critical messages written to the local0 facility to this pipe. add the following statement to your syslog.conf file.
local0.crit   |/etc/syslog.pipes/criticalMessages

sending out messages

the final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i've included an example below, let's call it /usr/bin/syslogMailer:
#!/bin/bash

# syslogMailer: a script to read stdin and turn each line into an alert
# email typically this is used to read a named-pipe written to by syslog
#
#   example usage: syslogMailer < /etc/syslog.pipes/criticalMessages
#

alertRecipient="fireman@example.com"      # the mail recipient for alerts
TMOUT=1                                   # don't wait > 1 second for input

# process each line of input and produce an alert email
while read line
do
   # remove any repeated messages
   echo ${line} | grep "message repeated" > /dev/null 2>&1
   if test $? -eq 1
   then
      # send the alert
      echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
   fi
done

cron

# m h  dom mon dow   command
0-59/5 * * * * /usr/bin/syslogMailer < /etc/syslog.pipes/criticalMessages > /dev/null 2>&1

Thursday, March 10, 2011

Mount LVM volumes imported from a remote system

Assuming you import an LVM disk from another system, you must rescan the LVM  in order to mount it.

vgchange -ay

Once completed, you can use normal mount commands to mount the disk

Tuesday, March 08, 2011

Merging multiple EMC NAR files

For my EMC SAN I have a script that exports the performance data (NAR files) to a folder every hour. This allows me to easily review the files at a later time and even import them into Excel or other tools.
The problem comes when attempting to merge a large number of files for reviewing a long time period (several weeks or months). The EMC Navisphere UI has a function to merge two NAR files, but with hundreds or thousands of files, this is impossible.
I did some research and found that the Navisphere CLI can combine NAR files from command line. The basic syntax is as follows:
NaviSECCli.exe analyzer -archivemerge -data file1.nar file2.nar -out file3.nar
Using this command, I created a script that used the following logic:
  1. Rename the first file as Temp.Nar
  2. Begin loop though all the files
  3. Merge the current file with Temp.Nar, creating Output.Nar
  4. Rename Output.Nar to Temp.Nar
  5. Continue with loop
The only problem I found with this merge process is that it becomes slower as the files increase in size. So if you continually add to a single file (like I did), it will start out fast and gradually come to a crawl. This is fine for merging a few dozen files, but not the thousands that I was running into.
I reworked my script and came up with the following logic:
  1. Create an empty variable named tempFile
  2. Begin loop though all the files
  3. If tempFile is empty
    1. Assign it the current file
    2. Continue with loop
  4. If tempFile isn't empty
    1. Merge the current file and tempFile, creating a file with a random name
    2. Continue with loop
  5. Repeat loop until only 1 file remains
While this may seem slower at first glance, its faster because it works with small files more frequently. Only after multiple passes does the merge process begin working with larger files that can take several minutes to complete.

naviPath="C:\Program Files (x86)\EMC\Navisphere CLI\"

SET oShell = WScript.CreateObject("Wscript.Shell")
SET objArgs = Wscript.Arguments
folderspec = objArgs(0)

Set fso = CreateObject("Scripting.FileSystemObject")
DO WHILE fso.GetFolder(folderspec).Files.Count >1 
    MergeNAR(folderspec)
Loop


SUB MergeNAR(folderspec)
    dim tempFile
    tempFile=NULL
    Set f = fso.GetFolder(folderspec)
    Set fc = f.Files

    For Each f1 in fc
        IF IsNull(tempFile) THEN
            SET tempFile = f1
        ELSE
            strCmd = "'" & naviPath & "NaviSECCli.exe' analyzer -archivemerge -data '" & folderspec & "\" & f1.name & "' '" & folderspec & "\" & tempFile.Name & "' -out '" & folderspec & "\" & getTimeStamp & ".nar'"
            strCmd =  Replace(strCmd,"'","""")
            wscript.echo strCmd
            oShell.Run strCmd, 1, true
            f1.Delete
            tempFile.Delete
            tempFile = NULL

        END IF
    Next
END SUB


Function getTimeStamp()
        Dim intSeconds, intMilliseconds, strMilliseconds, intDatePart, intTimePart
        
        intSeconds = (Hour(Now) * 3600) + (Minute(Now) * 60) + Second(Now)
        intMilliseconds = Timer() - intSeconds
        intMilliseconds = Fix(intMilliseconds * 100)
        
        intDatePart = (Year(Now) * 10000) + (Month(Now) * 100) + Day(Now)
        intTimePart = (Hour(Now) * 1000000) + (Minute(Now) * 10000) + (Second(Now) * 100) & "." & intMilliseconds
        
        getTimeStamp = intDatePart & intTimePart 
End Function

Changing default number of machines users can add to a domain

I ran into an issue at work today where a specific user couldn't add a machine to the domain, but had been able to previously. A little research and I realized that by default, each user can only join up to 10 machines to the domain. I am not sure why there would be a limit like this (I would expect 0 or infinite), but here is how to increase the number (taken from http://blogs.technet.com/b/jhoward/archive/2005/04/18/403817.aspx)
  1. Start ADSI Edit (start/run/adsiedit.msc)
  2. Expand out the Domain node, right click on DC=,DC=com and select properties
  3. Scan down to ms-DS-MachineAccountQuota
  4. Modify the value as appropriate, or clear the value to remove the limit entirely.

Friday, March 04, 2011

Cool windows screensavers

I am currently running Ubuntu as my primary desktop and have enjoyed it greatly, I especially like the variety and quality of screensavers available.

However, due to work requirements, I am switching back to a Windows desktop (SSTP VPN, RDP Gateway, etc...) and am really going to miss being entertained by the screen savers when I go get some coffee.

I found http://www.reallyslick.com/index.html which appears to have ported several of the savers to windows. At first I was hesitant (can you say virus), but ran them through several scanners and they seem to be working fine. Not all of the savers are included (bouncing cow), but its enough of a variety to keep me happy for now.