Tuesday, March 08, 2011

Merging multiple EMC NAR files

For my EMC SAN I have a script that exports the performance data (NAR files) to a folder every hour. This allows me to easily review the files at a later time and even import them into Excel or other tools.
The problem comes when attempting to merge a large number of files for reviewing a long time period (several weeks or months). The EMC Navisphere UI has a function to merge two NAR files, but with hundreds or thousands of files, this is impossible.
I did some research and found that the Navisphere CLI can combine NAR files from command line. The basic syntax is as follows:
NaviSECCli.exe analyzer -archivemerge -data file1.nar file2.nar -out file3.nar
Using this command, I created a script that used the following logic:
  1. Rename the first file as Temp.Nar
  2. Begin loop though all the files
  3. Merge the current file with Temp.Nar, creating Output.Nar
  4. Rename Output.Nar to Temp.Nar
  5. Continue with loop
The only problem I found with this merge process is that it becomes slower as the files increase in size. So if you continually add to a single file (like I did), it will start out fast and gradually come to a crawl. This is fine for merging a few dozen files, but not the thousands that I was running into.
I reworked my script and came up with the following logic:
  1. Create an empty variable named tempFile
  2. Begin loop though all the files
  3. If tempFile is empty
    1. Assign it the current file
    2. Continue with loop
  4. If tempFile isn't empty
    1. Merge the current file and tempFile, creating a file with a random name
    2. Continue with loop
  5. Repeat loop until only 1 file remains
While this may seem slower at first glance, its faster because it works with small files more frequently. Only after multiple passes does the merge process begin working with larger files that can take several minutes to complete.

naviPath="C:\Program Files (x86)\EMC\Navisphere CLI\"

SET oShell = WScript.CreateObject("Wscript.Shell")
SET objArgs = Wscript.Arguments
folderspec = objArgs(0)

Set fso = CreateObject("Scripting.FileSystemObject")
DO WHILE fso.GetFolder(folderspec).Files.Count >1 
    MergeNAR(folderspec)
Loop


SUB MergeNAR(folderspec)
    dim tempFile
    tempFile=NULL
    Set f = fso.GetFolder(folderspec)
    Set fc = f.Files

    For Each f1 in fc
        IF IsNull(tempFile) THEN
            SET tempFile = f1
        ELSE
            strCmd = "'" & naviPath & "NaviSECCli.exe' analyzer -archivemerge -data '" & folderspec & "\" & f1.name & "' '" & folderspec & "\" & tempFile.Name & "' -out '" & folderspec & "\" & getTimeStamp & ".nar'"
            strCmd =  Replace(strCmd,"'","""")
            wscript.echo strCmd
            oShell.Run strCmd, 1, true
            f1.Delete
            tempFile.Delete
            tempFile = NULL

        END IF
    Next
END SUB


Function getTimeStamp()
        Dim intSeconds, intMilliseconds, strMilliseconds, intDatePart, intTimePart
        
        intSeconds = (Hour(Now) * 3600) + (Minute(Now) * 60) + Second(Now)
        intMilliseconds = Timer() - intSeconds
        intMilliseconds = Fix(intMilliseconds * 100)
        
        intDatePart = (Year(Now) * 10000) + (Month(Now) * 100) + Day(Now)
        intTimePart = (Hour(Now) * 1000000) + (Minute(Now) * 10000) + (Second(Now) * 100) & "." & intMilliseconds
        
        getTimeStamp = intDatePart & intTimePart 
End Function

4 comments:

ragoo said...

If you don't mind can you share the script to pull daily NVR files..if you don't want at least can share command to do that on clariion and logic..

I tried to create scripts for celerra perf stat but nothing working for me..if you know anything about celerra perf stat files..let me know too..

thanks in advance

Phil said...

Hi Radoo

The command is:

naviseccli -h SAN_IP_ADDRESS analyzer -archive -path PATH_TO_STORE_EXPORTED_NAR_FILES -all -o

Change SAN_IP_ADDRESS to the array's IP or hostname.
Change PATH_TO_STORE_EXPORTED_NAR_FILES to the path you what the file to be placed.

Once this is done I do another command to delete all the NAR files on the array to start afresh:

naviseccli -h SAN_IP_ADDRESS analyzer -archive -all -o -delete

I run these command on a weekly bases storing the files in a folder for each week - (using the Friday date - weekend)

Jan said...

can be missing something: is this a powershell script? or a wscript/vbs script???

Anonymous said...

What is the maximum size of the NAR file? I was combining files with your script and eventually it started creating 1 KB files. I definitely had NAR files over 250 MB but I think I hit a threshold.