Demand Technology Software
Cycle End Processing Explained.
This discussion of Cycle End processing also considers the following subtopics:
Using Windows Script Host (WSH) scripts
At the end of each collection cycle, the Collection service closes the
current collection file to make it available for processing. The end of a cycle
normally is used to initiate processing of the collection files in SAS or MXG.
Cycle end processing normally consists of scheduling a process to consolidate
Performance SeNTry data files at a central location. This section discusses
automating cycle end processing.
The Collection service stores data collection logs in the \Data subdirectory under the NTSMF24 program directory by default. The location of the data directory can also be changed using the SeNTry Administration program by modifying the Directory where SMF files are stored start-up parameter.
Hierarchical file management sets up Current, Previous and Archive Folders
under the \Data directory and manages collection files according to the criteria
you specify. The hierarchical file management option is specifically designed to
simplify your cycle end procedures. By default, old collection files are moved
immediately from the Current Folder to the Previous Folder. After the number of
days you specify, they are then migrated to the Archive Folder. Files are kept
in the Archive Folder for an additional period of time and then deleted
automatically according to your specifications. Because all automatic file
management is performed prior to issuing the Cycle End Command, the cycle end
procedure you execute can be as simple as a COPY *.* command pointing to the
Previous Folder.
Unique collection data log file names are automatically generated of the form
systemname.yyyymmddhhmmCustomQualifier.smf
where yyyymmddhhmm is a date/time stamp identifying when the file was
created. During the collection cycle, the Collection service can lock the
current data file so that no other application can access it. Or the current
data file can be Shared so that it can be processed by other Windows NT/2000
programs or commands while collection is occurring and the file is open. The
collection file is closed at Cycle End and is available for processing.
The Collection service automatically launches the Cycle End Command you
specify after all file management actions have been completed. You can specify
commands directly or execute a .BAT file script, PERL script, or other command
processing program. By default, the Cycle End command is launched immediately.
Or you can specify that the Cycle End command be launched at a random time
within a defined processing window (up to 60 minutes in duration).
Processing multiple files.
If the Collection service was restarted (due to a re-boot, for example)
during the last collection cycle, there will be multiple .smf collection files
in the \data\Previous folder at the time the Cycle End Command you
specified is launched. Consequently, the Cycle End processing you perform should
allow for the fact that multiple collection files may be available for
processing. A wildcard in your filespec normally accomplishes this.
For ease of processing in MXG, you will normally consolidate collection files
representing many different machines into one file. Use the COPY command
available in Windows NT/2000 with the /B option to consolidate multiple
Performance SeNTry collection data files into one large file for processing:
COPY /B C:\NTSMF24\data\Previous\*.SMF C:\NTSMF\Daily\NTDaily.SMF
The /B option of the COPY command concatenates multiple files without terminating each file with a x’3F’ end of file character. In the above example the file named NTDaily.SMF is associated with the MXG NTSMF input file name, for example. To process this data file in MXG, you would code a SAS language FILENAME NTSMF specification that references C:\NTSMF\Daily\NTDaily.SMF if you are running SAS under Windows NT/2000.
If you are copying files to send to an MVS mainframe for processing using SAS, use a combination of /A and /B switches, as follows to prevent embedded EOFs in the concatenated file:
copy /a f1.smf+f2.smf+f3.smf bigfile /b
Using the /a will cause the EOF mark to be removed from each of the source files f1.smf through f3.smf as they are concatenated into the destination file bigfile. the /b following the destination file bigfile ensures that an EOF mark is not appended to bigfile and subsequently transmitted to MVS.
More generally, to create a concatenated file usable under MVS:
copy /a *.smf bigfile /b
Using ftp
If you need to send daily .smf files to a non-Windows host machine for
processing or the machine you need to transfer the files to is separated by a
firewall, you may need to use the ftp utility to copy the files. Here is an
example of a simple .bat command file that utilizes ftp:
COPY /B C:\NTSMF24\Data\Previous\*.SMF C:\NTSMF24\Data\Consolidated\Consolidated.SMF
ftp -v -i -s:Transfer.ftp
exit
The following example script connects to a remote server, uses a valid Username and Password to log into the remote system. Once logged unto the remote system, the program changes directory (CD) to the remote directory C:\Consolidated and sends the file from the originating folder C:\NTSMF24\Data\Consolidated\Consolidated.smf to the remote file Consolidated.smf on the remote server. Both origination and remote servers have to have FTP enabled, with common Usernames and Passwords defined. In the script, replace the keywords Remote, Username, and Password with parameters that are valid for your environment.
open Remote
Username
Password
cd C:\Consolidated
send C:\NTSMF24\Data\Consolidated\Consolidated.SMF Consolidated.SMF
Bye
You can also using ftp wild cards to transfer multiple files in a single command execution stream. The easiest way to do this to create an ftp script that uses the mput subcommand to send multiple .smf files contained in the \Previous folder.
CD C:\NTSMF24\DATA\PREVIOUS\
ftp -v -i -s:C:\FTPROCESS\transfersmf.txt
exit
where transfersmf.txt is a text file containing the following ftp script:
Rem Transfersmf.ftp sends multiple files to a remote central server
open Remote
Username
Password
prompt
mput *
Bye
The prompt subcommand turns off ftp interactive mode. The * wildcard filespec means that all data files located in the \Previous folder will be transferred.
Invoking the Summarization utility
The Summarization utility is designed to run at Cycle End, although it can also execute in a standalone environment. Letting the Summarization utility perform aggregation of the detailed collection files, you can distribute a good deal of the load involved in post-processing .smf files across your network.
In this example, the Summary utility is invoked in the Cycle End command script after a step that consolidates multiple detail files, prior to invoking ftp to transfer the .sum.smf summarized file to a consolidation folder on a remote server:
copy /b C:\NTSMFv24\Data\Previous\*.SMF C:\NTSMFv24\Data\Previous\Allprevious.smf
rem Next run summarization rtn to summarize in 60 minute intervals
cd ..
cd ..
dmsumsmf.exe -fAllprevious.smf -s60
rem Next execute FTP transfer bat file to upload file
ftp -v -i -s:Transfer.ftp
exit
where Transfer.ftp consists of the following script:
open 10.0.0.7
ftpguest
password
cd transfersmf
send
C:\NTSMFv24\Data\previous\Allprevious.sum.SMF
Allprevious.sum.SMF
Bye
Historical files
You can also take advantage of a roll-up feature in the Summarization utility to create and maintain historical data files that can streamline your reporting procedures. To use the roll-up feature, concatenate the current data file with the previous historical data file. When you invoke the Summary Utility on the combined file, specify the number of consecutive days worth of data that you want to keep. Any data older than the number of days you specify will be dropped from the summarized output file. Using the Summarization utility, you can create and maintain as many historical files as you want, as the example below illustrates. The sample script below maintains a daily historical roll-up file that spans the last 14 days, a weekly roll-up that spans 26 weeks, and a monthly roll-up that spans the last two years.
At Cycle End, issue the following command:
summary-example1.bat @LastDataFile:p @StartupDir
where summary-example1.bat is coded as follows:
rem Summay utility example
rem Cycle End command: summary-example1.bat @LastDataFile:p @StartupDir
rem Maintains daily, weekly, and monthly history files
rem Daily roll-up file: keeps 14 days of history, summarized to 10 minute intervals
rem Weekly roll-up file: keeps 26 weeks of history, summarized to 1 hour intervals
rem Monthly roll-up file: keeps 24 months of history, summarized to 4 hour intervals
rem ******************************************************************
rem
rem Utilizes the following Directory structure:
rem <ntsmf root>\data\
rem - \Archive
rem - \Backup -- backup History files first
rem - \Current
rem - \History -- current History roll-up files
rem - \Previous
rem - \Work -- for temporary files
rem
rem summarize the contents of the \data\previous folder -- yesterday's smf data files
copy /B %1\*.smf %2\data\Work\%COMPUTERNAME%.combined.smf
%2\dmsumsmf.exe -f%2\data\Work\%COMPUTERNAME%.combined.smf -k2 -H"%2" -s10
rem Delete old backups
del del %2\data\Backup\*.old.sum.smf
rem Backup History files
ren %2\data\Backup\%COMPUTERNAME%.daily.sum.smf %2\data\Backup\%COMPUTERNAME%.daily.old.sum.smf
ren %2\data\Backup\%COMPUTERNAME%.weekly.sum.smf %2\data\Backup\%COMPUTERNAME%.weekly.old.sum.smf
ren %2\data\Backup\%COMPUTERNAME%.monthly.sum.smf %2\data\Backup\%COMPUTERNAME%.monthly.old.sum.smf
copy %2\data\History\*.smf %2\data\Backup\
rem
rem Keep 14 days worth of Daily data
rem
rem Step 1: move the Daily summary file to \Work
move %2\data\History\%COMPUTERNAME%.daily.history.sum.smf %2\data\Work\
rem Step 2: Concatenate yesterday's data with the current Daily history roll-up
copy /B %2\data\Work\%COMPUTERNAME%.combined.sum.smf+%2\data\Work\%COMPUTERNAME%.daily.sum.smf %2\data\History\%COMPUTERNAME%.daily.smf
rem Step 3: Run the Summarization utility on the combined file
%2\dmsumsmf.exe -f%2\data\History\%COMPUTERNAME%.daily.smf -k14 -H%2 -s10
rem
rem Keep 26 weeks worth of Weekly data
rem
move %2\data\History\%COMPUTERNAME%.weekly.sum.smf %2\data\Work\
copy /B %2\data\Work\%COMPUTERNAME%.combined.sum.smf+%2\data\Work\%COMPUTERNAME%.weekly.sum.smf %2\data\History\%COMPUTERNAME%.weekly.smf
%2\dmsumsmf.exe -f%2\data\History\%COMPUTERNAME%.weekly.smf -k182 -H%2 -s60
rem
rem Keep 24 months worth of Monthly data
rem
move %2\data\History\%COMPUTERNAME%.monthly.sum.smf %2\data\Work\
copy /B %2\data\Work\%COMPUTERNAME%.combined.sum.smf+%2\data\Work\%COMPUTERNAME%.monthly.sum.smf %2\data\History\%COMPUTERNAME%.monthly.smf
%2\dmsumsmf.exe -f%2\data\History\%COMPUTERNAME%.monthly.smf -k730 -H%2 -s240
rem
rem Clean up Work & History folders
del %2\data\Work\*.smf
del %2\data\History\%COMPUTERNAME%.daily.smf
del %2\data\History\%COMPUTERNAME%.weekly.smf
del %2\data\History\%COMPUTERNAME%.monthly.smf
Using WSH scripts
Windows Script Host (WSH) scripts written in either VBscript or Javascript provide a flexible alternative to shell scripts. You can specify that a WSH script be launched at Cycle End. Launching WSH scripts at Cycle End requires the 2.4.7.3 version of the collection agent.
The Script Center on the Microsoft TechNet web site is an excellent resource for anyone that wants to learn about WSH script basics. If you have access to the Microsoft Developer Network (MSDN) web, you can access many additional resources for developing WSH scripts.
WSH technology is built into Windows 2000 and above, so there is no extra software you need to start developing WSH scripts. Utilizing built-in scripting objects like Wscript, WshShell, WshEnvironment, and WshArguments, your script can execute other programs, access environment variables and retrieve script arguments. A file system scripting object allows your script to manipulate folders and files. Finally, using WMI, your script can access any of the configuration information that is accessible in the vast WMI repository.
WARNING: When you specify the execution of a WSH script at Cycle End, make sure you specify the //B runtime parameter when you invoke wscript.exe or cscript.exe. The //B runtime parameter runs your script in batch mode, suppressing all screen output and keyboard interaction. If you do not specify the //B runtime parameter and your WSH script issues a Wscript.Echo statement that writes to the display or initiates any other interaction with the screen, your script will hang.
For example, specify the following Cycle End command in your DCS parameters:
Wscript.exe //B "@StartupDir\CycleEndscript1.vbs" /datapath:@LastDataFile:p /rootdir:@StartupDir
For tips on configuring the collection service in order to debug a Cycle End WSH script, see the section entitled Debugging WSH scripts below.
The following VBscript example illustrates the functions typically performed in a Cycle End script using WSH. It performs the following processing steps:
-
It performs some string manipulation of today's date to create a unique file name.
-
It validates the arguments /datapath and /rootdir arguments that are passed to it from the command line invocation.
-
It calls the NtDACmd utility to update the NTSMF PDB Sql Server database.
-
It then uses the Copy command with the /B option to consolidate multiple files into a single daily file, embedding the current date into the consolidated file's name. (The file system scripting object has no comparable function.)
-
It calls the DmSumsmf utility to create a summarized version of the consolidated daily data file.
-
Finally, it copies the consolidated daily data file somewhere -- Step 6 is the section of the Cycle End script that requires customization for your environment.
VBscript comment statements -- any statement that starts with a ' character -- are shown in an alternate color in this listing.
' VBScript source code
' Creates consolidated daily .smf file in /Previous folder
' Creates summarized daily .smf file from consolidated file
' Calls NtDACmd to initiate an NTSMF PDB database update
' Finally, copies the summarized file to a designated location
'
' The script accepts two parameters:
' /datapath -- the fully qualified path to the \data\Previous\ folder
' where yesterday's .smf collection files are located
' /rootdir -- the fully qualified path to the NTSMF root folder
' where the DmSumsmf and NtDACmd utilities are located
' Sample invokation at CycleEnd:
'
' Wscript.exe //B "@StartupDir\CycleEndscript1.vbs" /datapath:@LastDataFile:p
' /rootdir:@StartupDir
'Step 1 -- Create a unique file name
Dim CreatedTodayDate
Dim CreatedTodayDateAsString
Dim EditDate 'mmddyy format from standard Date function in VBscript
CreateTodayDate = Date
CreatedTodayDateAsString = CStr(CreateTodayDate)
'Wscript.Echo "Today's Date String is " &
CreatedTodayDateAsString
dim datestrlen
datestrlen = Len(CreatedTodayDateAsString)
dim loc
loc = InStr(CreatedTodayDateAsString,"/")
EditDate = Left(CreatedTodayDateAsString,loc-1)
if len(EditDate) = 1 then
EditDate = "0" & EditDate
End if
CreatedTodayDateAsString = Right(CreatedTodayDateAsString,datestrlen-loc)
datestrlen = datestrlen - loc
loc = InStr(CreatedTodayDateAsString,"/")
EditDate = EditDate & "-" & Left(CreatedTodayDateAsString,loc-1)
CreatedTodayDateAsString = Right(CreatedTodayDateAsString,datestrlen-loc)
datestrlen = datestrlen - loc
EditDate = EDitDate & "-" & CreatedTodayDateAsString
'Wscript.Echo "EditDate is " & EditDate
'Step 2 -- Validate the arguments passed to the script
Set WshShell = CreateObject("Wscript.Shell")
Set objEnv = WshShell.Environment("Process")
LogonServer = objENV("COMPUTERNAME")
Dim namedArgs
Dim PrevFolder, RootFolder
Dim objFSO1,objPreviousFolder,fc1,allthefilenames
Set objFSO1 = CreateObject("Scripting.FileSystemObject")
Set namedArgs = WScript.Arguments.Named
'WScript.Echo "datapath specified: " &
namedArgs.Item("datapath")
'WScript.Echo "rootdir specified: " & namedArgs.Item("rootdir")
PrevFolder = namedArgs.Item("datapath")
RootFolder = namedArgs.Item("rootdir")
'Change the next assignment statement to generate a
different name for consolidated data file
DailyFileName = PrevFolder & "\" & LogonServer & "." & _
"consolidated_daily_filefor" & "." & EditDate & ".smf"
'WScript.Echo DailyFileName
If objFSO1.FolderExists(PrevFolder) then
Else
'WScript.Echo "Invalid datapath
argument: " & PrevFolder
Return
End If
If objFSO1.FolderExists(RootFolder) then
Else
'WScript.Echo "Invalid root directory
argument: " & RootFolder
Return
End If
'Step 3 -- Call the NtDACmd to update the NTSMF PDB
database
command = RootFolder & "/" & "ntdacmd.exe -d " & PrevFolder & " -threads 2"
Dim execCommand2
Set execCommand2 = WshShell.Exec(command)
'Wscript.Echo execCommand2.StdOut.ReadAll
'Step 4 -- Execute the Copy coammnd to consolidate the
daily .smf data files
dim copyspec
copyspec = """" & PrevFolder & "\*.smf" & """"
'Wscript.Echo "Copy .smf Files in Previous Folder to: " &
DailyFileName
dim command
command = "cmd.exe /c " & """" & "copy /b " & copyspec & " " & DailyFileName &
""""
'Wscript.Echo "Copy command string: " & command
Dim execCommand
Set execCommand = WshShell.Exec(command)
'Wscript.Echo execCommand.StdOut.ReadAll
'Step 5 -- Execute the DmSumsmf utility to summarize the
daily .smf data files
command = RootFolder & "/" & "dmsumsmf -f" & DailyFileName & " -h" & RootFolder
'Wscript.Echo "Dmsumsmf command string: " & command
Dim execCommand1
Set execCommand1 = WshShell.Exec(command)
'Wscript.Echo execCommand.StdOut.ReadAll
'Step 6 -- Finally, copy the consolidated summary file
somewhere ===>
'Careful. Multiple sum.smf output files may be created
command = "cmd.exe /c " & """" & "copy " & PrevFolder & "\*.sum.smf
\\STPUTL04\NTSMF" & """"
'Wscript.Echo "Copy command string: " & command
'Dim execCommand3
'Set execCommand3 = WshShell.Exec(command)
'Wscript.Echo execCommand3.StdOut.ReadAll
Debugging WSH scripts. An old fashioned, but effective, way to debug WSH scripts is to use the Wscript.Echo method to write to the screen to trace the flow of program execution. This sample script is laced with Wscript.Echo statements that were used in debugging, but subsequently have been commented out. In order to view the output from Wscript.Echo statements while you test your script during Cycle End processing, you must
 |
remove the //B parameter that suppresses interactive output from the Wscript invocation statement on the Cycle End command line, |
|
allow the Performance SeNTry collection service to interact with the desktop (as illustrated below), and |
|
restart the Performance SeNTry collection service after you have applied this Service Control Manager parameter change so that it can take effect. |
This will allow you to view all Wscript.Echo output when the script executes during Cycle End command processing.
When you execute Wcsript.exe or ccript.exe to run your script, the //B parameter indicates batch execution mode. Batch mode suppresses command-line display of user prompts and script errors, which is important when you run a script during Cycle End command processing. Without the the //B parameter, your script will hang, waiting for user input if you execute a Wscript.Echo statement that writes to the screen.
WARNING: When you have finished debugging your script, make sure you restore the //B parameter to the script invocation command line before you distribute to a production server. If you do not allow the collection service to interact with the desktop, as illustrated above, any attempts to write to the screen using Wscript.Echo statements will be invisible. In this case, not only will the script hang, but you will not be able to tell that it is hung.