Weekend Scripter: Playing Around with the Windows PowerShell Tokenizer

ScriptingGuy1

//

Microsoft Scripting Guy Ed Wilson here. Once again, the Scripting Wife and I are on the road. This weekend we are heading to Charleston, South Carolina, to hang out and to see a couple music groups. The Scripting Wife is driving, and I am playing around with Windows PowerShell. We have my Zune HD plugged into our car radio and are listening to Jimmy Buffett—ARRH! She just interrupted my favorite song—“A Pirate Looks at 40”—to put on…Johnny Horton? Yes, I think it is Johnny Horton.

Cool. I need to think cool. Before we left, I used one of my Windows PowerShell scripts to check the weather for Charleston. Tomorrow, it is going to be a high of 89 degrees with 87 percent humidity! Oh, well, it is nothing a little bit of ice tea, sunglasses, an aloha shirt, and a Panama hat cannot handle.

I have been wanting to play around with the Windows PowerShell Tokenizer for a while. The Tokenizer is used to break a Windows PowerShell script into pieces of code called tokens. Using the Tokenizer, you can find commands or variables in a Windows PowerShell script. A road trip is the perfect occasion to write Windows PowerShell scripts that explore new techniques. Because it is Friday evening and the script is just for fun, there are no deadlines and no specific guidelines to which the script must adhere.

The ParseScriptCommands.ps1 script is my first attempt. It is shown here.

ParseScriptCommands.ps1

$errors = $null
$logpath = “C:\logs\commandlog.txt”
$path = “C:\data\PSExtras”
Get-ChildItem -Path $path -Include *.ps1 -Recurse |
ForEach-Object {
  $script = $_.fullname
  $scriptText = get-content -Path $script
  [system.management.automation.psparser]::Tokenize($scriptText, [ref]$errors) |
  Foreach-object -Begin {
    “Processing $script” | Out-File -FilePath $logPath -Append } `
  -process { if($_.type -eq “command”)
    { “`t $($_.content)” | Out-File -FilePath $logpath -Append } }
}
notepad $logpath

The first thing I do in the ParseScriptCommands.ps1 script is initialize three variables. The first one is used to collect any errors by Tokenizer. The second variable is used for the log file, and the third one specifies the directory that contains the Windows PowerShell scripts that need to be parsed. These commands are shown here:

$errors = $null
$logpath = “C:\logs\commandlog.txt”
$path = “C:\data\PSExtras”

The next four commands are standard Windows PowerShell cmdlets. The Get-ChildItem cmdlet retrieves only Windows PowerShell scripts (that have the .ps1 extension) from the script directory specified earlier. The –recurse parameter is required when retrieving the files from the folder. The resulting fileinfo objects are piped to the Foreach-Object cmdlet where the full path to each script is stored in the $script variable. Next, the Get-Content cmdlet reads each Windows PowerShell script and stores the content of the file in the $scriptText variable. This section of the script is shown here:

Get-ChildItem -Path $path -Include *.ps1 -Recurse |
ForEach-Object {
  $script = $_.fullname
  $scriptText = get-content -Path $script

The psparser .NET Framework class in the system.management.automation namespace has the Tokenize static method. The first parameter is a variable containing the contents of a Windows PowerShell script, and the second parameter is a reference variable to hold the errors. The second parameter must be supplied when calling the Tokenize method. The tokens are then piped to the next section of the script. This command is shown here:

[system.management.automation.psparser]::Tokenize($scriptText, [ref]$errors) |

I use the Foreach-Object cmdlet to process each token. The first thing I do is write the full script path to the log file. Next, I check to see if the type property of the token object is a command—if it is, I write the command to the log file as well. When I have processed all of the scripts in the folder, I display the contents of the log file. This section of the script is shown here:

  Foreach-object -Begin {
    “Processing $script” | Out-File -FilePath $logPath -Append } `
  -process { if($_.type -eq “command”)
    { “`t $($_.content)” | Out-File -FilePath $logpath -Append } }
}
notepad $logpath

When the script runs, a text file appears that is similar to the one shown in the following image.

Well, believe it or not, we are just about to arrive in Charleston. The Scripting Wife has abandoned Johnny Horton and is now listening to Kiss. I guess she is growing impatient. The GPS says we will be there in 6 miles, so I need to shut down my laptop and prepare to disembark. Check back tomorrow. I have some pretty cool ideas for improving my tokenizing script. I might take my laptop down to park that overlooks Fort Sumpter and work on that script tomorrow morning. Who knows what I will do really? It’s the weekend, and tomorrow is another day.

If you want to know exactly what we will be looking at tomorrow, follow us on Twitter or Facebook. If you have any questions, send e-mail to us at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.

Ed Wilson and Craig Liebendorfer, Scripting Guys

ScriptingGuy1

Follow