Summary: Matthew Hitchcock, Microsoft MVP, delves into how to troubleshoot problems in the Azure VMDSC Extension.
If you joined me yesterday in Advanced Use of Azure Virtual Machine DSC Extensions, you saw how I created an advanced Desired State Configuration (DSC) file to configure my Azure VM. Taking parameter input means I can reuse my DSC file across different machines and different environments. Today I want to spend some time talking about troubleshooting because I recently experienced many hours of frustration with DSC on Azure VMs. Hopefully sharing what I learned can save you some pain.
In my example yesterday, I gave the wrong value for a parameter so my configuration is failing. Let’s take a look at some of the ways we can troubleshoot Azure VM DSC Extension issues.
Understanding what’s going on inside the VM
Good troubleshooting starts with understanding what should be going on. To check that everything is healthy in the VM, we can:
- Check the status and progress
- Read the DSC log
- Read the parameter values provided to the configuration
Check the status and progress
We can use the following command to examine the status of the VM DSC Extension (the equivalent to checking the Local Configuration Manager):
Get-AzureVM -Service "MyService" -Name "MyVM" | Get-AzureVMDSCExtension
This shows us the configuration that has been assigned and an indication of some of the parameters we specified.
Get-AzureVM -Service "MyService" -Name "MyVM" | Get-AzureVMDSCExtensionStatus
This gives us an indication of what stage the configuration is for the DSC Extension. We can examine the state, which will show if there has been an error. In addition, the last time stamp tells us when the last event happened.
Unfortunately, there is not a great deal of information here. A status message of “Errors occurred while processing the configuration ‘Configuration Name’” generally indicates an issue in reading the PowerShell code you provided and applying it to a MOF file. In short, there is something wrong with your code.
We’ll need to go through the DSC code with a bit of a fine-tooth comb. Fortunately, my configuration is still small.
Ah ha! The issue is likely that I have a dependency set where the name doesn’t actually match! The DSC engine didn’t understand when it could run this part of the configuration, so it decided to stop before it did something bad.
I can correct this as follows:
I can use the same commands that I used yesterday to upload the configuration (using the -Force parameter to overwrite the existing one) and to assign it to the Azure VM. I’ll once again monitor the progress by using the Get-AzureVMDSCExtension and Get-AzureVMDSCExtensionStatus cmdlets. Have some patience...it can take some time to start working again.
Read the DSC log
Reading the DSC log is the equivalent of using -Wait -Verbose when pushing a DSC file. As we saw when we used Get-AzureVMDSCExtensionStatus, there is a DSCConfigurationLog property. This is initially empty, but if there is a value in there, it means that the DSC file has been successfully converted into a MOF file and is running or has run. We can use the following command to list it in our session:
Get-AzureVM -Service "MyService" -Name "MyVM" | Get-AzureVMDSCExtensionStatus | Select-Object -ExpandProperty DSCConfigurationLog
The following image displays the output as would be seen in a push configuration using -Wait –Verbose:
We see that the issue is that the domain is not found. So how do we check which domain we assigned as a parameter?
Read the parameter values provided to the configuration
If we need to validate the parameter values that we specified when we assigned the DSC file to the virtual machine, we can use the following command:
Get-AzureVM -Service "MyService" -Name "MyVM" | Get-AzureVMDSCExtension | Select-Object -ExpandProperty Properties
Seems like that’s where the issue is! We have a bad parameter value.
Updating parameter values
So, we have seen that the issue is a bad parameter value. My domain isn’t called hitchy.com. Oops!
To update parameter values, I’ll go back to assigning the configuration to the VM as I did yesterday. This time, I’ll get it right. I don’t have to rush and scramble to do this—I can simply wait until the DSC Extension shows an error, which will stop the process. Then I can update the parameters in the original assignment command and run it again.
Now I can check again and see that the parameter set in the DSC agent is correct.
Great! We can now watch the DSC file complete by periodically using the Get-AzureVMDSCExtension cmdlet. Also remember that we can use the DSCConfigurationLog property again if we want to see a play-by-play of how it all happened.
When I see that it has successfully completed, I can log in with my domain credentials and verify that the share exists on the server. Job done!
How I wish I had approached Azure DSC troubleshooting
If I could go back to when I started working with this and talk to my slightly younger self, I'd give myself the following advice:
Desired State Configuration is all about “eventual consistency.” The script automation most of us are used to allows us to watch everything happening interactively. When I start a script, I am used to seeing it run and work. With DSC, we need to trust it to complete and have only a periodic peek.
When I saw that a server had not been performing an action for some time, it was so tempting to “give it a kick.” I had to break that habit and resist, trusting in my LCM settings and lack of errors in logs and allow it to complete in its own time.
Build configurations by testing incrementally
As someone comfortable with DSC syntax, I flew into writing configurations for my Azure VMs. I wasn't really testing as I went because I knew the syntax. I had used it before and I couldn't see a reason why it wouldn't work.
After writing quite monolithic configurations and starting to assign them to VMs, I was hitting errors (such as the incorrect dependency error I showed), which weren't that helpful. I couldn't understand where my code could possibly be wrong because everything looked the same as how I would write a standard DSC file that I would push to a machine running Windows or assign to a pull server.
In the end, I went back to the basics, starting with an empty configuration, adding configuration elements one-at-a-time, and testing before I moved forward.
Had I adopted this practice from the start, I would have achieved much more, much faster; sent many less ranting emails; and spent less time questioning my ability!
Fall back to DSC within a Windows host to test
When something that you added to a configuration causes it to fail, a great way to see what's wrong is by trying it in a push configuration within another Windows VM. Compile the configuration into a MOF file and push it to the local system (one you can test and break). Look for compilation errors when you create the MOF file, and then when you run the configuration, use the -Wait and -Verbose switches so you can see what fails.
Stick with it
The most important piece of advice I can give is to stick with it. Changing your habits to start configuring your infrastructure with code rather than manually is a hard thing to do. It can take longer to do the simplest tasks and it can be frustrating when error messages are not clear and you realize you have spent two days trying to join a machine to the domain.
As tempted as I have been at times to "just do this manually and fix it later," I am glad I didn't. The rewards of breaking the manual habit and succeeding with automation are huge. The result of my personal struggles on a recent project mean that our teams around the world can spin up an environment to showcase a solution and show value faster. So I have to say it was worth it for me.
That's all for today folks. I hope this helps you along with the DSC Extension and encourages you to try it. When you have your servers building themselves, it truly is magical. It also gives you more time to sharpen your skills as a reward for your work!
If you have suggestions for additional troubleshooting actions that could be taken with the Azure VM DSC Extension, I would love to add them to my list. You can contact me on Twitter @hitchysg.
Thanks for taking the time to share this information, Matthew.
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at firstname.lastname@example.org, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy