App-V: Process Exit Codes and why they matter when troubleshooting virtual applications

In a previous blog post (https://blogs.technet.com/b/gladiatormsft/archive/2013/04/27/on-the-art-of-troubleshooting-app-v-applications.aspx) I discussed the “art” of troubleshooting virtual applications and mentioned the fact that there are a lot of custom recipe’s out on the Internet (many courtesy of our own MVP’s.) People ask me how they figure these out. My answer is usually along the lines of “they troubleshot/revered engineered/mangled though until they came up with something that worked.” We know the next question: How do you learn to do that? To keep my promise in helping people to *LEARN* to troubleshoot virtual applications, I have been working on scenarios using real-world applications. I figured a good place to start was using an application that fails and leaves NO error message whatsoever. This is a good way to demonstrate the use of Process Monitor to track process exit codes in order to resolve issues.

The application we will be looking at is Google Chrome 15 sequenced with App-V 5. I chose this one because it is really easy to create a bad sequence (using the Google Enterprise Installer.) All you do is during monitoring – is accept all defaults (click – next – next – next etc.) Now if you ever have had to sequence Google Chrome and did your research on the Internets, you have found many complicated recipes for Google Chrome. My favorite is Aaron Parker’s (an APP-V MVP) 4.6 recipe: https://stealthpuppy.com/virtualisation/sequencing-google-chrome-15/  (Yes, I am a fan of the stealth puppy.) It is one of the many examples of one of our solid MVP’s sharing quality APP-V packaging information to prevent others from having to reinvent the wheel.

If you would like to follow along and try this at home, download the Google Chrome 15 Enterprise Installer (https://www.google.com/chrome/eula.html?msi=true) – sequence it accepting all defaults on a clean sequencer template (default exclusions) and enjoy the troubleshooting!

Symptom: I double-clicked on the virtual application – AND NOTHING HAPPENS!

So this is the issue. You have virtualized an application(Google Chrome 15) with App-V 5 and it adds/publishes successfully. However, when you go to launch the application, nothing happens. You may notice the process starts in Task Manager but then disappears. A not so nice silent exit with NO error message whatsoever. Now is a good time to break out Process Monitor and capture a simple reproduction of the launch and subsequent silent exit.

Process Exit Codes

So whenever you get strange exits or “lack of launch” with no error – your best recourse is to try to find out why those key processes exited. This time we are going to take advantage of the fact that Process Monitor tracks exit codes.

First, inventory the EXE’s with Chrome. You can do it with a local install or view the shortcut to the EXE to get the main executable. (In 4.6, you would look in the OSD file.) The key EXE’s I find with Google Chrome are CHROME.EXE and GOOGLEUPDATE.EXE.

Now, capture the issue with Process Monitor. Once you have captured the session, you will want to apply a display filter for the following operations:

-       Process Start

-       Process Exit

Apply the filter and start from the beginning. Well look! Not a lot of events! This is much easier to walk through.

The WMIPRVSE.exe is noise (WMI background process) but notice that in addition to the CHROME.EXE process, there is a “GOOGLEUPDATE.EXE” process. What is interesting is that the shortcut I clicked on was supposed to load the CHROME.EXE process. This will cause me to want to look at Parent PIDs to see what spawned what because for some strange reason, the GOOGLEUPDATE.EXE process started before the CHROME.EXE process. The chrome.exe process was spawned by Parent PID 832 – which turns out to be EXPLORER.EXE. GOOGLEUPATE.EXE appears to have been spawned by PID 1620 which is the AppVClient.exe process.

Looking at the command details for GoogleUpdate.exe, it begins to make sense – GoogleUpdate is a service process (GoogleUpdate.exe/svc) so it was captured as a virtual service therefore it must start before the application does. One of the many reasons why the App-V virtualization engine hooks CreateProcess.

You can add the “Create Process” operation to your filter and you can see Process Create events which is an easier way of looking at who spawned who.

In the detail we see Exit Status codes from both processes when they exit. If the exit status is 0, the process exited normally. If the exit status is a simple number, you can often look these up with the NET HELPMSG command.

I usually work these from the bottom up. Chrome.EXE exited with a status code of 6. 

That is an invalid handle. The GoogleUpdate.exe process exited with a numerical representation of an HREF (which is better to understand once converted to hex.)

Convert -2147024894 to hex and you get 0x80070002. That’s a well-known HREF

HINT – download this conversion tool - https://www.microsoft.com/en-us/download/details.aspx?id=985

It says Exchange but it will resolve standard Windows API HREF codes as well. The above resolves to ERROR_FILE_NOT_FOUND. At this point, it is time to reset the filter and create a new filter where we include the process name GOOGLEUPDATE.exe and exclude success. Then exclude NAME_NOT_FOUND
(noise.) Look for PATH_NOT_FOUND. Well what do you know? There it is!

And now we have to reset the filter to make sure there were no additional success events (due to path searching and alternate location searches.)

Looking at the last one registering a PATH_NOT_FOUND, unlike the previous ones where additional searches found the intended file, the attempt to create several files beneath the update folder did not. Verifying this in the package directory, we see the file is not there. By default, on a local install, the file is not there. However upon first launch when the Google Update service runs the files are downloaded successfully.

What are your Options Now?

At this point, there are a few paths we can take:

Add in those missing files? Adding in those missing files will generate the same results because there is a check from the application to download the most up-to-date binaries for the update engine. Process Monitor revealed the same ERROR_FILE_NOT_FOUND exit code.

Disable Virtual Services? You can disable virtual services altogether in either the deployment_config.xml file (for the application) or you can disable the virtual services in the registry (quick way is [reg add HKLM\Software\Microsoft\AppV\Subsystem\Disable /v "Virtual Services /t REG_DWORD /d 1 ) client-wide.

NOTE:Upon disabling virtual services – the ISSUE STILL REMAINED. Again since the symptom is the same – I go back to the exit codes with process Monitor. This time, the GOOGLEUPDATE.EXE process did not start as desired.

Yet this time chrome still failed because of the bad handle issue. So simply taking the update component out of the sequence is not good enough. You need to be able to configure the application not to auto-update (if possible.) This goes back to the “knowing of your application” principle.

Application Disabling/Sequencing Exclusion of the Particular Service

If disabling the service through the App-V subsystem is not successful, you can still virtualize the application with App-V if there is a mechanism within the application to disengage the service altogether. To be honest, this is the best way, otherwise, the application will treat the service as a dependency for functionality. In the case of Google Chrome, you can disable this through the following registry key:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Update

Value: AutoUpdateCheckPeriodMinutes

Data Type: REG_SZ

Data: 0

In addition you will want to add in a directory\VFS exclusion for the [{ProgramFiles}]\Google\Update directory.

So I hope this was able to shed some light on how you get from a default-installation-based failed sequence to a working one through the use of a good reverse engineering tool. Of course, don’t forget to share your success stories in the form of recipes. You can do this on the App-V Sequencing Recipe forum here: https://social.technet.microsoft.com/Forums/en-US/prescriptiveguidance/threads