I have encountered an interesting issue where a series of applications that have been virtualized with App-V 4.6 will fail due to an operational error. Ironically it has nothing to do with any kind of limits within App-V or is even related to the virtualization model of App-V. It happens because a key component of App-V that allows for the processing of OSD information fails due to interference from third-party software.
Crash Followed by Error
The problem begins when the majority of the applications experience an error with the SFTLP process. SFTLP.EXE will crash with the following error:
SFTLP.EXE – Application Error
The instruction at “0x0000XXXX” referenced memory at “0x0000XXXX”. The memory could not be “read.”
NOTE: You will see a slightly different error if Windows Error Reporting (WER) has been turned on.
Since SFTLP is an integral part of building the application’s virtual environment and it has crashed while in process, the launch will not complete leading to the following client launch error:
The Application Virtualization Client could not launch
An unexpected error occurred.
Report the following error code to your System Administrator. Error Code: xxxxxx-xxxxxx19-00001001
Further troubleshooting proved difficult as we were even unable to properly collect a user dump via WER (Windows Error Reporting) to determine why the SFTLP.EXE process was crashing. We tried attaching several debuggers to the AEDebug key and eventually was able to get further information on the crash. The reason WER was not very helpful was an inability to access the protected information within the bubble but we were at least able to know it was an access violation occurring. Other interesting elements also played a factor in the difficulty of troubleshooting:
- When the SFTLOG.TXT was set to verbose, the only thing that was revealed was that the SFTLP process encountered access violation exit code of 5 which we already knew.
- Use of the UnhandledExceptionLogger file SFTLDR.LOG also revealed nothing since this error occurred well before the primary executable of the virtual application was launched.
- Launching Procmon (Process Monitor) – either through the use of a command prompt or /externalcapture failed as the launch would start and immediately stop.
At this point, we determined we needed to look at something that was likely happening inside the operating system at the kernel level. Whenever something like this happens and I suspect that it may be third-party software with filter/kernel drivers, I will look at disabling unknown filter drivers. Bear in mind disabling non-Microsoft services and startup items using MSCONFIG will NOT disable third-party filters. You have to disable these filters by using Device Manager (I would advise this rather than directly editing the registry values when troubleshooting.) In Device Manager, you can help yourself identify these 3rd-party filter drivers by opening it up directly as an admin using devmgmt.msc or from the System Control Panel – and under “View” selecting “Show Hidden Devices.” This could be a tedious process, but absent any other troubleshooting data, it may be your best bet for narrowing down the issue. You can disable the driver from this interface and reboot in order to reproduce the issue with the driver disengaged.
AUTORUNS – A Shortcut
I use Autoruns as a quick way of telling me which filter drivers are 3rd-party. The Drivers tab will list of the drivers and their publisher. It helps me to quickly isolate the non-Microsoft filter drivers. NOTE: If there is a driver WITHOUT a publisher, I would definitely start with those. You can download Autoruns here: http://technet.microsoft.com/en-us/sysinternals/bb963902
In the issue above, we proceeded to disabled unknown filters in the Kernel until we came to one called “SysPlant for NT.” This particular driver registered under: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SysPlant. The specific driver build in question came from Symantec and was at version 12.1.1000.157. Sure enough, when we disengaged the driver and rebooted all was well. Further research revealed that it was tied to Symantec Endpoint Protection. Even after trying Service and Process Inclusions for the SysPlant driver and the SepMasterService, it still was not helping with the driver engaged. We found that as long as we installed Symantec Endpoint Protection without the Device and Application control feature, we were able to avoid this problem as well.