The Case of the Temporary Registry Profiles


Microsoft Customer Support Services (CSS) is one of the biggest customers of the Sysinternals tools and they often send me interesting cases they’ve solved with them. This particular case is especially interesting because it affected a large number of users and the troubleshooting process made use of one of Process Monitor’s lesser-known features. The case opened when a customer contacted Microsoft support reporting that several of their users would occasionally get this error message when loggging on to their systems:

image

This caused Windows to create a temporary profile for the user’s logon session. A user profile consists of a directory, %UserProfile%, into which applications save user-specific configuration and data files, as well as a registry hive file stored in that directory, %UserProfile%\Ntuser.dat, that the Winlogon process loads when the user logs in. Applications store user settings in the registry hive by calling registry functions that refer to the HKEY_CURRENT_USER (HKCU) root key. The user’s loss of access to their profile made the problem critical, because whenever that happened, the user would apparently lose all their settings and access to files stored in their profile directory. In most cases, users contacted the company’s support desk, which would ask the user to try rebooting and logging in until the problem resolved itself.

As with all cases, Microsoft support began by asking about the system configuration, inventory of installed software, and about any recent changes the company had made to their systems. In this case, the fact that stood out was that all the systems on which the problem had occurred had recently been upgraded to a new version of Citrix Corporation’s ICA client, a remote desktop application. Microsoft contacted Citrix support to see if they knew of any issues with the new client. They didn’t, but said they would investigate.

Unsure whether the ICA client upgrade was responsible for the profile problem, Microsoft support instructed the customer to enable profile logging, which you can do by configuring a registry key as per this Knowledge Base article: How to enable user environment debug logging in retail builds of Windows. The customer pushed a script out to their systems to make the required registry changes and shortly after got another call from a user with the profile problem. They grabbed a copy of the profile log off the system from %SystemRoot%\Debug\UserMode\Userenv.log and sent it into Microsoft. The log was inconclusive, but did provide an important clue: it indicated that the user’s profile had failed to load because of error 32, which is ERROR_SHARING_VIOLATION:

image

When a process opens a file, it specifies what kinds of sharing it allows for the file. If it is writing to the file it may allow other processes to read from the file, for example, but not to also write to the file. The sharing violation in the log file meant that another process had opened the user’s registry hive in a way that was incompatible with the way that the logon process wanted to open the file.

In the meantime, more customers around the world began contacting Microsoft and Citrix with the same issue, all had also deployed the new ICA client. Citrix support then reported that they suspected that the sharing violation might be caused by one of the ICA client’s processes, Ssonvr.exe. During installation, the ICA client registers a Network Provider DLL (Pnsson.dll) that the Windows Multiple Provider Notification Application (%SystemRoot%\System32\Mpnotify.exe) calls when the system boots. Mpnotify.exe is itself launched at logon by the Winlogon process.The Citrix notification DLL launches the Ssonvr.exe process asynchronous to the user’s logon:

image

The only problem with the theory was that Citrix developers insisted that the process did not attempt to load any user registry profile or even read any keys or values from one. Both Microsoft and Citrix were stumped.

Microsoft created a version of Winlogon and the kernel with additional diagnostic information and tried to reproduce the problem on lab systems configured identically to the client’s, but without success. The customer couldn’t even reproduce the problem with the modified Windows images, presumably because the images changed the timing of the system enough to avoid the problem. At this point a Microsoft support engineer suggested that the customer capture a trace of logon activity with Process Monitor.

There are a couple of ways to configure Process Monitor to record logon operations: one is to use Sysinternals PsExec to launch it in the session 0 so that it survives the logoff and subsequent logon and another is to use the boot logging feature to capture activity from early in the boot, including the logon. The engineer chose the latter, so he told the customer to run Process Monitor on one of the system’s that persistently exhibited the problem, select Enable Boot Logging from the Process Monitor Options menu, and reboot, repeating the steps until the problem reproduced. This procedure configures the Process Monitor driver to load early in the boot process and log activity to %SystemRoot%\Procmon.pmb. Once the user logged encountered the issue, they were to run Process Monitor again, at which point the driver would stop logging and Process Monitor would offer to convert the boot log into a standard Process Monitor log file.

After a couple of attempts the user captured a boot log file that they submitted to Microsoft. Microsoft support engineers scanned through the log and came across the sharing violation error when Winlogon tried to load the user’s registry hive:

image

It was obvious from operations immediately preceding the error that Ssonsvr.exe was the process that had the hive opened. The question was, why was Ssonsvr.exe opening the registry hive? To answer that question the engineers turned to Process Monitor’s stack trace functionality. Process Monitor captures a call stack for every operation, which represents the function call nesting responsible for the operation. By looking at a call stack you can often determine an operation’s root cause when it might not be obvious just from the process that executed it. For example, the stack shows you if a DLL loaded into the process executed the operation and, if you have symbols configured and the call originates in a Windows image or other image for which you have symbols, it will even show you the names of the responsible functions.

The stack for Ssonsvr.exe’s open of the Ntuser.dat file showed that Ssonsvr.exe wasn’t actually responsible for the operation, the Windows Logical Prefetcher was:

image

Introduced in Windows XP, the Logical Prefetcher is a kernel component that monitors the first ten seconds of a process launch, recording the directories and portions of files accessed by the process during that time to a file it stores in %SystemRoot%\Prefetch. So that multiple executables with the same name but in different directories get their own prefetch file, the Logical Prefetcher gives the file a name that’s a concatenation of the executable image name and the hash of the path in which the image is stored e.g. NOTEPAD.EXE-D8414F97.pf. You can actually see the files and directories the Logical Prefetcher saw an application reference the last time it launched by using the Sysinternals Strings utility to scan a prefetch file like this:

strings <prefetch file>

The next time the application launches, the Logical Prefetcher, executing in the context of the process’s first thread, looks for a prefetch file. If one exists, it opens each directory it lists to bring the directory’s metadata into memory if not already present. The Logical Prefetcher then maps each file listed in the prefetch file and references the portions accessed the last time the application ran so that they also get brought into memory. The Logical Prefetcher can speed up an application launch because it generates large, sequential I/Os instead of issuing small random accesses to file data as the application would typically do during startup.

The implication of the Logical Prefetcher in the profile problem only raised more questions, however. Why was it prefetching the user’s hive file in the context of Ssonsvr.exe when Ssonsvr.exe itself never accesses registry profiles? Microsoft support contacted the Logical Prefetcher’s development team for the answer. The developers first noted that the registry on Windows XP is read into memory using cached file I/O operations, which means that the Cache Manager’s read-ahead thread will proactively read portions of the hive. Since the read-ahead thread executes in the System process, and the Logical Prefetcher associates System process activity with the currently launching process, that a specific timing sequence of process launches and activity during the boot and log on could cause hive accesses to be seen by the Logical Prefetcher as being part of the Ssonsvr.exe launch. If the order was slightly different the next boot and log on, Winlogon might collide with the Logical Prefetcher, as seen in the captured boot log.

The Logical Prefetcher is supposed to execute transparently to other activity on a system, but its file references can lead to sharing violations like this on Windows XP systems (on server systems the Logical Prefetcher only prefetches boot activity, and it does so synchronously before the boot process proceeds). For that reason, on Windows Vista and Windows 7 systems, the Logical Prefetcher makes use of a file system minifilter driver, Fileinfo (%SystemRoot%\System32\Drivers\Fileinfo.sys), to watch for potential sharing violation collisions and prevent them by stalling a second open operation on a file being accessed by the Logical Prefetcher until the Logical Prefetcher closes the file.

Now that the problem was understood, Microsoft and Citrix brainstormed on workarounds customers could apply while Citrix worked on an update to the ICA Client that would prevent the sharing violation. One workaround was to disable application prefetching and another was to write a logoff script that deletes the Ssonsvr.exe prefetch files. Citrix published the workarounds in this Citrix Knowledge Base article and Microsoft in this Microsoft Knowledge Base article. The update to the ICA Client, which was made available a few days later, changed the network provider DLL to 10 seconds after Ssonsvr.exe launches before returning control to Mpnotify.exe. Because Winlogon waits for Mpnotify to exit before logging on a user, the Logical Prefetcher won’t associate Winlogon’s accesses of the user’s hive with Ssonsvr.exe’s startup.

As I said in the introduction, I find this case particularly interesting because it demonstrates a little known Process Monitor feature, boot logging, and the power of stack traces for root cause analysis, two key tools for everyone’s troubleshooting arsenal. It also shows how successful troubleshooting sometimes means coming up with a workaround when there’s no fix or you must wait until a vendor provides one. Another case successfully closed with Process Monitor! Please keep sending me screen shots and log files of the cases you solve.


Comments (70)

  1. @stefang

    >This is exactly the problem with the windows file sharing model: In an ideal world it should be possible for the prefethcer to open a file for reading in a way that never interferes with other processes. Today this is impossible because winlogon opens the file in a way that disallows any file sharing.

    I think you’re missing the point. It’s up to the process that opens the file to decide what kinds of sharing it allows. If a process wants to allow other processes to read from a file it’s got opened, it can do so. Presumably it will prevent concurrent access for a non-arbitrary reason. Winlogon choses to not allow other processes to read the raw contents of a hive that’s loaded. It’s taking advantage of the power of the sharing model to enforce it’s wish.

    If a process wants to read the raw contents of a hive it can save a copy with RegSaveKey or create a volume snapshot, which allows full access to the hive file with consistent contents for the snapshot’s point in time.

  2. @Robert

    It looks like you missed this important aspect of the case:

    When a process opens a file, it specifies what kinds of sharing it allows for the file. If it is writing to the file it may allow other processes to read from the file, for example, but not to also write to the file.

  3. @Robert

    That’s my point: Windows sharing semantics do allow a process to specify that full sharing is allowed when they open a file for reading.

    The prefetcher specifies full sharing, but Winlogon doesn’t, so both can’t have the file open at the same time.

  4. Dmitry says:

    Mark, a small typo in this excellent article – "could cause hive accesses to been by the Logical Prefetcher as being part". Been -> be seen

    Interesting investigation, thanks 🙂

  5. Thanks for an excellent article. It demonstrates what I’ve always believed about much of the Windows OS- It is WAY more complex than it should be: Its like the wild west in there with only a few general guidelines and no regulation.

    MS does have a lot of great ideas, but many of the ideas and their implementations make me _shudder_. The registry is one idea that was poorly done. Prefetch may be a good idea for some systems, but not any pc’s that I have used. The fact that prefetch gets bitten by registry only shows the severe limitations on the registry implementation, like so many other parts of the OS.

    OT: My wish is that the Windows OS only had a single file called "WINDOWS.SYS" that only MS can update (or developers who want to dare take the chance). I’ve been stung too many times by DLL hell and application updates that messed around with the OS system files.

  6. Alexei says:

    > Microsoft and Citrix brainstormed on workarounds customers could apply while Citrix worked on an update to the ICA Client that would prevent the sharing violation.

    What I don’t understand is why Citrix should solve  a problem which is obviously created by Microsoft Windows? This is MS team who should introduce the patch, not Citrix.

  7. X-STAR says:

    Hi,Mark,

    It’s intersting.

    And I always use Process Monitor and Process Explorer to slove my problem.

    Those sysinternals tools make my life easy.

    Thank you very much!

  8. Alex Railean says:

    Great article, and it is also one of the most detailed descriptions of the "prefetch" feature I’ve seen so far.

  9. David J. Veer says:

    Mark,

    Very interesting article, thank you! I actually had this error message a little while ago too but it turned out to be something not related to the Citrix ICA software at all. I never did figure out what the problem was because at the time I had never heard about Process Monitor, but this information might help me troubleshoot it better in the future.

    Thanks!

    David

  10. Mick says:

    @Michael Fitzpatrick – [Tuesday, August 11, 2009 2:25 AM]

    > Thanks for an excellent article. It demonstrates what I’ve always believed about much of the Windows OS- It is WAY more complex than it should be: Its like the wild west in there with only a few general guidelines and no regulation.

    Really? How? What would make it simpler? Do you think the combination of the X window system and Linux/BSD kernel is more elegant/simpler?

    >  The registry is one idea that was poorly done.

    Why is it poorly done? Are INI files better? Or maybe move to OS X’s version of the registry called PList’s? They all have their pitfalls.

    > The fact that prefetch gets bitten by registry only shows the severe limitations on the registry implementation, like so many other parts of the OS.

    I thought the problem was due to pre-fetch, not the registry. They fixed the issue in Vista/7 by fixing pre-fetch. Read the article again.

    > OT: My wish is that the Windows OS only had a single file called "WINDOWS.SYS" that only MS can update (or developers who want to dare take the chance). I’ve been stung too many times by DLL hell and application updates that messed around with the OS system files.

    Did you write this response in 1998? Yes, I also wish that every necessary dependency was in A SINGLE FILE!! Then, every time I needed to patch a bug in the networking code, I’d also have to patch the kernel, the filesystem, the wlan, the HAL, etc.

  11. ryan says:

    I have occasionally run into this very issue.  Good to know the reason!

  12. John says:

    I agree with Alexei; this is a Microsoft problem.  And I don’t see how this is specific to Citrix at all; this issue could have shown up in any application.

  13. JM says:

    @Alexei: A patch that changes an OS component requires much more exhaustive QA. There’s a trade-off between the number of problems you solve and the number of problems you potentially introduce.

    Even if Microsoft had said "yes, it’s an OS problem and we’re going to patch it too" (which they may very well have said or may still say, the story is silent on that), Citrix still would have said "that’s great, now please advise us on how to patch *our* software since we can do it quicker and our customers are waiting".

    When your customers are in trouble, you don’t ask "who’s responsible for this and when are they planning to do something about it", you ask "what can we (all people willing to contribute) do to fix this problem as quickly as possible". In that sense, whether Microsoft decided or ever will decide to patch it is utterly irrelevant to both Citrix and their customers (except insofar as it might impact their current solution). All the people who *might* experience the same problem in some other form aren’t in the picture yet.

    @John: that’s probably why they changed it for Vista and Windows 7. But that still doesn’t mean you automatically go back and patch your existing OS. The fact that this bug was so hard to reproduce in the first place is probably a sign this behavior rarely causes trouble, at least not of the magnitude seen here.

  14. lexxmt says:

    On Windows Vista some issue with Windows Media Player sharing service. If it loads and opens user profile before system :).

  15. liys says:

    “The developers first noted that the registry on Windows XP is read into memory using cached file I/O operations, which means that the Cache Manager’s read-ahead thread will proactively read portions of the hive proactively. Since the read-ahead thread executes in the System process, and the Logical Prefetcher associates System process activity with the currently launching process”

    Excuse my ability of understanding. I’m still having trouble understand the root cause of the problem. The above sentence to be specific.

    So the cache mgr will read the ntuser.dat when the system boots, and Logical Prefetcher will record cache mgr’s access pattern, but why it has anything to do with Ssonsvr.exe shouldn’t it be running in the context of the System process?

    Thanks,

  16. JM says:

    @liys: the Prefetcher associates the activities of the System process with the user process that’s currently launching. It does this because much activity is indirectly done by the system on behalf of the process, especially during process startup — capturing this activity makes for better prefetching. Keeping this activity separate only makes sense at boot time, when the System process is the only thing running (this is not literally true, by the way, but a good enough approximation).

    The fact that Prefetcher can’t distinguish between System process activity on behalf of the process being prefetched and system activity that’s unrelated is what causes the problem. Simply not recording System process activity at all would make prefetching much less effective.

  17. tom says:

    The discussion above makes an interesting point: Citrix is a responsible company that cares about its customers.  Whoever was at fault, they put in the workaround so their customers won’t run into the problem.

    I’ve seen plenty of companies that tracked an issue outside of their code and then said, "Nyah nyah, it’s their fault, don’t try to get us to do anything."  Gets even worse in the case of conflicting interpretations of a spec.  "No, we’re right and they’re wrong."

    It’s also pretty common in the open-source community.  The calling app considers the problem to be in the dependency, while the dependency thinks that it’s a corner case that’s not worth the time to investigate.  Meanwhile, you just want the problem to be fixed.

  18. Mark Lee says:

    Nice write up. Our company ran into this issue a few months back. At that time, I captured the same Userenv.log error you have posted. That info helped isolate the problem with the Citrix client. Our company is on a older version 9.2 of the ICA client. Our user base is over 100K. All the machines work off a common XP build with same version of Citrix. The problem was only occuring with a single business unit. This was the part that has us baffled. Do you have any details on what could spur the problem for some users and not others with same desktop build? Thanks.

  19. tOM Trottier says:

    Maybe that’s behind the 10 second all-idle delay I experience when I boot my Lenovo 3000 n100. Proactive prefetch prevention.

    tOM

  20. liys says:

    @JM

    Thanks for the explanation. Bur sorry I still don’t get it.

    I know the root cause is the ntuser.dat sharing violation, the prefetcher and Winlogon both trying to open the file in an incompatible way. But why this is happening?

    My guess is that because the way and the timing the Ssonvr.exe was launched during boot time? or any process with this timing could cause the problem? Could somebody elaborates it?

    Another qustion – why "Prefetcher associates the activities of the System process with the user process that’s currently launching"? why not associates it with the cache mgr’s read-ahead thread which is in the System process? because System is not an executable?

    Thanks,

  21. ashdisp says:

    I always use Process Monitor and Process Explorer,good!thank you!

  22. jmh says:

    What is the command to get process monitor to run as session 0 using psexec?

  23. CraigB says:

    I read and appreciate every one of your "The Case of…" articles.  I’m slowly learning how to make better use of the Sysinternals tools and learning the deeper workings of Windows.  Thank you for taking the time to create/update the tools and explaining how they work in real life situations.

  24. Ravi Sujanani says:

    Thank you for sharing this exciting issue. I know a lot of people around me are aware of boot logging feature. But the other method of using PsExec to log the login activity is a new one. Reading that stack info is tough one because that doesn’t shows any third party and one won’t even dream of the prefetcher doing that.

  25. sea says:

    Excelente!

    Mark is my hero! Thanks for sharing!

  26. SGS says:

    Great article, but noticed a small typo, "…proactively read portions of the hive proactively…"

  27. Peter says:

    Thank you for the very educational article. I didn’t know about the "Enable Boot Logging" feature nor the part about showing the process stack information.

    I found that there is still one small typo in the article:

    "activity during the boot and log on could cause hive accesses to seen by the Logical Prefetcher"

    should be:

    "activity during the boot and log on could cause hive accesses to be seen by the Logical Prefetcher"

  28. Robert says:

    > The Logical Prefetcher is supposed to execute transparently to other activity on a system, but its file references can lead to sharing violations like this on Windows XP systems (on server systems the Logical Prefetcher only prefetches boot activity, and it does so synchronously before the boot process proceeds).

    So this turns out to be a file system design flaw: the fact that a process cannot read a file without affecting the remainder of the system.

  29. Robert says:

    @Mark

    I think file sharing is an issue. What I am thinking of is an option to open a private copy of a file for reading, without preventing other processes from opening the original file with sharing restrictions. Effectively the copy would only need to be made when another process starts writing to the file, so the prefetching mechanism would still work.

  30. DriverDude says:

    What scares me is this is an intermittent problem, with a serious usability problem ("lost" profiles) that requires a deep understanding of Windows internals to debug. How many others might have been affected by this issue but don’t have Citrix’s infulence and skills to be able to resolve this? How many other applications were wrongly accused of screwing up people’s logins, when it was really Windows’ fault?

    The Citrix KB article is "Created Sep 5, 2008". Let’s assume it took some time to debug and write that article, so maybe the problem surfaced in 2007. Vista was released in 2007, and if a minifilter driver was added to address this problem, that means the problem was known much earlier than 2007. (I doubt even MS would risk adding a minifilter when Vista was in beta or RC.)

    So why wasn’t this addressed earlier in XP, or at least a KB article posted about it?

    I think this is a Prefetch design flaw:

    a. the Prefetcher incorrectly associates unrelated System activity with the current process. At best it does unnecessary work the next time an app starts, and at worst it caused this very visible problem.

    b. the Prefetcher is unable to pre-fetch without interfering with other processes. The prefetcher is supposed to be transparent, like the Cache Manager, so why is it using the same API that other apps use? If the CM can read-ahead without interference, why can’t the Prefetcher use the same mechanism?

    I applaud your efforts and am glad that you’re posting so *everybody* learns from this. But I often wonder why a couple of (former) outsiders are the ones solving the obscure Windows problems.

  31. stefang says:

    @Mark

    >The prefetcher specifies full sharing, but Winlogon doesn’t, so both can’t have the file open at the same time.

    This is exactly the problem with the windows file sharing model: In an ideal world it should be possible for the prefethcer to open a file for reading in a way that never interferes with other processes. Today this is impossible because winlogon opens the file in a way that disallows any file sharing.

    Compare this with how SQL server works:

    Assume transaction A takes an exclusive lock on a record, transaction B is then blocked from even reading the same record. This is very similar to the situation with the file system.

    But in SQL server transaction B can specify the option NOLOCK wich indicates that no locks are taken and no locks are honored. This means that transaction B must be prepared to handle inconsistent data, but at least there is no risk that transaction A can be affected by B’s reading.

    So, the problem with the windows file sharing model is that it is impossible for the logical prefetcher to specify something equivalent to NOLOCK when it opens the file.

    Adding the special minidriver to Vista and Windows 7 sounds like a weird hack to me. Why could you not extend the windows file sharing model to include a NOLOCK option instead ?

  32. stefang says:

    @mark

    >It’s up to the process that opens the file to decide what kinds of sharing it allows. If a process wants to allow other processes to read from a file it’s got opened, it can do so.

    Yes, I know this.

    My point is that in this case everything had been much simpler if the prefetcher could have used a NOLOCK option to open a file and read from it while being fully aware that the data might be inconsistent. In this case the prefetcher did not care at all about the content of the file.

    But I fully understand that adding such a NOLOCK feature at this point in time is probably impossible – there are probably programs that open files exclusively and then writes confidential information to the files assuming that noone can read the information because the file is opened exclusively.

  33. AG says:

    Interesting. Precisely I faced today several profile related errors when trying to coarce XP to use an existent folder to store the profile.

    It was quite easy to test by deleting the three ntuser files and then logging in to have that user HKCU recreated. ntuser.dat and friend DO get created, but somehow the user wasn’t allowed to write in the resulting hive. The registry permissions, instead of containing an entry for the user, had an entry for CREATOR OWNER, and the user wasn’t able to write anything there (of course, the accounts weren’t Administrators).

  34. @stefang – Code reading from a file can tell Windows NOT to lock it at all, i.e. let other processes read from, write to, and even delete the file at will.  That seems very much like a NOLOCK to me.

  35. fraursenohafe says:

    Hey:) i have been observing the forum lots of times – Thought I would say how much i enjoy visiting.

  36. John3058 says:

    Mr Russinovich,

    If everyone would give you a cent every time they used ProcessExplorer…  

    I get this message (or a similar one, didnt pay attention) sometimes when I start a PC but log on by using Remote Desktop Connection from another PC. Never bothered to investigate, though, just restart, log on again and Windows loads my profile with my customized desktop and all.

  37. stefang says:

    @Paul

    No, the difference is that with the current windows sharing semantics, one program can prevent any other programs from accessing a file just by opening the file in excusive mode – denying all other processes the possibility to open the file even for reading.

    If program A has a file opened exclusively, it is impossible for program B to open the file – it does not matter what sharing options program B specifies. Even worse, if program B has the file open, program A will be unable to open the file because it requires exclusive access. So it is currently not possible for a windows program to open a file in a way that is fully transparent to other programs.

    With my suggested NOLOCK option, program B would be able to specify that it wanted to open the file even if other programs had the file open for exclusive access.

  38. Andy Helsby says:

    I’m facing this issue at the moment but I’m not using citrix – just rdp to a Windows2003 terminal server. I strongly suspect it’s the Symantec Antivirus 10 client that is doing this due to the messages that uhpclean fires up.

    I’ve upgraded to v11 of Symantec to see if this fixes it – if not, then I may be able to look at this information to try and get an idea of what is going on but I’m not sure how to fix it!

  39. Andy Helsby says:

    Oh – I meant to say that adding that user environment debug log didn’t work for me.

  40. Jack says:

    Hey, I’m a programmer from way way back. HP1000 systems and HPs RTE OS. That ages me pretty good.

    Anyway, to get around all this file locking nonsense, back than we also had a file system implementation similar to Windows, we just located the file by reading the dirctory

    and accessed the data by reading the raw disc sectors through a supported OS EXEC call.

    Thus avoiding the file system all together to get at what was needed around about ways.

    Yhis must be possible in Windows as well. Rhus this NOLOCK issue can be solved with a little finguer work.

    Pardon a comment from a retired old timer.

  41. Andy Helsby says:

    ok – for some reason my log is working now. For what it’s worth – enabling the boot logging will eat up your disk space faster than a competitor at a hot dog eating competition. My C drive lost 10gb of space in about an hour. Unfortunately my problem happens sometime overnight after a reboot so it’s impractical to use this log function 🙁

    I am running procmon based on the ntuser.dat file for my user so I will be able to at least see what programs access ntuser.dat

  42. @Andy Helsby: of course, the amount of space consumed by boot logging depends on system activity, and duration of capture.  I wonder if scheduling procmon to run with a scheduled task at some point may be an option – you could set the filter as desired, choose the option to Drop Filtered Events, and perhaps log to a backing file on a different drive…  Pop into the Process Monitor forum at http://forum.sysinternals.com/forum_topics.asp?FID=19 for specific questions, etc.

  43. Andy Helsby says:

    Thanks Molotov – after 1hr20minutes of purely watching for activity on a specific users ntuser.dat the server crashed and rebooted 🙁

    I’ll drop into the forum when i have some more info – thanks for the help and pointers.

  44. fraursenohafe says:

    Hullo  i’ve been visiting the website for awhile  Finally needed to say how much i treasure this place:)

  45. BillG says:

    Its a blog, not a book, stop picking on typos people.

  46. Alpesh Patel says:

    I have the same issue and I think it it Symantec that causing it. Similar to Andy.

    I reached out the Symantec and they don’t have any idea. I don’t have version 11 to upgrade and test.

    Workaround for me is to install UPHClean-Setup.msi

  47. jay says:

    I’ve seen the same:

     unable to mount user hive

       => temporary profile

        => user very confused

    On four machines now.

    I also used boot logging to track it down.

    But I didn’t look at the call stacks.

    In my cases it was Live Mesh and uninstalling it fixed it. Perhaps it was also the prefetcher though I believe two of the machines were running Vista, one Server 2008, one XP, and the article says this work better in Vista..

  48. Madster says:

    At my college we have a lab full of WinXP.

    We could never access our profiles, always getting the temps.

    Admins just shrugged. We just figured profile support was really bad under WinXP. This for abour 3 years and ongoing.

  49. userno001948311 says:

    This kind of "error" had happened to me after I have installed and uninstalled my Kaspersky IS 2009 for many times.

  50. Anteaus says:

    A lot of these issues stem from the fact that we’ve lost the plot with desktop PC’s, in that we’ve forgotten what the P stands for -and that is IS a P, and not M for mainframe.

  51. terry says:

    It does look like winlogon is saying "something else (prefetch) has the file open, so I can’t get exclusive access to it … fatal error".

    I suppose this wouldn’t happen if prefetch used something like VSS to access the data, so the file doesn’t appear to be in use at all to regular processess.

    I’m surprised that prefetch is trying to read ntuser.dat at all. You’d think ntuser.dat (and possibly other files involved in authentication and policy enforcement, or which are data files that might change regularly) would need to be permanently exempted from prefetch.

  52. Damon says:

    Like several people that have posted here I want to know why Microsoft did not come up with a prefetching fix that prevent this type of sharing violation in XP?  

    In this article Mark does a great job of explaining what’s going on wtih Citrix and how their company developed a fix for their client however what about all the other applications that may have this problem on XP?

    I work for a company that has 1000’s of computers and we support about 2500 applications, migrating to a later OS is not an easy option and can take a over year to do so.  We see this problem on a daily basis however it is very difficult to predict when and what computer it will occur so setting up the procmon to capture the event has proven very difficult.  Setting up procmon after it has happened will not work because the computer requires a reboot for boot logging to begin and rebooting casues the problem to go away.

    Unless someone knows how to force the procmon to begin boot logging by only logging out and logging back on?

  53. Kantoorpand of Bedrijfspand huren says:

    I have the same issue and I think it it Symantec that causing it. Similar to Andy.

    I reached out the Symantec and they don’t have any idea. I don’t have version 11 to upgrade and test.

    Workaround for me is to install UPHClean-Setup.msi

    This is not working for me. Any more ideas?

    Thanks a lot.

  54. Ashish Sharma says:

    Thanx Mark for sharing this with us.

  55. Josh says:

    Citrix’s fix is such a hack. It add ten seconds to user logon because they made the choice to start a process in a Network Provider dll. Why not just start a thread in the dll that waits ten seconds then starts the Ssonsvr.exe process?

  56. Josh says:

    N/M after thinking about it more I realized my above solutions won’t work. It’s still a hack, though, IMO. While I assume they have very good reasons to do so, the primary problem still appears to me to be the fact that they’re using a Network Provider to launch a process at boot time. Rather than hanging boot time (unless I am mistaken that it does), before Ssonsvr.exe stops it should delete its own prefetch file as suggested unless there’s another way to disable prefetch for a single file/process.

  57. SFNR1 says:

    We had a similar problem with server 2003 x64 (terminalserver). one user gets a lock on ntuser.at. funny thing was that over night the lock was gone and 24 hours later the lock was there again. the prefetch was already disabled, the userprofile was a local one, etc. i captured the whole day/night with procmon and searched through millions of events. finally it was the "wmiprvse.exe" which was started by the Windows Management Instrumentation service. I disabled that (i think the Windows Firewall needs that), nothing, the lock was still there. After a reboot i captured again and voilà, the lock was gone and it didn’t come back.

    wmiprvse.exe was going through all userprofiles and was touching the ntuser.dat (ask god or bill gate why!). The third time it does that on this one profile, it gets a sharing violation and the profile is locked.

    Thanks Mark for that great bunch of tools making my life much easier :-).

  58. Jon Heal says:

    This issue also occurred on one of our machines with GoogleUpdater enabled. Google’s updating process is especially insidious as in order to fully disable it, one must disable services, delete registry entries and disable or delete scheduled tasks.

    Plus it contributes to this Temp Registry Profile problem! Garbageware.

  59. Loni Hamilton says:

    i wish someone would respond to me.  i have info of all thats going on.  i have  gone through all these new problems due to the worm i been fighting since 2008 aug.  the main worm is still untouched.  and i am still spreading it and cant do anything about it.  this registry key situation started off when i tried to block the hacker at the beginning, when i was winning, he changed it so everything said access denied, and later while loggin in, would give himself my status and degrade me 1 step lower.  the keys on my machine would be blocked, and at times, would block me from being able to log on to the computer.   and i formated so many times.

    this worm seems to be above boot, possibly device firmware alterations and bios.  anything after that would be beyond me.  i been fighting since aug 2008 and still cant rid the worm and want someone to help.  

    i am affecting computers 24/7 using pings that i cant even  touch.  help….

    this worm is independant of the operating system.  and even a low level formating with all devices taken out that emit signals, and new restore disk, and all the extras.  when doing all this, the worm is affecting the computer even still.  any machine brought into my house gets infected.  can anyone help…

    and dont tell me to use antivirus, i tried all variations of all antiviruses, firewalls, onestop, and all others.  they all dont detect.  i have info allowing me to know that the other worms were decoys.  no one is responding.    if all else, send this to someone who can investigate or at least want to hear the details i have of whats going on.

  60. Dion Arap says:

    @Loni Hamilton

    "Get help from Microsoft

    Get online support (or toll-free telephone support in the U.S. and Canada) for security-related issues such as viruses and security updates."

    http://www.microsoft.com/athome/security/protect/support.mspx

  61. andreas says:

    I have this problem too at my computer at home. If my wife has been logged on to her account and I log on to mine this happens ~50% of the time.

    ASFAIK we don’t have any Citrix stuff installed so there’s no fix for us. It’s a bit weak by MS to not solve it properly themselves, since it’s obviously a bug in how the cache manager interacts with the rest of the system at logon.

    At least now I know how to find out what application the prefetcher will think was associated with the registry read so I could write a script tot delete those pf-files at logoff.

  62. MattW says:

    Several comments suggested a flaw in the file system regarding access sharing and file locks and targeted the Logical Prefetcher (LP) as the problem.  However, these seem to miss the point, as the LP already utilizes read-only access and specifies full sharing.  The problem in this case really lies with Winlogon because it requires exclusive access.

    The article does identify a weakness of the LP in that 1) it cannot distinguish between simultaneous System processes that are non-related (such as the cached IO file operations) from those initiated by the launching process and 2) future LP activity can initiate unnecessary reads, specifically those from the previous unrelated System processes, which could cause collisions with Winlogon.  The LP has been modified in Vista and Windows 7 to work around this issue as it pertains to the LP.

    I wonder if the LP can be reengineered to only associate the simultaneous System processes that are actually initiated or indeed related to a launching process and also if Winlogon should be modified in a way to play more nicely?  In the meantime, using the tools and debugging techniques outlined can help identify troublesome startup processes and a log-off script deleting the offending prefetch files can create a work around.

    This is a very interesting and informative article, none the less.

  63. Erika says:

    We encounterd a similiar problem with our Antivirus. The Realtime Scanner accesses all ntuser.dat when the computer starts. When the user logs on quickly, nothing is reported. When the user logs on after some time, there is a sharing violation from Rtvscan.exe on the users ntuser.dat.

    When the user logs on exactly when Rtvscan access his ntuser.dat, the Temporary Registry Profile problem occurs.

    I sent procmon.log to the MS Engineers, but in my opinion they where not able to interpret the log file…they should be more engineers like Mark understanding the Windows logon process…

  64. YUSPINO says:

    Yes, yes. The google Updater is the cause. I have this problem with Windows 7. when uninstalling Google updater I could go to a "termporary profile" normally in real time.

    Thanks.

    Mark superhero.

  65. Kemp says:

    I can’t believe no one has mentioned this yet… Was the official solution (or workaround) to this problem in the new client really to just sit there delaying the user’s boot for 10 seconds while nothing happens? As someone who was forced for several years to use a networked Windows system that had hideously slow logins (we’re talking several minutes minimum and up to 10-15 minutes and sometimes more during peak times), I shudder to think that software may be deliberately delaying this to an even further extent.

  66. Kemp says:

    Or did I read it wrong and it’s the program in the new thread which waits 10 seconds? Still seems a bit of a kludge because on systems where the login has different timing, this could cause services provided by the app to be unavailable for a few seconds after login apears to have finished. Fine for a human who can’t react that fast, but I wonder how long it’ll take for a company to file a bug along the lines of "Our script which does X and Y after [or during] boot using operations enabled by your software no longer works on 30% of our computers."

  67. Karl Kiniger says:

    OK, just captured a similar race between spoolsv and winlogon on XP embedded.

    This has been  giving us major headaches alreay so I am glad I came over this thread to find the root cause. On our systems about 1 of 20 boots are failing because of this.

    grrr..

    Karl

  68. Ian W. Rudge says:

    This was written to provide a means of reassociating profiles which have become detached from their accounts:  

    http://iwrconsultancy.co.uk/software/reprofiler/reprofiler.htm

    I’ll admit that at this early stage of development there are still one or two scenarios which it can’t handle. That said it does work well enough to get most users out of dissociated-profile trouble.

  69. foxyshadis says:

    @Karl

    Sounds like you need to clean out your printer drivers and start reinstalling them until you find out which one is breaking logon.

    @all

    The vast majority of these problems are caused by third parties hooking the logon in various ways. Microsoft can only be faulted for giving them the ability to shoot themselves in the foot, but they’re still to blame for actually wrecking your user experience; blaming Microsoft for lack of foresight on the first implementation of a feature is pointless. XP & 2003 are long since out of mainstream support. The point of this blog is to show how you would determine which third-party component is doing it, so you can either disable it or make an informed bug report the developer.

  70. Matthew says:

    Interesting artical and thanks for sharing!