IMPORTANT ANNOUNCEMENT FOR OUR READERS!
AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!
We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!
Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.
If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.
NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!
As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!
********** UPDATE **********
This is now fixed in the following updates:
For Windows 8.1, 2012 R2, 2012 install:
- KB3132080 Logon freezes after you reset your password in Windows 8.1, or Stop error 0x1000007e in Windows Server 2012 R2: http://support.microsoft.com/kb/3132080/EN-US
For Windows 10 TH2 build 1511 install:
- KB3135173 Cumulative update for Windows 10 Version 1511: February 9, 2016: http://support.microsoft.com/kb/3135173/EN-US
Hi, Linda Taylor here, Senior Escalation Engineer from the Directory Services team in the UK.
I have been working on this issue which seems to be affecting many of you globally on windows 8.1, 2012 R2 and windows 10, so I thought it would be a good idea to explain the issue and workarounds while we continue to work on a proper fix here.
The symptoms are such that after a password change, logon hangs forever on the welcome screen:
The underlying issue is a deadlock between several components including DPAPI and the redirector.
Why does it happen?
So far we have seen this issue in the following circumstances (always after a password change/reset which is done somewhere other than the users machine – i.e. on the DC or in a portal)
1. If the user has a home drive which maps to a DFS like path for example: \\contoso.com\homefolders\user1
2. If The following GPO is applied:
Computer configuration\administrative templates Windows Components\File Explorer\ “Set a default associations configuration file”
And the XML file is stored on a DFS based path. For example \\contoso.com\netlogon
3. If you use GPP to map drives during logon to a DFS path like \\contoso.com\someShare
This issue happens due to a deadlock between DPAPI, Credential manager and the Redirector (RDR).
It goes like this…
1. When the user logs on, the profile service tries to map network home folder to \\contoso.com\…
2. To do this, we need to have a call created in RDR, and this requires a SMB session setup to dcname.contoso.com
3. The SMB session setup requires a security blob created to authenticate with the target server, which is the DC.
4. To create the security blob, Kerberos will check saved credentials by calling DPAPI.
5. DPAPI cannot decode the saved credential because the master key is not available because the user’s password is reset on DC, so it will need to query the DC for a master key. This requires a named pipe call to \\dcname.contoso.com\IPC$\protected_storage
6. To connect to this named pipe, RDR found it is the same as previous call in#2 (same fqdn DC name \\dcname.contoso.com) so now session setup is queued…
7. The Kerberos thread will hang forever, and the profile service will hang forever until a reboot.
8. After reboot, the user still cannot logon with the same symptom. (note: a different user CAN log on).
The problem occurs on client computers with Windows 8.1, Windows 10 and also Windows Server 2012 R2 (for example RDS scenario).
The problem occurs most frequently after an admin password reset which has occurred elsewhere (not the on the client computer to which the logon is happening) but it can also occur when the password change is not recent if the user is logging onto a machine where the cached credentials are old and they have changed their password on some other machine some time ago.
So, what can you do to get out of this problem?
There are several options for working around the issue:
1. If you have the mentioned policy – move the XML to some file share which is not DFS based and is not on a DC.
2. Assuming no-one wants to change home drive paths because there are many users and it’s a hassle, the other option is to disable Credential manager and clear the cached credentials.
When there are no entries for credential manager, there are no reasons to access the DC to refresh the keys. Hence the dead-lock.
This could be done by disabling this policy:
3. Dig your way out by connecting to the machine remotely and deleting the entry under C:\Users\<username>\AppData\Roaming\Microsoft\Protect\[Problem Users SID].
Noteif there is a roaming profile you can also delete this entry on the profile server and it works. This is because the profile is downloaded before the drive mapping takes place.
4. Another (not so great) option is to boot the PC without any network cable and log on with the old password. Then connect it back.
This issue also has a long history and there are other variant’s of this deadlock which were fixed before. See below list of related fixes:
Related previous Fixes for Windows 8.1 an Windows 2012 R2:
|Windows 8.1 & 2012 R2 KB#
|X64 file versions for dpapisrv.dll and lsasrv.dll
|https://support.microsoft.com/en-us/kb/3101183 “You can’t log on to a domain-joined computer in Windows 8.1 or Windows Server 2012 R2”
Released in October 2015.
Note: this is the latest but does not resolve this variant of the issue.
LSASRV.dll version 6.3.9600.18088.
|3038562 Cannot access DPAPI data after an administrator resets your password on a Windows Server 2012 R2-based domain controller
|The pre-requisite to 3101183 and 3038562 is April update:
2919355 Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 Update: April 2014
This contains win8.1 version of KB2927267 https://support.microsoft.com/en-us/kb/2927
267 “You cannot log on to Windows after the admin changes your password”
|Updates LSASRV.dll 6.3.9600.17042
There are no previous fixes for Windows 10.
There are some related past fixes for similar deadlocks in windows 8 also:
|Windows 8 KB#||X64 file versions for dpapisrv.dll and lsasrv.dll|
|3084956 You can‘t log on to a domain-joined computer in Windows 8 or Windows Server 2012
http://support.microsoft.com/kb/3084956/EN-US– Released September.
(NOTE: This is the windows 8 equivalent of KB3101183.)
|X64 file versions:
DPAPISRV.dll 6.2.9200.21645 and LSASRV.dll Version 6.2.9200.21582
|KB3049843 https://support.microsoft.com/en-us/kb/3049843 – You cannot access DPAPI data after an administrator resets your password on a Windows Server 2012-based domain controller||Dpapisrv.dll 6.2.9200.21442|
|2927267: https://support.microsoft.com/en-us/kb/2927267||Lsasrv.dll 6.2.9200.20931|
Finally, Microsoft is actively working on fixes for Windows 8.1 / WS 2012 R2 and Windows 10 TH2 for the current issue described above. I will share the release dates and article #’s once known. If you experience this issue please ensure you have all of the above fixes in place and use the workarounds noted above and keep an eye out on updates to this blog.