The Case of the Random DFS Access Denial

Hi, this is Bobby from the Microsoft Directory Services support team. Today we are going to discuss some confusing behavior that I’ve observed regarding inconsistent DFS access. While the root of this is issue is easy to remediate, it can be difficult to find the offending servers.

Symptoms

When first presented with this issue, I was asked to help determine the nature of users getting prompted to connect to a DFS root. DFS roots are defined in “How DFS Works as”

The starting point of the DFS namespace. The root is often used to refer to the namespace as a whole. A root maps to one or more root targets, each of which corresponds to a shared folder on a separate server. The DFS root must reside on an NTFS volume. A DFS root has one of the following formats: \\ServerName\RootName or \\DomainName\RootName.

The dialog was as follows:

Prior to calling Microsoft support, the customer was able to successfully connect to the NETLOGON and SYSVOL share of the domain without issue (\\contoso.com\sysvol and \\contoso.com\netlogon) Rebooting the client machine or having the client logoff would at times alleviate the problem. After a random period of time, users could access the DFS root, would later lose access, and users who did not work before would start working.

Investigating the issue, it was noticed that the "Distributed File System" MMC would change as time passed. For reference, I have included a picture of what it should look like when all is ok.

Using the "Distributed File System" MMC to check the status of the root would provide differing results at different times.

In this picture, it appears that three of the four root targets are inaccessible. When the MMC was checked later (or the user logged off and then back on) the results changed:

When using the “DFS Management” MMC from 2003 R2, The display is different, but the behavior is the same:

Initial Troubleshooting

When we began troubleshooting the issue, we started to look at the output of “dir” and dfsutil /pktinfo. This was to test the access to the DFS root and connectivity to the root shares.

At this point, each target appeared accessible. This was an unexpected result. I would have expected that we would get access denied to all of the targets. If we could reach each target, why were getting random “access is denied” when attempting to get a directory listing of the root?

We used the dfsutil /pktflush command to purge the referral cache thus forcing a request for new referrals. Then we ran the dir commands again that returned different results:

Using Netmon, we took a trace to see what we can find.

The first thing we attempted was to request a DFS referral for \\contoso.com\root

We were returned four targets for the root.

This was represented in the pktinfo as follows:

At this point we attempted to create a connection to the first server returned in the referral.

However we were returned: Status_access_denied.

Normal troubleshooting would be to attempt to connect to the share \\2003-02\root. In this case, we executed dir /b \\2003-02\root.

When we attempt to do this, we received more unexpected results:

I would have actually expected this to get an “access is denied” too.

Looking further in the network trace, we noticed that we received a new referral when we attempted to connect to 2003-02 server directly. To illustrate this, I have several screenshots of the referral network traffic.

The DFS service returned the following (note that ‘TargetPath: Index:1’ is not 2003-02 but another server):

Again the pktinfo:

When connecting to the path \\2003-02\root, we were instead accessing the root share on 2003-01, as detailed in the trace:

Cause and Remediation

When connecting to a root DFS share, the DFS service returns referrals for all of the participants on the root target set. So when we executed the command dir \\2003-dc1\root, the DFS service on 2003-dc1 then responded with a new DFS root referral for the path \\2003-dc1\root.  This behavior accounts for the entries in the referral cache for paths to each root (i.e. \\2003-dc1\root, \\2003-01\root, \\2003-02\root, \\2003-03\root). DFS referrals are retuned in a random order for servers in the site of the client. This will cause differencing results for each time the client requests a referral.

The root cause of the issue was actually incorrect share permissions on the “root” share of 2003-02. This can be observed locally using net share sharename

Comparing this to the same output of 2003-01, we see the problem:

In small environments, connecting to the server and examining the share permissions is an acceptable method, however in large environments this can prove cumbersome. I prefer to use subinacl to verify the permissions remotely. (Subinacl can be downloaded here: https://www.microsoft.com/downloads/details.aspx?FamilyID=E8BA3E56-D8FE-4A91-93CF-ED6985E3927B&displaylang=en)

To fix this we could connect to the server and add the permissions. Or it could be altered remotely with the following:. The syntax is : subinacl /share \\2003-02\root /grant=everyone=r

Note: Read is the default share permission in 2003 and later. Your permissions may be different based upon your business needs.

Using DFSGUI.MSC to check the status of the DFS root targets has been deprecated and is no longer available in 2008. This however does not change the behavior of the referrals that are returned.

Using DFSdiag.exe from Windows 2008 does not return any additional information as to the root cause.

Further reading

For more information about the technologies discussed here, please refer to “How DFS Works” https://technet.microsoft.com/en-us/library/cc782417(WS.10).aspx

- Bobby ‘Bucket Head’ McMillan