O’DFS Shares! Where Art Thou? – Part 3/3
Hello, this is Sabin and Shravan from the Microsoft Directory Services Support team. We are back to cover the third and final part of this series to troubleshoot slowness/delay experienced by clients in accessing a DFS Namespace. As a recap, in Part 1 of this blog, we reviewed the referral process for Domain Based Namespaces wherein we illustrated a working scenario of DFS access. In Part 2 we troubleshot slow access of DFS Shares where user is seeing a delay DFS Servers are in the same site as the client.
In the third part of this series for troubleshooting slow access of DFS Shares, we will review a scenario where a user is seeing a delay while trying to access a DFS share and picks a DFS server outside its own site instead of the local DFS server. This is also referred as “Out of Site Referral”.
STEPS TAKEN:
We reproduced the problem (slow DFS access) and ran the following tools:
Step1: A referral cache view (dfsutil /pktinfo) for a slow DFS access issue:
dfsutil /pktinfo
2 entries...
Entry: \Ms2\rootdfsn\Data
ShortEntry: \Ms2\rootdfsn\Data
Expires in 1780 seconds
UseCount: 1 Type:0x1 ( DFS )
0:[\Ms1\Data] State:0x100 ( TARGETSET )
1:[\Ms2\DataReplica] State:0x110 ( ACTIVE TARGETSET )Entry: \contoso.com\rootdfsn
ShortEntry: \contoso.com\rootdfsn
Expires in 257 seconds
UseCount: 1 Type:0x81 ( REFERRAL_SVC DFS )
0:[\Ms1\rootdfsn] State:0x100 ( TARGETSET )
1:[\Ms2\rootdfsn] State:0x110 ( ACTIVE TARGETSET )DfsUtil command completed successfully.
Analysis:
1. In this case, as you can see the root referral (0x81) shows two TARGETSETs with one server as each indicating a cost boundary –
0:[\Ms1\rootdfsn] State:0x100 ( TARGETSET )
1:[\Ms2\rootdfsn] State:0x110 ( ACTIVE TARGETSET )
2. As we can see, MS2 is set to ACTIVE. Ideally, the client should have chosen the first root target (MS1) in the domain-based root referral, connect to the root server and navigate the subfolders under the root folder. On encountering a link folder, the root server should send a status message to the client, indicating that this is a link folder that requires redirection.
3. If the link referral is not in the cache, the Vista client should have connected to the IPC$ named pipe of the root server in the user’s context (or in the context of the LocalSystem account for pre-vista clients) and request a link referral from the root server which in turn returns a list of link targets.
4. However, as we can see, MS2 (second server in the list) is set as the ACTIVE server since the client was unable to connect and/or request a link referral from the first targetset i.e. MS1.
5. Note: It’s important to note here that the client spent some time trying to traverse the root referral list until it was able to get a link referral and finally get to the link target. This can cause and/or add to the delay in accessing a DFS share.
6. Finally, as we can see in the referral cache the client gets a link referral from MS2 and eventually accesses the target on MS2.
Question:
Why was the client not able to access MS1 DFS root?
Step 2: The following is an excerpt from dfsdiag /testdfsconfig /dfsroot:\\contoso.com\rootdfsn
Starting TestDfsConfig ....
Retrieving All the Root Targets ....Validating DFS Service ....
Validating DFS Service on MS1.
DFSDIAG_ERROR - SYS - The RPC server is unavailable.
Validating DFS Service on MS2. DFSDIAG_INFO - APPL - DFS Service on MS2 is OK.
Validating Registry Entries ....DFSDIAG_ERROR - SYS - The network path was not found.
DFSDIAG_WARNING - APPL - MS1's Registry not accessible;Ignoring this for comparison.
DFSDIAG_INFO - APPL - Not a single comparison occured due to errors.
Finished TestDfsConfig.
Analysis:
As evident from above, the DFS service on MS1 doesn’t seem to be running and/or we are not able to bind to this box due to RPC errors.
Step 3: Run dfsdiag /testdfsintegrity /dfsroot:\\contoso.com\rootdfsn /full
Starting TestDfsIntegrity ....
DFSDIAG_ERROR - SYS - The RPC server is unavailable.
DFSDIAG_ERROR - SYS - The specified domain either does not exist or could not be contacted. DFSDIAG_WARNING - APPL - Unable to access dfs metadata of \\MS1\rootdfsn.
Finished TestDfsIntegrity.
Analysis:
More evidence on accessing/binding issues with MS1 root server
Note: Alternatively, we could run dfsdiag /testreferral /dfspath:\\contoso.com\rootdfsn\data /full which collectively runs all the above tests.
Step 4: Try to access the DFS share on MS1 using NetBIOS and FQDN
\\ms1\data - Error: Unspecified Error
\\ms1.contoso.com\data - Error: The network path was not found
“Ping ms1.contoso.com” returns “Destination Host Unreachable”.
Step 5: Run a Netmon trace (network capture) while trying to access the DFS share from client
The following is an excerpt from a Netmon output for the same setup. Not all frames are shows as some have been removed for brevity.
35 VistaDFSClient DC1 DFS DFS:Get DFS Referral Request, FileName: \contoso.com\rootdfsn, MaxReferralLevel: 4
36 DC1 VistaDFSClient DFS DFS:Get DFS Referral Response, NumberOfReferrals: 2 VersionNumber: 4
Dfs: Get DFS Referral Response, NumberOfReferrals: 2 VersionNumber: 4
NumberOfReferrals: 2 (0x2)
DfsPath: \contoso.com\rootdfsn
DfsAlternatePath: \contoso.com\rootdfsn
TargetPath: Index:1 \Ms1\rootdfsn
TargetPath: Index:2 \Ms2\rootdfsn
40 VistaDFSClient MS1 ARP ARP:Request, VistaDFSClient(10.10.10.6) asks for MS1(10.10.10.3)
46 VistaDFSClient MS1 ARP ARP:Request, VistaDFSClient(10.10.10.6) asks for MS1(10.10.10.3)
47 VistaDFSClient MS1 ARP ARP:Request, VistaDFSClient(10.10.10.6) asks for MS1(10.10.10.3)
48 VistaDFSClient MS1 ARP ARP:Request, VistaDFSClient(10.10.10.6) asks for MS1(10.10.10.3)
83 VistaDFSClient MS1 ARP ARP:Request, VistaDFSClient(10.10.10.6) asks for MS1(10.10.10.3)
84 VistaDFSClient MS1 ARP ARP:Request, VistaDFSClient(10.10.10.6) asks for MS1(10.10.10.3)
114 VistaDFSClient MS2 SMB SMB:C; Tree Connect Andx, Path = \\MS2\ROOTDFSN, Service = ?????
115 MS2 VistaDFSClient SMB SMB:R; Tree Connect Andx, Service = A:
116 VistaDFSClient MS2 SMB SMB:C; Transact2, Query Path Info, Query File Basic Info, Pattern = \contoso.com\rootdfsn
117 MS2 VistaDFSClient SMB SMB:R; Transact2, Query Path Info, Query File Basic Info
120 MS2 VistaDFSClient SMB SMB:C; Tree Connect Andx, Path = \\MS2\IPC$, Service = ?????
121 MS2 VistaDFSClient SMB SMB:R; Tree Connect Andx, Service = IPC
241 VistaDFSClient MS2 SMB SMB:C; Transact2, Query Path Info, Query File Basic Info, Pattern = \contoso.com\rootdfsn\Data
242 MS2 VistaDFSClient SMB SMB:R; Transact2, Query Path Info - NT Status: System - Error, Code = (599) STATUS_PATH_NOT_COVERED
243 MS2 VistaDFSClient DFS DFS:Get DFS Referral Request, FileName: \Ms2\rootdfsn\Data, MaxReferralLevel: 4
244 MS2 VistaDFSClient DFS DFS:Get DFS Referral Response, NumberOfReferrals: 2 VersionNumber: 4
Dfs: Get DFS Referral Response, NumberOfReferrals: 2 VersionNumber: 4
NumberOfReferrals: 2 (0x2)
DfsPath: \Ms2\rootdfsn\Data
DfsAlternatePath: \Ms2\rootdfsn\Data
TargetPath: Index:1 \Ms1\Data
TargetPath: Index:2 \Ms2\DataReplica
245 VistaDFSClient MS2 SMB SMB:C; Tree Connect Andx, Path = \\MS2\DATAREPLICA, Service = ?????
246 MS2 VistaDFSClient SMB SMB:R; Tree Connect Andx, Service = A
ANALYSIS:
1. Packets 35-36 – Shows the DFS referral request from Vista client to domain controller DC1. Then DC1 comes back with a referral response and provides a list in the following order:
TargetPath: Index:1 \Ms1\rootdfsn
TargetPath: Index:2 \Ms2\rootdfsnNOTE: The referral order is important. As we can see, MS1 is the first server in the list followed by MS2.
2. Packets 40- 84 - Shows the Vista attempting an ARP request to access MS1 (first server in the list) and eventually failing over to MS2. Notice the delay identified by the second column in the parsed capture information.
3. Packets 114 -242 – Shows the client trying to connect to the IPC$ of root (MS2) and navigating shares before getting a STATUS_PATH_NOT_COVERED response (expected) from server indicating the share is a “link folder” as against normal share and requires a link referral.
4. Packets 243-244 – Shows a link referral request from client and MS2 responds with the following referral order:
TargetPath: Index:1 \Ms1\Data
TargetPath: Index:2 \Ms2\DataReplicaNOTE: As you can see, MS1 is the first in the list. However the client chooses MS2 due to previous failures accessing DFS on MS1.
5. Packets 245 – Shows the client accessing the \\MS2\datareplica (target replica for data on MS1) and the user navigates/accesses the files (file.txt) as needed.
Netmon 3.3 Tip: You can parse the network captures using a combination of the following filters:
SMB
DFS
IPV4.address == a.b.c.d && IPV4.address == w.x.y.z
{Where a.b.c.d and w.x.y.z are DFS client/Server IP addresses}
RESOLUTION:
Based on the above steps, we can clearly see that the client is unable to access MS1 via DFS and NetBIOS/FQDN. The traces also indicate the failure accessing MS1 and eventual failover to MS2 server causing the delay of about 40 seconds as evident in the information above. When we checked the services on MS1, the DFS and Server services were running. However as identified above with the ping test, there was an issue with routing to MS1 and once the network issue was resolved, the access to DFS on MS1 was working as expected.
This brings us to the end of our 3 part series. We hope that this blog help solidify your knowledge with DFS behavior and troubleshooting issues with slow access to DFS Shares when you run into them in your environment. Good Luck! Ciao!
-Sabin Nair and Shravan Kumar