Demo details for my TechEd 2014 presentation on File Server Networking

During my TechEd 2014 presentation on File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2, I did a demo showing a Scale-Out File Server Cluster. The demo consisted of showing the networking details of this cluster, looking both from the cluster perspective and the client side. I was asked after the presentation to share the details of the PowerShell script that I used. So, here it goes.

 

First, here is the output of the script. It’s broken into a few parts:

  • It starts by showing the cluster configuration (cluster nodes, cluster networks, cluster resources and cluster shared volumes)
  • Note that the name of the cluster itself is JOSE-S and the name of the Scale-Out File Server is JOSE-SO.
  • Then it shows the IP addresses on each node, plus what’s registered on DNS for the scale-out file server name.
  • Next, it outputs the SMB server network information, with focus on the scopes defined on the server.
  • Then we switch to the client view of connections, after just showing the directory from the 3 shares created. This shows that multichannel won’t engage until you have a read or a write.
  • Finally, after running a quick workload on the shares, it shows the full set of 24 connections (3 shares * 2 NICs * 4 connections per NIC).
  • The 10.1.x.100 IP addresses are associated with the cluster name used for cluster management. They don't relate directly to the Scale-Out File Server.
  • The scope name starting with "FE80" is the internal cluster scope used for inter-node communication.

 

Cluster Information

PS C:> Get-ClusterNode -Cluster JOSE-S

Name ID State
---- -- -----
JOSE-S1 2 Up
JOSE-S2 1 Up
JOSE-S3 3 Up

PS C:> Get-ClusterNetwork -Cluster JOSE-S

Cluster Name State Address Role
------- ---- ----- ------- ----
JOSE-S Cluster Network 1 Up 10.1.2.0 3
JOSE-S Cluster Network 2 Up 10.1.1.0 3

PS C:> Get-ClusterResource -Cluster JOSE-S

Name State OwnerGroup ResourceType
---- ----- ---------- ------------
Cluster Disk 1 Online Cluster Group Physical Disk
Cluster IP Address Online Cluster Group IP Address
Cluster IP Address 10.1.1.100 Online Cluster Group IP Address
Cluster Name Online Cluster Group Network Name
JOSE-SO Online JOSE-SO Distributed Network Name
Scale-Out File Server (\JOSE-SO) Online JOSE-SO Scale Out File Server

PS C:> Get-ClusterSharedVolume -Cluster JOSE-S

Name State Node
---- ----- ----
Cluster Disk 2 Online JOSE-S2
Cluster Disk 3 Online JOSE-S1
Cluster Disk 4 Online JOSE-S3

IP Addresses for each node, DNS for Scale-Out Name

PS C:> Get-NetIPAddress -CimSession <node>

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S1 12 10.1.1.100 24
JOSE-S1 12 10.1.1.2 24
JOSE-S1 14 10.1.2.100 24
JOSE-S1 14 10.1.2.2 24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S2 12 10.1.1.3 24
JOSE-S2 14 10.1.2.3 24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S3 12 10.1.1.4 24
JOSE-S3 14 10.1.2.4 24

PS C:> Resolve-DNSName JOSE-SO

Name Type TTL Section IPAddress
---- ---- --- ------- ---------
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.1.3
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.1.2
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.1.4
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.2.3
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.2.4
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.2.2

IP Addresses on each node per Scope name

PS C:> Get-SmbServerNetworkInterface -CimSession JOSE-SO

PSComputerName ScopeName IPAddress
-------------- --------- ---------
JOSE-S1 FE80::9CBF:78F5:DFA8:803B 10.1.2.2
JOSE-S1 FE80::9CBF:78F5:DFA8:803B 10.1.1.2
JOSE-S1 JOSE-S 10.1.2.100
JOSE-S1 JOSE-S 10.1.1.100
JOSE-S1 JOSE-SO 10.1.2.2
JOSE-S1 JOSE-SO 10.1.1.2

PSComputerName ScopeName IPAddress
-------------- --------- ---------
JOSE-S2 FE80::69AF:5813:D729:CEFB 10.1.1.3
JOSE-S2 FE80::69AF:5813:D729:CEFB 10.1.2.3
JOSE-S2 JOSE-SO 10.1.2.3
JOSE-S2 JOSE-SO 10.1.1.3

PSComputerName ScopeName IPAddress
-------------- --------- ---------
JOSE-S3 FE80::F4BE:F77B:7EAC:6CED 10.1.1.4
JOSE-S3 FE80::F4BE:F77B:7EAC:6CED 10.1.2.4
JOSE-S3 JOSE-SO 10.1.2.4
JOSE-S3 JOSE-SO 10.1.1.4

Simple connection just to check directory

PS C:> Dir \JOSE-SO<share>

    Directory: \JOSE-SOShare1

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/4/2014 5:40 PM 1717986918 testfile.dat
4

    Directory: \JOSE-SOShare2

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/4/2014 5:40 PM 1717986918 testfile.dat
4

    Directory: \JOSE-SOShare3

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/4/2014 5:40 PM 1717986918 testfile.dat
4

SMB Connection

PS C:> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO Share1 3.02 True JOSE.TESTAdministrator
JOSE-SO Share2 3.02 True JOSE.TESTAdministrator
JOSE-SO Share3 3.02 True JOSE.TESTAdministrator

TCP connections used by the client

PS C:> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5 10.1.1.2 49453 445
10.1.1.5 10.1.1.3 49470 445
10.1.1.5 10.1.1.4 49461 445

Putting some load on the shares

PS C:> C:SQLIOSQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>
sqlio v1.5.SG
using system counter for latency timings, 10000000 counts per second
8 threads reading for 10 secs from files \JOSE-SOShare1testfile.dat, \JOSE-SOShare2testfile.dat and \JOSE-SOShare3testfile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 16 outstanding
buffering set to not use file nor disk caches (as is SQL Server)
using current size: 16384 MB for file: \JOSE-SOShare1testfile.dat
using current size: 16384 MB for file: \JOSE-SOShare2testfile.dat
using current size: 16384 MB for file: \JOSE-SOShare3testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 26822.20
MBs/sec: 209.54
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 13
Max_Latency(ms): 171
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 1 2 4 6 8 8 9 8 8 7 6 5 5 4 3 3 2 2 1 1 7

SMB Connection

PS C:> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO Share1 3.02 True JOSE.TESTAdministrator
JOSE-SO Share2 3.02 True JOSE.TESTAdministrator
JOSE-SO Share3 3.02 True JOSE.TESTAdministrator

SMB Multichannel Connection

PS C:> Get-SmbMultichannelConnection

ClientIPAddress ServerIPAddress CurrentChannels
--------------- --------------- ---------------
10.1.1.5 10.1.1.2 4
10.1.1.5 10.1.1.3 4
10.1.1.5 10.1.1.4 4
10.1.2.5 10.1.2.2 4
10.1.2.5 10.1.2.3 4
10.1.2.5 10.1.2.4 4

TCP connections used by the client

PS C:> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5 10.1.1.2 49511 445
10.1.1.5 10.1.1.2 49513 445
10.1.1.5 10.1.1.2 49516 445
10.1.1.5 10.1.1.2 49480 445
10.1.1.5 10.1.1.3 49518 445
10.1.1.5 10.1.1.3 49497 445
10.1.1.5 10.1.1.3 49512 445
10.1.1.5 10.1.1.3 49515 445
10.1.1.5 10.1.1.4 49514 445
10.1.1.5 10.1.1.4 49517 445
10.1.1.5 10.1.1.4 49492 445
10.1.1.5 10.1.1.4 49509 445
10.1.2.5 10.1.2.2 49498 445
10.1.2.5 10.1.2.2 49501 445
10.1.2.5 10.1.2.2 49508 445
10.1.2.5 10.1.2.2 49505 445
10.1.2.5 10.1.2.3 49500 445
10.1.2.5 10.1.2.3 49510 445
10.1.2.5 10.1.2.3 49506 445
10.1.2.5 10.1.2.3 49503 445
10.1.2.5 10.1.2.4 49507 445
10.1.2.5 10.1.2.4 49502 445
10.1.2.5 10.1.2.4 49499 445
10.1.2.5 10.1.2.4 49504 445

  

Next, here’s the PowerShell script behind what was shown above. Here are a few notes on the script:

  • It shows some text with a simplified view of the cmdlet about to be run, but the actual cmdlet is usually a bit more complex since it usually includes additional parameters and some formatting.
  • I tried to use some color and extra blank lines so things would be easier to read.
  • The “read-host” cmdlets are used to pause the script until I press enter to continue to the next phase.
  • I pipe the results to “Out-Host” to make sure everything shows in proper order on the screen (without it, some of the longer running cmdlets might show out of order).
  • The script is run from a client, so cmdlets that target the cluster use the –Cluster parameter and cmdlets that target a specific server use the –CimSession parameter.
  • SQLIO is used as a workload generator. For more details, see this other blog post on using SQLIO.
  • I make a point of using the PowerShell cmdlets for network information (Resolve-DnsName instead of NSLOOKUP.EXE, Get-NetTcpConnection instead of NETSTAT.EXE).

 

Cls

Write-Host -ForegroundColor Green "Cluster Information"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterNode -Cluster JOSE-S"
Get-ClusterNode -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterNetwork -Cluster JOSE-S"
Get-ClusterNetwork -Cluster JOSE-S | FT Cluster, Name, State, Address, Role -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterResource -Cluster JOSE-S"
Get-ClusterResource -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterSharedVolume -Cluster JOSE-S"
Get-ClusterSharedVolume -Cluster JOSE-S | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses for each node, DNS for Scale-Out Name"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-NetIPAddress -CimSession <node>"
"JOSE-S1", "JOSE-S2", "JOSE-S3" | % { Get-NetIPAddress -CimSession $_ -AddressFamily IPv4 -IPAddress 10.1.* | Sort IPAddress | FT PSComputerName, ifIndex, IPAddress, PrefixLength -AutoSize } | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Resolve-DNSName JOSE-SO"
Resolve-DnsName JOSE-SO | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses on each node per Scope name", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-SmbServerNetworkInterface -CimSession JOSE-SO"
1..3 | % { Get-SmbServerNetworkInterface -CimSession JOSE-S$_ | ? ScopeName -ne "*" | Sort ScopeName | FT PSComputerName, ScopeName, IPAddress -AutoSize } | Out-Host

Read-Host

Write-Host -ForegroundColor Green "Simple connection just to check directory", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Dir \JOSE-SO<share>"
Dir \JOSE-SOShare1testfile.dat, \JOSE-SOShare2testfile.dat, \JOSE-SOShare3testfile.dat | Out-Host

Write-Host -ForegroundColor Green "SMB Connection", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort -AutoSize | Out-Host
Read-Host

Write-Host -ForegroundColor Green "Putting some load on the shares", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:> C:SQLIOSQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>"
C:SQLIOSQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN \JOSE-SOShare1testfile.dat \JOSE-SOShare2testfile.dat \JOSE-SOShare3testfile.dat

Write-Host -ForegroundColor Green "SMB Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "SMB Multichannel Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-SmbMultichannelConnection"
Get-SmbMultichannelConnection | Sort ClientIPAddress, ServerIPAddress | FT ClientIPAddress, ServerIPAddress, CurrentChannels | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort –AutoSize | Out-Host

 

You can find the slides and the recording for the presentation at https://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/DCIM-B337