Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Virtual)

This post is a part of the nine-part “ What’s New in Windows Server & System Center 2012 R2 ” series that is featured on Brad Anderson’s In the Cloud  blog. Today’s blog post covers Windows Server 2012 R2 Storage and how it applies to the larger topic of “Transform the Datacenter.” To read that post and see the other technologies discussed, read today’s post: “What’s New in 2012 R2: IaaS Innovations.”

 

1) Overview

 

In this document, I am sharing all the steps I used to create a Windows Server 2012 R2 File Server demo or test environment, so you can experiment with some of the new technologies yourself. You only need a single computer (the specs are provided below) and the ISO file with the Windows Server 2012 R2 Preview available as a free download.

 

The demo setup includes 5 virtual machines: 1 domain controller, 3 file servers and 1 file client/VMM Server. It uses the new Shared VHDX feature to provide shared storage for the guest failover cluster and it showcases both Storage Spaces and Scale-Out File Servers.

 

If you're not familiar with these technologies, I would strong encourage you to review the TechEd 2013 talks on Hyper-V over SMB (Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance) and Shared VHDX (second half of Application Availability Strategies for the Private Cloud).

 

Here’s a diagram of the setup:

clip_image002

 

Here are the details about the names, roles and IP addresses for each of the computers involved:

VM Computer Role Net2 (DC/DNS)
Host JOSE-EW Hyper-V Host 192.168.100.99/24
VM1 JOSE-D DNS, DC 192.168.100.100/24
VM2 JOSE-A1 Cluster Node 1 192.168.100.101/24
VM3 JOSE-A2 Cluster Node 2 192.168.100.102/24
VM4 JOSE-A3 Cluster Node 3 192.168.100.103/24
VM5 JOSE-V VMM / Client 192.168.100.104/24
  JOSE-A Cluster 192.168.100.110/24
  JOSE-F Scale-Out​ File Server N/A

 

Following these steps will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:

  • Hyper-V
  • Networking
  • Domain Name Services (DNS)
  • Active Directory Domain Services (AD-DS)
  • Shared VHDX
  • Storage Spaces
  • Failover Clustering
  • File Servers
  • PowerShell
  • SQL Server
  • Virtual Machine Manager

 

2) Required Hardware and Software, plus Disclaimers

 

You will need the following hardware to perform the steps described here:

 

You will need the following software to perform the steps described here:

 

Notes and disclaimers:

  • A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
  • If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
  • There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
  • For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
  • Obviously, a single-computer solution can never be tolerant to the failure of that computer. So, the configuration described here is not really continuously available. It’s just a simulation.
  • The specific Shared VHDX configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where the Shared VHDX files are stored either on a Cluster Shared Volume (CSV) directly or on a file share on a Scale-Out File Server Cluster.
  • The specific Storage Spaces configuration shown in this blog post is not supported. Microsoft Support will only answer questions and assist in troubleshooting configurations where Storage Spaces uses a physical machine (not a VM) and uses one of the certified JBOD hardware solutions (see https://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1)
  • Because of the two items above, the configuration described here should only be used for demos, testing or learning environments.
  • The Storage Spaces configuration shown here is not capable of showcasing some of the new features like Tiering or Write-back Cache, since it uses virtual disks without a proper media type. To use those features, you’ll need a storage pool with physical SSDs and HDDs.

 

3) Summarized instructions for experts

 

If you are already familiar with Failover Clustering, Scale-Out File Servers and Hyper-V, here’s a short introduction to what it takes to configure this environment on a single computer. A detailed step-by-step is available in the following sections of this document.

  • Configure a Hyper-V capable computer with at least 8GB of RAM with Windows Server 2012 R2. Make sure to load the Failover Clustering feature. No need to actually create a failover cluster on the physical machine.
  • Manually attach the Shared VHDX filter to the disk where the VHDX files will be located:
    fltmc.exe attach svhdxflt D:\
  • Create OS VHD or VHDX files as you would regularly, on an SMB file share. Create your VMs as you would regularly would, using the OS VHD or VHDX file. Both Generation 1 and Generation 2 VMs are fine.
  • Create your VHDX data files to be shared as fixed-size or dynamically expanding, on the disk where you manually attached the Shared VHDX filter. Old VHD files are not allowed. Differencing disks are not allowed.
  • Add the shared VHDX data files to multiple VMs, using the Add-VMHardDiskDrive and the “-ShareVirtualDisk” option. If using the GUI, check the box for “Share virtual disk” when adding the VHDX data files to the VM.
  • Inside the VM, install Windows Server 2012 R2 or Windows Server 2012. Make sure to install the Hyper-V integration components.
  • Configure Storage Spaces using the Virtual SAS disks exposed to the VMs.
  • Validate and configure the cluster inside the VMs as you normally would. Configure a Scale-Out file server and create the file shares for testing.
  • Install VMM 2012 R2 Preview and use it to manage the Scale-Out File Server.

 

The rest of this document contains detailed step-by-step instructions for each of the items outlined above.

 

4) Configure the physical host

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restart at the end
Install-WindowsFeature Failover-Clustering -IncludeManagementTools
Install-WindowsFeature Hyper-V -IncludeManagementTools -Restart

# Create a new Internal Switch
New-VMSwitch -Name Net2 -SwitchType Internal
Get-NetAdapter *Net2* | Rename-NetAdapter -NewName vNet2

# Configure vNet2 (parent port for internal switch) with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias vNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias vNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias vNet2 -IPAddress 192.168.100.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias vNet2 -ServerAddresses 192.168.100.1

# Create 5 VMs, assumes there is a base VM ready at D:\VMS\BaseOS.VHDX
1..5 | % { New-VHD -Path D:\VMS\VM$_.VHDX –ParentPath D:\VMS\BaseOS.VHDX }
1..5 | % { New-VM -Name VM$_ -Path D:\VMS –VHDPath D:\VMS\VM$_.VHDX -Memory 2GB -SwitchName Net2 }
# Give the VMM VM a little more RAM
Set-VM -VMName VM5 -MemoryStartupBytes 4GB

# Create 3 data VHDX files
1..3 | % { New-VHD -Path D:\VMS\Data$_.VHDX -Fixed -SizeBytes 20GB }

# Manually attach volume D:\
fltMC.exe attach svhdxflt D:\
# NOTE 1: Non-CSV deployments of Shared VHDX are not supported. See the disclaimer section of this document.
# NOTE 2: Manual attached is not saved across reboots. You will have to re-issue the command after each reboot.

# Add the 3 data VHDX files to each of the 3 VMs, with Sharing option
1..3 | % { $p = ”D:\VMS\Data” + $_ + ”.VHDX” ; 2..4 | % { $v = ”VM” + $_; Write-Host $v, $p; Add-VMHardDiskDrive -VMName $v -Path $p -ShareVirtualDisk } }

# Start all the VMs
Get-VM | Start-VM

 

5) Configure JOSE-D (DNS, DC)

 

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature AD-Domain-Services, RSAT-ADDS, RSAT-ADDS-Tools

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.100 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# create AD forest, reboots at the end
Install-ADDSForest -CreateDNSDelegation:$false -DatabasePath "C:\Windows\NTDS" -DomainMode "Win2012" -DomainName "JOSE.TEST" -DomainNetBIOSName "JOSE" -ForestMode "Win2012" -InstallDNS:$true -LogPath "C:\Windows\NTDS" -SYSVOLPath "C:\Windows\SYSVOL"

 

6) Configure JOSE-A1 (Cluster A)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.101 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Create 1 Pool, 7 Spaces and Initialize them
# NOTE 3: Running Storage Spaces in a guest is not supported. See the disclaimer section of this document.
# NOTE 4: This unsupported configuration cannot simulate Tiering or Write-back cache, since there are no SSDs.

$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 1 -StoragePool (Get-StoragePool -FriendlyName Pool1)

New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 1GB
2..7 | % { New-VirtualDisk -FriendlyName Space$_ -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 3GB }

1..7 | % {
   $Letter ="PQRSTUV”[($_-1)]
$Number = (Get-VirtualDisk -FriendlyName Space$_ | Get-Disk).Number
Set-Disk -Number $Number -IsReadOnly 0
   Set-Disk -Number $Number -IsOffline 0
Initialize-Disk -Number $Number -PartitionStyle MBR
   New-Partition -DiskNumber $Number -DriveLetter $Letter -UseMaximumSize 
   Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

7) Configure JOSE-A2 (Cluster A)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.102 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart

8) Configure JOSE-A3 (Cluster A)

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.103 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

9) Configure Cluster JOSE-A

# Validate cluster
Test-Cluster -Node JOSE-A1, JOSE-A2, JOSE-A3

# Create cluster
New-Cluster –Name JOSE-A -Node JOSE-A1, JOSE-A2, JOSE-A3 -StaticAddress 192.168.100.110

# Rename and configure networks
(Get-ClusterNetwork).Name = “Internal”
(Get-ClusterNetwork).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1

# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"

# Add remaining disks to Cluster Shared Volumes
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume

# Create Scale-Out File Server
Add-ClusterScaleOutFileServerRole JOSE-F

# Create SMB shares
1..6 | % {
MD C:\ClusterStorage\Volume$_\Share
New-SmbShare -Name Share$_ -Path C:\ClusterStorage\Volume$_\Share -FullAccess JOSE.Test\Administrator
Set-SmbPathAcl -ShareName Share$_
}

 

10) Configure JOSE-V

# Preparation steps: Install WS2012R2, rename computer, enable remote desktop

# Install required roles and features, restarts at the end
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature

# Configure Internal NIC with a static IP address for DNS / DC
Get-NetAdapter | Rename-NetAdapter -NewName Internal
Set-NetIPInterface -InterfaceAlias Internal -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Internal -Confirm:$false
New-NetIPAddress -InterfaceAlias Internal -IPAddress 192.168.100.104 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Internal -ServerAddresses 192.168.100.100

# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart

# Map the SMB shares
1..6 | % { 
    $d ="PQRSTU"[($_-1)] + “:”
New-SmbMapping -LocalPath $d -RemotePath \\JOSE-F\Share$_ -Persistent $true
}

# Create the test files
1..6 | % { 
   $f ="PQRSTU"[($_-1)] + “:\testfile.dat”
fsutil file createnew $f (256MB)
   fsutil file setvaliddata $f (256MB)
}

# Configure SMB Bandwidth Limits to avoid overloading the virtual environment

Add-WindowsFeature FS-SMBBW
Set-SmbBandwidthLimit -Category Default -BytesPerSecond 16MB

# Run SQLIO (assumes SQLIO.EXE was copied to C:\SLQIO)

c:\sqlio\sqlio.exe -s9999 -kR -t1 -o16 -b8 -BN -LS -frandom -dPQRSTU testfile.dat

# To remove a node while the workload is running (wait a few minutes for rebalancing)
Remove-ClusterNode -Cluster JOSE-A –Name JOSE-A3

# To add a node while the workload is running (wait a few minutes for rebalancing)
Add-ClusterNode -Cluster JOSE-A –Name JOSE-A3

 

 

11) Running the tests without VMM (from JOSE-V)

Start Performance Monitor

Start a performance monitor session

Switch to “Histogram Bar” view to show the performance side-by-side

Add a counter for SMB Server Shares, Data bytes/sec, _total instance for JOSE-A1, JOSE-A2 and JOSE-A3.

clip_image003

Query the cluster shared volume ownership on Cluster A, with 2 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name State
--------- ---- -----
JOSE-A1 Cluster Disk 6 Online
JOSE-A1 Cluster Disk 3 Online
JOSE-A1 Cluster Disk 4 Online
JOSE-A2 Cluster Disk 7 Online
JOSE-A2 Cluster Disk 2 Online
JOSE-A2 Cluster Disk 5 Online

 

Run SQLIO to issue 8KB IOs

C:\sqlio\sqlio.exe -s9999 -kR -t1 -o16 -b8 -BN -LS -frandom -dPQRSTU testfile.dat

clip_image004

  

Add a 3rd node and wait for it to take 1/3 of the load

Add-ClusterNode -Cluster JOSE-A -Name JOSE-A3

Wait 2 and a half minutes to transition to the following state.

clip_image005

Re- query the cluster shared volume ownership on Cluster A, now with 3 nodes

Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize

OwnerNode Name State
--------- ---- -----
JOSE-A1 Cluster Disk 3 Online
JOSE-A1 Cluster Disk 6 Online
JOSE-A2 Cluster Disk 5 Online
JOSE-A2 Cluster Disk 7 Online
JOSE-A3 Cluster Disk 2 Online
JOSE-A3 Cluster Disk 4 Online

 

12) Installing VMM on JOSE-V

# Install .NET Framework
Install-WindowsFeature NET-Framework-Core

# Install Windows Server 2012 R2 ADK
C:\ADK\adksetup.exe
# Select only the “Deployment Tools” and “Windows PE” options

# Install SQL Server 2012 SP1
# New SQL Server standalone installation, Feature Selection: Database Engine Services

# Install VMM 2012 R2
# Features selected to be added: VMM management server, VMM console
# Database: VirtualManagerDB database will be created on JOSEBDA-V
# Service Account: Local System account

13) Configuring VMM on JOSE-V

Bring the File Server Cluster under VMM management

Select Fabric and use the option to add Storage Devices

clip_image006

Add a Windows-based file server

clip_image007

Specify the full path to the file server cluster:

clip_image008

Verify the File Server Cluster was properly discovered by VMM

Check the provider

clip_image009

Check the Storage Spaces discovery

clip_image010

Check the Scale-Out File Server and file share discovery

clip_image011

 

Remove the File Server Cluster node (demo starts with 2 nodes)

Under the properties of the File Server Cluster, remove node 3

clip_image012

Check progress under running Jobs

clip_image013

 

While running a workload, add a File Server Cluster node

Under the properties of the File Server Cluster, add node 3 specifying the full path of to the server

clip_image014

Check progress under running Jobs

clip_image015

 

14) Verifying systems’ configuration

 PS C:\> Get-PhysicalDisk -CimSession JOSE-A1, JOSE-A2, JOSE-A3 | Sort PSComputerName, Size

 

PS C:\> Get-Disk -CimSession JOSE-A1, JOSE-A2, JOSE-A3 | Sort PSComputerName, Size

Number Friendly Name OperationalStatus Total Size Partition Style PSComputerName
------ ------------- ----------------- ---------- --------------- --------------
4 Microsoft Storage Space Device Online 1 GB MBR JOSE-A1
10 Microsoft Storage Space Device Online 3 GB MBR JOSE-A1
8 Microsoft Storage Space Device Online 3 GB MBR JOSE-A1
0 Virtual HD ATA Device Online 40 GB MBR JOSE-A1
4 Microsoft Storage Space Device Online 3 GB MBR JOSE-A2
5 Microsoft Storage Space Device Online 3 GB MBR JOSE-A2
0 Virtual HD ATA Device Online 40 GB MBR JOSE-A2
4 Microsoft Storage Space Device Online 3 GB MBR JOSE-A3
5 Microsoft Storage Space Device Online 3 GB MBR JOSE-A3
0 Virtual HD ATA Device Online 40 GB MBR JOSE-A3

PS C:\> Get-ClusterNode -Cluster JOSE-A | FT Cluster, NodeName, State, Id -AutoSize

Cluster NodeName State Id
------- -------- ----- --
JOSE-A JOSE-A1 Up 2
JOSE-A JOSE-A2 Up 1
JOSE-A JOSE-A3 Up 3

PS C:\> Get-ClusterResource -Cluster JOSE-A | FT –AutoSize

Name State OwnerGroup ResourceType
---- ----- ---------- ------------
Cluster IP Address Online Cluster Group IP Address
Cluster Name Online Cluster Group Network Name
Cluster Pool 1 Online Cluster Group Storage Pool
JOSE-F Online JOSE-F Distributed Network Name
Scale-Out File Server (\\JOSE-F) Online JOSE-F Scale Out File Server
WitnessDisk Online Cluster Group Physical Disk

PS C:\> Get-ClusterSharedVolume -Cluster JOSE-A | FT -AutoSize

Name State Node
---- ----- ----
Cluster Disk 2 Online JOSE-A3
Cluster Disk 3 Online JOSE-A2
Cluster Disk 4 Online JOSE-A3
Cluster Disk 5 Online JOSE-A1
Cluster Disk 6 Online JOSE-A2
Cluster Disk 7 Online JOSE-A1

PS C:\> Get-SmbShare Share* -CimSession JOSE-F | FT -AutoSize

Name ScopeName Path Description PSComputerName
---- --------- ---- ----------- --------------
Share1 JOSE-F C:\ClusterStorage\Volume1\Share JOSE-F
Share2 JOSE-F C:\ClusterStorage\Volume2\SHARE JOSE-F
Share3 JOSE-F C:\ClusterStorage\Volume3\Share JOSE-F
Share4 JOSE-F C:\ClusterStorage\Volume4\Share JOSE-F
Share5 JOSE-F C:\ClusterStorage\Volume5\SHARE JOSE-F
Share6 JOSE-F C:\ClusterStorage\Volume6\Share JOSE-F

  

15) Final Notes

 

  • Keep in mind that there are dependencies between the services running on each VM.
  • To shut them down, start with VM5 and end with VM1, waiting for each one to go down completely before moving to the next one.
  • To bring the VMs up again, go from VM1 to VM5, waiting for the previous one to be fully up (with low to no CPU usage) before starting the next one.
  • I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
  • Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.

 

To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.