Introducing the Windows Server Storage Migration Service

Hi Folks, Ned here again with big news: Windows Server 2019 Preview contains an entirely new feature!

The Storage Migration Service helps you migrate servers and their data without reconfiguring applications or users.

  • Migrates unstructured data from anywhere into Azure & modern Windows Servers
  • It’s fast, consistent, and scalable
  • It takes care of complexity
  • It provides an easily-learned graphical workflow

My team has been working on this critter for some time and today you’ll learn about what it can do now, what it will do at RTM, and what the future holds.

Did I mention that it comes in both Standard and Datacenter editions and has a road map that includes SAN, NAS, and Linux source migrations?

Come with me…

Why did we make this?

You asked us to! No really, I dug through endless data, advisory support cases, surveys, and first party team conversations to find that our #1 issue keeping customers on older servers was simply that migration is hard, and we don’t provide good tools. Think about what you need to get right if you want to replace an old file server with a new one, and not cause data loss, service interruption, or outright disaster:

  • All data must transfer
  • All shares and their configuration must transfer
  • All share and file system security must transfer
  • All in-use files must transfer
  • All files you, the operator, don’t have access to must transfer
  • All files that changed since the last time you transferred must transfer
  • All use of local groups and users must transfer
  • All data attributes, alternate data streams, encryption, compression, etc. must transfer
  • All network addresses must transfer
  • All forms of computer naming, alternate naming, and other network resolution must transfer
  • Whew!

If you were to wander into this Spiceworks data on market share (a little old but still reasonably valid), you’ll see some lopsided ratios:

(source)

A year and a half later, there are a few million Windows Server 2016 nodes in market that have squeezed this balloon, but there’s even more Windows Server 2012 and still plenty of 2008 families, plus too much wretched, unsupported Windows Server 2003. Did you know that Windows Server 2008 Support ends in January of 2020? Just 20 months away from the end of life for WS2008 and we still have all this 2003!

The Storage Migration Service (Updated: July 17, 2018)

Important: This section of the blog post is going to change periodically, as with the Windows Insider preview system and Windows Admin Center’s preview extension system, I can give you new builds, features and bug fixes very rapidly. You’ll want to check back here often.

Windows Server 2019 and the Storage Migration Service are not supported in production environments!

In this first version, the feature copies over SMB (any version). Targets like Azure File Sync servers, IaaS VMs running in Azure or MAS, or traditional on-prem hardware and VMs are all valid targets.

The feature consists of an orchestrator service and one or more proxy services deployed. Proxies add functionality and performance to the migration process, while the orchestrator manages the migration and stores all results in a database.

Storage Migration Service operates in three distinct phases:

  1. Inventory – an administrator selects nodes to migrate and the Storage Migration Service orchestrator node interrogates their storage, networking, security, SMB share settings, and data to migrate
  2. Transfer – the administrator creates pairings of source and destinations from that inventory list, then decides what data to transfer and performs one more or transfers
  3. Cutover – (not yet available) – the administrator assigns the source networks to the destinations and the new servers take over the identity of the old servers. The old servers enter a maintenance state where they are unavailable to users and applications for later decommissioning, while the new servers use the subsumed identities to carry on all duties.

What’s new in 17713

  •  Cutover added
  •  Double the transfer performance
  •  Many UI improvements in Windows Admin Center
  •  Many bugs fixed

Walk through

Requirements

You’ll need the following to start evaluating this feature:

  • At least two Windows Server 2019 Preview computers or VMs with the Storage Migration Service features installed. One will operate as the orchestrator and one as the destination of the migration. Note: it is supported to make the orchestrator and destination the same computer, such as in a small environment with a single server to migrate. A larger environment will usually have a single Orchestrator and many destinations, however, so that’s how the steps are documented below.
  • The Windows Admin Center installed on some computer, such your laptop or desktop
  • The Storage Migration Service preview extension for Windows Admin Center installed

Supported source operating systems VM or hardware (to migrate from):

  • Windows Server 2003
  • Windows Server 2008
  • Windows Server 2008 R2
  • Windows Server 2012
  • Windows Server 2012 R2
  • Windows Server 2016
  • Windows Server 2019 Preview

Supported destination operating system VM or hardware (to migrate to):

  • Windows Server 2019 Preview*

* Technically your destination for migration can be Windows Server 2012 or later, but we aren’t testing that currently. And when we release the faster and more efficient target proxy system in a few weeks it will only run on Windows Server 2019.

Security:

  • All computers above domain-joined
  • You must provide a migration account that is an administrator on selected source computers
  • You must provide a migration account that is an administrator on selected destination computers
  • The following firewall rules must be enabled INBOUND on source and destination computers:
    • “File and Printer Sharing (SMB-In)”
    • “Netlogon Service (NP-In)”
    • “Windows Management Instrumentation (DCOM-In)”
    • “Windows Management Instrumentation (WMI-In)”

Install

Use the Windows Admin Center, Server Manager, or PowerShell to install the Storage Migration Service.

The features to install on the orchestrator node are:

  • “Storage Migration Service”
  • “Storage Migration Service Proxy” (installs automatically when orchestrator selected)
  • “Storage Migration Service tools” (installs automatically when orchestrator selected, under Remote Server Administration Tools, Feature Administration Tools)

The feature to install on the Destination nodes where you intend to migrate to:

  • “Storage Migration Service Proxy”

Run your first test inventory and transfer

Now you’re ready to start migrating.

  1. Logon to your Windows Admin Center instance and connect to the orchestrator node as an administrator.

2.  Ensure the Storage Migration Service extension is in the Tools menu (if not, install the extension) and click it.

3. Observe the landing page. There is a summary that lists all active and completed jobs. Jobs contain one or more source computers to inventory, transfer, and cutover as part of a migration.

Example, with several previous jobs

4. You are about to begin the Inventory Click New Job. Enter a job name, click Next.

5. Enter Source Credentials that are an administrator on your source (to be migrated from) computers and click Next.

6. Click Add Device and add one or more source computers. These must be Windows Server and should contain SMB Shares with test data on them that you want to migrate.

7. Click Start Scan and wait for the inventory to complete.

8. Observe the results. You can open the Details at the bottom of the page with the caret in the lower right. When done reviewing, click Finish Inventory.

9. You are now in the Transfer You are always free to return to the Inventory phase and redefine what it gathers, get rid of the job, create a new job, or proceed forward to data transfer. As you can see, each phase operates in a similar fashion by providing credentials, setting rules, defining nodes, then running in a result screen.

10. Provide credentials, destination computers mapped to the source computers, ensure each server you wish to migrate is set to Included in transfer, review your settings, then proceed with Start Transfer.

11. Observe the migration. You will see data transfers occur in relative real time (periodically refreshed) as the orchestrator copies data between source and destination nodes. When complete, examine the destination server and you’ll find that Storage Migration Service recreated all shares, folders, and files with matching security, attributes, characteristics (see Known Issues below for not-yet-released functionality here). It’s copy rate is currently like single-threaded robocopy performance, per node. Note the Export option that allows you to save a complete database dump of the transfer operations for auditing purposes.

12. Move to the cutover phase.

13. Add the Cutover destination credentials (same as Transfer destination credentials).

14. Pair the network interfaces between source and destination NICs so that IP addresses can be moved. You have the option to move the old source computer to DHCP or use a new static IP address. You also have the option to randomly rename the old source computer or specify a new name, as the destination will take over the old name as part of cutover.

15. Begin the cutover and allow the destination computer to take over the network and names of the old source. Both source and destination will reboot several times apiece. The progress bar will show how far until the operation completes.

Its time to complete is dependent on:

a. Server reboot times

b. AD Replication time (for domain joins and computer accounts being known to all users)

c. DNS Replication time

Known issues

Important! During the preview phase we need you to always run latest binaries before reporting bugs. To get the latest:

  1. Windows Server 2019 Preview – download from https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewserver
  2. Storage Migration Service extension for Windows Admin Center – download using the extension manager https://docs.microsoft.com/en-us/windows-server/manage/windows-admin-center/configure/using-extensions . If you already have a previous extension installed, uninstall it, then install the latest extension.

Applies to:

  • Windows Server 2019 Insider Preview Build 17713
  • Storage Migration Service Extension Version 0.0.17709

The following issues are known:

  • Extension:
    • Credentials must be added in the Cutover phase even though unchanged from Transfer stage
  • Orchestrator:
    • Change detection when running multiple transfer passes not complete, optional destination data preservation not added yet
    • No EFS, compressed files, reparse point support yet
    • Database size limited by free space on system drive, stored under c:\programdata\storagemigrationservice
    • Not yet performance optimized for transfer.
  •  Destination:
    •  An admin has to manually enable firewall rules (a coming update will remove this requirement)

Roadmap

What does the future hold? We have a large debt of work already accumulated for when we complete the SMB and Windows Server transfer options. Things on the roadmap – not promised, just on the roadmap 😊:

  1. Network range and AD source computer scanning for inventory
  2. Samba, NAS, SAN source support
  3. NFS for Windows and Linux
  4. Block copy transfers instead of file-level
  5. NDMP support
  6. Hot and cold data detection on source to allow draining untouched old files
  7. Azure File Sync and Azure Migrate integration
  8. Server consolidation
  9. More reporting options

Feedback and Contact

Please make sure the issue isn’t already noted above before filing feedback and bugs!

  • Use the Feedback Hub tool include in Windows 10 to file bugs or feedback. When filing, choose Category “Server,” subcategory “” It helps routing if you put Storage Migration Service in the title.
  • You can also provide feature requests through our UserVoice page at https://windowsserver.uservoice.com/forums/295056-storage. Share with colleague and industry peers in your network to upvote items that are important to you.
  • If you just want to chat with me privately about feedback or a request, send email to smsfeed@microsoft.com. Keep in mind that I may still make you go to feedback hub or UserVoice.

 

Now get to testing and keep an ear out for updates. We plan to send out new builds of Windows Server 2019 and the Storage Migration service very often until we get close to RTM. I will post announcements here and on twitter.

Here’s some background music to help

Ned “reel 2 real” Pyle