Shelf-ware - the SCVMM Library

One of the things we heard fairly early on while planning VMM was that offline storage of VHDs was going to become an increasingly large management task. Based on this need, we created the "VM Library" feature to allow users to save virtual machines that weren't currently running to a managed location. As we developed the requirements for this feature, we quickly realized that people wanted to store more than VMs, they wanted to store all of the stuff that they used to create VMs as well (ISOs, templates, OS profiles, Hardware profiles etc.) so we changed the feature's name to "library" instead of "VM library". Before defining the feature in detail, we had to make some key design decisions and at the risk of boring you, I'll walk through some of that decision making process.

Does the library contents reside in the file system or in a database?

This is an age old debate and ultimately we decided to keep meta data in our SCVMM database but we kept all of the VHDs, ISOs etc. on file shares. While this made our life a little bit harder (we were now responsible for designing a system that kept meta data and the file system data in sync) it gave customers more flexibility. They could continue to use existing file replication technology and still had the ability to use Windows explorer or other file management tools to organize the information in any way that they wanted. It also made it easier to integrate with existing systems.

Is the library a physical thing or just a logical thing?

We decided that the answer to this question was "both". If you go to the library section of our UI, you'll see that you can either view the library as a logical collection of items (my favorite view is to click on the library server tree-node on the left and then group by "type") or you can view the actual physical folder hierarchy within the library share. Logical views are very convenient but sometimes the physical location of a file is important as well so you can use either view interchangeably. Templates, OS profiles and hardware profiles are all purely logical constructs so you won't find those items on disk, they're in our database. It's likely that we'll ultimately have on disk representations of those objects as well in the future though.

SCVMM Library Logical View

SCVMM Library Physical View

Does the library contain every object the SCVMM knows about or just stuff physically in the file shares that the library is comprised of?

This one we went back and forth on but ultimately decided that the library isn't really an "inventory" of everything in the world that we manage, it's where you keep stuff that is not running or associated with a host. You can think of it as a virtual 'shelf' to store stuff that you're not currently using. With this decision, there are only two places an object can live and be managed by SCVMM - either registered to a host or stored in the library. If you put something in a random folder or unmanaged file share, you're on your own.

What about co-location of library objects with the host for rapid deployment?

This decision was pretty easy. We decided to allow a host server to also be a library server. In this manner, you can store templates or other library objects local to a physical host so that deployment from the library to that host would be very fast.

Do we build Powershell cmdlets to organize library contents?

We initially started working on "move" and "copy" and "create" cmdlets to help users organize the library directly from our UI but that turned into a very slippery slope. One of the benefits of providing direct access to our library files via the file system is that you can organize directly through the Windows explorer interface (create directories, copy, move etc.). Aside from being a lot of extra work for us to expose all of that through our UI as well, it was unlikely that we'd get parity with the full set of file system operations. As a result, we made the call to cut this functionality from our UI but we added some requirements to ensure that as stuff was moved around in the file system, SCVMM would track the changes and have them reflected in our UI. By default we "refresh" the library share every hour but you can change this value or just refresh manually in the library UI or via cmdlet. We also tag each of objects in the library with GUID so that if you move things between shares, we just update the path property since we recognize the object even as the location changes.

Once we made these decisions, the rest of the functionality worked itself out pretty seamlessly but some of our beta feedback showed that we had overlooked one key scenario.

Throughout the design process, we had assumed that customer would typically have a handful of library servers (probably less than five) but lots of hosts (probably tens or hundreds). This is true for most data centers as admins like to consolidate and it wasn't uncommon to see large library servers attached to SAN arrays with fast storage for rapid deployment. With retail customers however, it was a different story. These guys had thousands of stores worldwide and needed to have a library server, often the same physical machine as the host computer, in each location so they effectively had at at least as many library servers as hosts. Some customers had library servers attached to regions of world and again they ended up with many more libraries than we had designed for. Typically, each of the library servers had the same file content replicated to across many sites (in the same way that they published software installation files or content).

Taking this feedback into consideration, we did a couple of things:

1) Began testing the system with lots and lots of library servers :-)

2) Added the "Library Group" concept.

If you right click on a library server in the library section of our UI, you'll see a property called "library group" which lets you associate a library server to a host group. Now, when you go to create a new virtual machine and you want to chose a base VHD from the library to start with, you can have the SCVMM UI scope your choices to library servers associated with the host group that you want to put the VM into.

As an example, you might create 25 geographic regions and assign a host group to each region. If you are creating a VM for the "Pacific northwest" region/host group, you'll want to use library objects from the library server associated with that region to prevent large file copy operations across very long distances. To enable this use case, you right click on the library file share and set the "library group" equal to "Pacific northwest". Now, when you launch the new VM wizard, you'll notice that when you select a library object to build a VM, you can scope the objects down by "library group" to ensure you use the closest copy of the file you need. In my example, you'd select "Pacific northwest" for this value. Similarly, from the command line, you can query for a library object and filter by library host group to ensure that you use the closest copy of the file.

The rest of the library concepts are pretty self-explanatory and there is plenty of additional content on the SCVMM library here as well.

Rakesh

P.S. - Happy Holidays!