Cloudblitz: A Tool to Deploy HPC Clusters in Azure

  • Article

I'm happy to announce the availability of Cloudblitz, a tool (framework) to deploy HPC clusters in Azure programmatically via Powershell, then submit jobs to them via Powershell as well.

You can find it at https://cloudblitz.codeplex.com

The tool builds on the HPC scheduler for Azure and the Powershell commandlets for Azure to solve a common problem, i.e. provide a scriptable way to create and interact with an Azure PaaS cluster. I refer you to the Codeplex page for the complete description and documentation. In this blog post, I'll cover how to build it and use it.

1. Make sure that you download and install all the dependencies listed on the codeplex page.

2. Download and extract the source code in its own directory.

3. Open a Visual Studio 2012 command prompt as administrator (dear old cmd.exe)

4. Go to the "BuildScripts" directory and run BuildCloudBlitz. cmd. This will build the required modules and copy the results in a "Packages" directory.

5. Open a Windows Azure Powershell prompt as administrator. Go to the CloudBlitz.ClusterTemplate directory

6. Edit the ServiceDefinition.csdef file (it is just an XML file). In particular note:

<WorkerRole name="ComputeNode" vmsize="Small">

...

<WebRole name="HeadNode" vmsize="Small">

Choose your VM size and type it in. In general, a small head node is sufficient (the head node does NOT run SQL, rather it uses SQL Azure, which reduces its memory requirements). If you're planning to run thousands of tasks at a time, go for a large instance.

7. Go to the CloudBlitz.Powershell\Scripts directory

8. Edit the ConfigurationVariables.ps1 file. In particular note:

$subscriptionName = "<your subscription name>"
$subscriptionId = "<your subscription id>"
$administratorName = "<admin name for the hpc cluster>"
$systemPassword = "<admin password for hpc cluster>"
$certificateFileNameRoot="<name of certificate>"
$deploymentName="<name of service>"
$deployLocation="<data center location>"
$HeadNodeInstanceCount = 1
$ComputeNodeInstanceCount = <n. of desired compute nodes>
$removePreviousDeploymentAndService = $true

9. Run InstallAndExportCertificate. ps1 This will create a .cer and a .pfx self-signed certificate file, install it in the local user certificate store.

10. Via the Azure portal, upload the .cer file as a management certificate.

11. Run the DeployDeployerRole.ps1 script. It will upload the packages, create an azure worker role that deploys a hpc cluster as per configuration, then remove the deployer role. Such a temporary role is necessary to avoid any firewall and connectivity issues. Most people will have ports 80 and 443 open for http and https. The Azure HPC scheduler deployment also requires a connection to SQL Azure on port 1433, which is normally closed. By letting the temporary role manage the process, such connections are all internal to Azure.

You can now connect to your cluster.cloudapp.net via RDP and start submitting jobs in the "traditional" way via the job manager or the command line.

Alternatively, you can use the set of powershell commandlets we developed from your workstation. They will be the subject of the next blog post :-)