HPC Schedulers

The scheduler is the core of a high-performance computing cluster:

-          It allocates computing resources to the jobs that are submitted to the cluster.

-          It maintains the status of those jobs to ensure orderly completion or termination.

We provide a scheduler with the Compute Cluster Pack. Our scheduler understands four types of jobs:

-          Sequential, typically for set-up operations: The job contains one task, one sequential executable, e.g. to copy some input data.

-          Parallel with message-passing, common in engineering and scientific applications: The job contains one task, i.e. the MPI start-up program mpiexec. This in turn will start as many instances of a named MPI application as there are cores allocated to it. For instance:

o    Job submit /numprocessors:16 mpiexec –wdir \\myshare myapp.exe

will start 16 instances of myapp.exe, a parallel application, on 16 cores across a number of nodes allocated by the scheduler. \\myshare will be the working directory for all those myapp.exe. Most finite element applications (e.g. for car crash, fluid dynamics simulations) have a parallel computing core based that uses message-passing.  

-          Parametric sweeps, common in financial applications: The job contains multiple independent tasks. Typically, they are exactly the same executable but with different parameters, hence the name. Montecarlo simulations used for financial trading are mostly parametric sweeps.

-          Task flows, again common in engineering: A sequence of tasks of any kind that automates a particular workflow. Note that we handle just simple dependencies and no exceptions. For instance, a setup task must execute before a parallel computation and a visualization task must execute afterwards.

Our scheduler comes with a COM API that makes it simple to use it programmatically. For instance, you can write a macro in Excel that calls that COM API and passes a section of the spreadsheet to an executable running on CCS for computing. This makes the computing power of a large number of machines available on the desktop with relatively little effort.

There are a number of commercial and open-source HPC schedulers that offer more advanced features (e.g. they support cycle stealing too), but may also be more difficult to set up and use, or more expensive than ours:

-          Platform Computing’s LSF, which integrates with ours

-          Altair’s PBS Pro

-          Digipede

-          Sun’s Grid Engine.

-          Download a trial version of our hpc stack at www.microsoft.com/hpc

-          Familiarize yourselves with hpc schedulers here and here.

Try out some sample code on our community site: www.windowshpc.net/files