Azure Batch Preview

I am excited to announce the preview of Azure Batch, a new Azure platform service that provides job scheduling and large-scale resource management for developing and delivering Big Compute applications native in the cloud. 

What is Batch?

Azure Batch originated from seeing our customers repeatedly building their own job queues, schedulers, and VM scaling for applications that used a parallel processing pattern. Developing and supporting this job scheduling infrastructure was time-consuming and took resources away from work on their core service.                                   

With Azure Batch, you submit jobs. We start the VMs, run your tasks, handle any failures, and then shut things down as work completes. This batch computing or parallel processing pattern is used across industries and applications. It’s not just for high performance computing (HPC), but performance is at our core.  

Common uses for batch computing in the cloud include media transcoding and rendering, image analysis and processing, financial risk calculation and Monte Carlo simulations, engineering simulations and stress analysis, weather forecasting, energy exploration and software test execution.  

The batch pattern is widely used and well suited for the cloud, where resources can be provisioned on demand at very high scale and used efficiently to process a lot of work. The programming model is simple—call an executable or script with a set of parameters. There can be a single task per job or hundreds of thousands. The batch pattern makes it easy to use the cloud and take advantage of scale and reliability without needing to learn about managing multiple instances, fault domains, error handling, and other concepts that our service handles for you.  

Batch on Azure Today

Azure Batch is used in production today by several Microsoft teams. One example is Azure Media Services which uses Batch for its audio and video encoding. Media Services decides which jobs to run, and submits the encode tasks to Batch. Auto-scaling rules help Media Services uses their VMs efficiently, limit idle capacity, and deliver different tiers of service.                   

Azure Batch is also used by the Azure engineering teams to test Azure itself. A common test execution framework is used to manage and schedule tests, which are executed by Batch. Both services operate at high scale and require high reliability.  

Using Azure Batch

We invite you to join our public preview by registering for the preview, learning about the service, and sharing your feedback on the experience and capabilities that you would like to see.                           

This preview is targeted at developers to introduce the service and support early adopters using Batch. The API supports two styles of integration. The first is for developers producing a service that requires batch processing and need fine-grained control over running tasks and the VMs the tasks run on. The second is a higher-level abstraction for integrating batch applications with Azure. This includes parallel or cluster applications, as well as any work that you want to execute in a batch pattern in the cloud. The Azure Batch API supports REST, a C# object model, and Python.  

We call the experience for integrating applications Batch Apps. These capabilities come from GreenButton, a company that Microsoft acquired in May. The API makes it easy to build plugins for desktop applications to submit jobs to a “cluster” in the cloud. Azure Batch then splits the job into one or more tasks to process the application. The job definition can describe data that should be moved to the cloud for the job, how the data is stored and copied to compute VMs, and then results merged and returned

Roadmap

Over the next months we will be adding additional features including application lifecycle management, rich monitoring, role-based access for submitting jobs and service management, custom VM images, and Linux. We will also be moving the Batch Apps management UI to the new Azure portal.                                                            

We look forward to working with you on Azure Batch, and welcome your feedback and suggestions.  

Getting Started

Sign up for the preview

Learn about Batch

Create an account in the portal once approved for the preview

Alex Sutton

Group Program Manager, Azure Big Compute