Creating an Azure Data Factory Instance
Chris Testa-O’Neill is a Features Engineer in the Analytics and Data Science team at Microsoft. This is a preview for his talk at the upcoming SQLBits, which takes place between the 5th and 8th April. Be sure to check it out!
There are a variety of methods that can be used to create an instance of an Azure Data Factory (ADF). This blog explores how you can create an ADF instance using the Azure Portal, PowerShell and Azure Resource Manager templates.
Azure Portal
The first step is to create the Azure Data Factory. This can be performed within the Azure portal by clicking on the New icon, pointing to Databases and then clicking Data Factory.
At this point the provisioning blade for ADF appears. It is in this blade that you define a name for the Data Factory instance. You can then assign the instance to a subscription that you own.
The resource group enables you to define whether your ADF instance will reside in a resource group that already exists, or in a resource group that you create. Resource groups are important for a few reasons; they contain services that can have the billing be visible on that container, you can define access control on a resource group, and resources held within that container can communicate with each other without the need to write complex IaaS scripts to further define communication between services.
The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. You decide how you want to allocate resources to resource groups based on what makes the most sense for your organisation. Therefore, it is important to understand the objective for creating a resource group so it can be planned appropriately.
Finally, you will then assign the Data Factory instance to a region of your choice.
You can view a video here on how to setup an ADF instance in the Azure Portal.
PowerShell
You also have the ability to deploy Azure Data Factory instances using PowerShell. This requires that Azure PowerShell is installedon your computer. Once this is set up you can use the following PowerShell commands to create a Data Factory instance. The following commands set the Azure context to a subscription and then defines a Data Factory instance defining the resource group where the instance is hosted, followed by the name and the location.
You can view 3 minute this video on how to create an ADF instance here.
Azure Resource Manager templates
When dealing with ADF in production scenarios, or dealing with multiple ADF instances you can also make use of Azure Resource Manager (ARM) templates to ensure the deployment of multiple instances of ADF is much easier. ARM templates are JavaScript Object Notation (JSON) files that defines one or more resources to deploy to a resource group. It also defines the dependencies between the deployed resources. The JSON files can be parameterize to provide the flexibility to deploy ADF instances of different names, in different resource groups and regions.
There is an excellent article from the product group that will enable you to use ARM templates in a variety of Data Factory scenarios. I would highly recommend that you read this.
With the Data Factory instance created, it is then time to create the relevant linked services, data sets and pipelines to perform the data orchestration activities. These will be covered off individually in other blog posts.
---
Hopefully a lot of readers will have heard of the SQLBits conference that runs every year in the UK, but if you haven't then suffice to say it’s the largest SQL Server conference in Europe! There will be many people from Microsoft, both at the stand and hosting talks, ready to talk with you about SQL Server and help answer any questions you might have.
This year we will also be running an “Ask the Experts” style “SQL Clinic”, where you can get the opportunity to spend some time with a SQL Server SME. Be sure to look out for information in your booklets on how to arrange a session! I will be running a session on Azure Data Factory, where we will run through technical aspects of ADF and show you a real world solution.
Join us at the Telford International Centre between the 5th and 8th April. Not only can you see hundreds of sessions, but you can speak directly to members of the Microsoft Product group as well. Hopefully we can meet some of you there, come along to our stand and stop for a chat!