Infrastructure as Code with existing Azure Data Factory

  • Article

I create an ARM template from already existing Azure Data Factory. I'd like to share the tips for that.

The following link is the useful resource to develop ARM template of Azure Data Factory from scratch.

1. Getting Azure Data Factory ARM Template

The easiest way is to use Azure Portal. Try to create a new Azure Data Factory from the portal. Unfortunately, you can not get "Automation Script" of Azure Data Factory from an existing Resource Group.

a. Click Automation options
b. Click Download button

Now you have Data Factory ARM template with some scripts.

2. Adding DataFactory configurations

If you have an Azure Data Factory instance, you might create several linked services, datasets, and Pipelines. You need to add these configurations to the ARM template which we downloaded at 1. Getting Azure Data Factory ARM Template.

You can get the configuration at the Data factory dashboard. Actions > Author and deploy. Then you can see the ARM template for the existing resource.

2.1. Copy and paste the configuration

Just copy paste the configurations to the ARM template which we downloaded at 1. However, it doesn't work as it is.

The ARM template structure is like this.

 {
 "$schema": "https://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
 "contentVersion": "1.0.0.0",
 "parameters": {
 "name": {
 "type": "string"
 },
 "location": {
 "type": "string"
 },
 "apiVersion": {
 "type": "string",
 "defaultValue": "2015-01-01-preview"
 }
 },
 "resources": [
 {
 "apiVersion": "[parameters('apiVersion')]",
 "name": "[parameters('name')]",
 "location": "[parameters('location')]",
 "type": "Microsoft.DataFactory/dataFactories",
 "properties": {},
 "resources": [
 COPY AND PASTE THE CONFIGURATIONS HERE! 
 ]
 }
 ]
}

No such thing like "Resources" after the "properties." You need to add that. Then copy and paste the configuration from the existing Azure Data Factory.


2.2. Edit

You need to edit several things from the configurations.

2.2.1. remove "hubname"

The existing configuration includes "hubname". Remove it. This is an example of mine.

 {
 "name": "AzureSqlLinkedService",
 "properties": {
 "description": "Azure Sql Database for testing Azure Data Factory telemetry",
 "hubName": "spikeadf_hub", <----- REMOVE THIS
 "type": "AzureSqlDatabase",
 "typeProperties": {
 "connectionString": "xxxxxxxxxxxxxx"
 }
 }
}

2.2.2. check the configuration of the connectionString

Your connection string will be masked in some part. Please convert it to the original value.

 "connectionString": "Data Source=tcp:aaaaaa.database.windows.net,1433;Initial Catalog=aaaaa;Integrated Security=False;User ID=someuser@aaaaa.database.windows.net;Password=**********;Connect Timeout=30;Encrypt=True"

2.3. Debugging

Sometimes, you might get an error. You can see that is very hard to understand.

Don't worry, just go to the Azure portal and you can see the new Azure Data Factory is created.
If you check the "Actions > Author and deploy," you can see the detail cause of the Error.

2.4. Deployment

The ARM template which you downloaded at 1. It includes "./deploy.ps1" Just execute on your PowerShell.

Conclusion

Now you successfully create an ARM template for the existing Azure Data Factory instance. Enjoy.

Resource