Azure Data Factory CICD
Updated: Apr 20, 2019
This diagram can help you understand the logistics behind CICD for Azure Data Factory. Below we will walk through some of the high level steps in creating a Release Pipeline that will update our upper Data Factory environments.
If you run into needing custom parameters for ADF CICD pipelines, check out ADF CICD Custom Parameters
Prerequisites:
Setup an Azure DevOps account
Setup Azure Git Integration, you need to do atleast 1 publish before the ADF_Publish branch gets created.
Setup:
1. Select Releases under the Pipelines tab and create a new pipeline.
2. When it ask you to select a template, choose start with an Empty Job
3. On the pipeline select: +Add an Artifact ; this will point to our Data Factory Git Repository
Choose the 2nd source type: Azure Repository..
Select the adf_publish branch, as this branch will automatically get created and updated when we do a publish from within the Data Factory UI.
4. Back in the pipeline, give your pipeline a name, I like to start with RPL for Release PipeLine and BRL for Build Pipeline.
5. Select +Add a Stage
6. Click the + on the Agent job and find the Azure Resource Group Deployment task. If a Agent Job does not exists, add one.
7. Understand the following steps:
Fill out Azure Subscription information. You may need to click advanced options and choose MSI authentication if the initial Authorization does not work. It will depend on how your organizational security is setup.
Template: Navigate to your ARMTemplateForFactory.json file.
Template Parameters: Navigate to your ARMTemplateParametersForFactory.json file
Override Template Paramaters: Here is where we can pass in environment vairables, the best practice for doing so would be to add secrets into a Azure Key Vault. We then can create variable groups(1 for each environment with DevOps libraries). We can pass in a variable to our values in the Override Template Parameters sections using $(myVarName)
Deployment Mode: *Choose Incremental, the other option will overwrite the entire resource group.
8. Finally, we can clone the QA stage and create our PROD stage, if you're using environment variables from Key Vault, change anything that isn't parameterized and not pointing to PROD.
