How to Stop/Start ADF Triggers based on annotations using GitHub actions.

Triggers are another way that you can execute a pipeline run. Triggers represent a unit of processing that determines when a pipeline execution needs to be kicked off. Currently, the service supports three types of triggers:

• Schedule trigger: A trigger that invokes a pipeline on a wall-clock schedule.
• Tumbling window trigger: A trigger that operates on a periodic interval, while also retaining state.
• Event-based trigger: A trigger that responds to an event.

What is Annotations in ADF

Azure Data Factory annotations are tags that you can add to your Azure Data Factory or Azure Synapse Analytics entities to easily identify them. An annotation allows you to classify or group different entities in order to easily monitor or filter them after an execution. Annotations only allow you to define static values and can be added to pipelines, datasets, linked services and triggers.

Problem Statement

Recently we started promoting Azure Data factory changes from lower environment to Prod environment using Github actions. One of the biggest problem statement we faced with Stopping/Starting ADF triggers.

we don’t want them to Start ADF triggers accidently (as they might have disabled on purpose) which are already in stopped state. So how do I deploy my code without starting them. I want to stop and start only the one’s which are currently running.

Resolution.

First and foremost in order to deal with ADF triggers we have added annotations to individual triggers. Now lets start looking into the steps to implement.

1. Login to Azure Portal → Go to ADF instance → Click on Manage → Click on triggers.
2. Lets open any existing trigger → it will open it in edit mode → Add new Annotation → click OK.

3. Once annotations are saved lets create a new build and package them in drop folder.

4. Now lets login to Github.com → and save our code to our GitHub repo → create a new GitHub action.

5. Add a new workflow in Azure PowerShell Inline script and lets start adding all variables.

Pass all required variables such as

• Resource group
• Data factory Name
• Annotations can also be variablized.

6. Now lets look into individual parameters which I have used in my YAML file, below command is going to pull all existing triggers from Azure Data Factory.

$Triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName

7. Next I want to fetch trigger which are in stopped state.

$stoppedTriggers = $Triggers | Where-Object { $_.Properties.RuntimeState -eq "Stopped" }

8. Now i want to save triggers which are already in running state.

 $startedTriggers = @()

9. Now I want to start my actual ADF deployment and stop triggers before I proceed further for that I am using foreach loop. Here I am comparing triggers based on Annotations and the one’s which are in running state.

Screenshot of current triggers few of them are running and few of them are in Stopped state.

foreach ($trigger in $Triggers) {
             if(($Trigger.Properties.Annotations[0] -eq $Annotations) -and ($Trigger.Properties.RuntimeState -eq "Started"))
                {
            Stop-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $trigger.Name -Force
            $startedTriggers += $trigger.Name
            Write-Host "Stopped trigger $($trigger.Name)."
            } elseif ($trigger.Triggerstate -eq "Stopped") {
                Write-Host "Trigger $($trigger.Name) is already in a stopped state."
            } else {
                Write-Host "Trigger $($trigger.Name) is in an unknown state: $($trigger.Triggerstate)."
            }
            }

This is how output looks, here you can see it has stopped the triggers which were running earlier and which has Annotation with “LDS”.

Here you can see it has trigger #2 which was running earlier and it has an Annotation of “LDS” and it has skipped #4 as we have given boolean condition to check Started status and “LDS” status.

Github actions output.

Stopped trigger BOM Fire and Weather_TumblingWindow_30Minutes.
True

Final stage after deployment of Templates now we need to check status of triggers again.

True
Started trigger BOM Fire and Weather_TumblingWindow_30Minutes.

Here is the full YAML template

- name: 'ADF triggers stop --> Deployment and triggers start step'
       uses: azure/powershell@v1
       with:
          inlineScript: |
            $ResourceGroupName = "${{ vars.DEFAULT_PROPERTIES_RESOURCEGROUP_VALUE1 }}" 
            $DataFactoryName = "${{ vars.FACTORYNAME1 }}"
            $Annotations = "LDS" 
            $Triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName
            $stoppedTriggers = $Triggers | Where-Object { $_.Properties.RuntimeState -eq "Stopped" }
            $startedTriggers = @()
            
            foreach ($trigger in $Triggers) {
             if(($Trigger.Properties.Annotations[0] -eq $Annotations) -and ($Trigger.Properties.RuntimeState -eq "Started"))
                {
            # Stop the Triggers that were in "Started" state
            Stop-AzDataFactoryV2Trigger -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $trigger.Name -Force
            $startedTriggers += $trigger.Name
            Write-Host "Stopped trigger $($trigger.Name)."
            } elseif ($trigger.Triggerstate -eq "Stopped") {
                Write-Host "Trigger $($trigger.Name) is already in a stopped state."
            } else {
                Write-Host "Trigger $($trigger.Name) is in an unknown state: $($trigger.Triggerstate)."
            }
            }

            $password = ConvertTo-SecureString "${{ secrets.SPN_CERT_PASSWORD }}" -asplaintext -force
            Connect-AzAccount -ApplicationId "${{ secrets.SPN_CLIENT_ID }}" -Tenant "${{ secrets.SPN_TENANT_ID }}"  -CertificatePath '/home/runner/work/_temp/spn.pfx' -CertificatePassword $password 
            Set-AzContext -Subscription "${{ secrets.SPN_SUBSCRIPTION_ID }}"
            New-AzResourceGroupDeployment -ResourceGroupName ${{ vars.DEFAULT_PROPERTIES_RESOURCEGROUP_VALUE1 }} -TemplateFile "/home/runner/work/GithubRepostore/GithubRepostore/drop/ARMTemplateForFactory.json" -TemplateParameterFile "/home/runner/work/GithubRepostore/GithubRepostore/drop/ARMTemplateParametersForFactory.json"

            # Start the Triggers that were in "Started" state before stopping
            foreach ($triggerName in $startedTriggers) {
                Start-AzDataFactoryV2Trigger -ResourceGroupName ${{ vars.DEFAULT_PROPERTIES_RESOURCEGROUP_VALUE1 }} -DataFactoryName ${{ vars.FACTORYNAME1 }} -Name $triggerName -Force
                Write-Host "Started trigger $triggerName."
            }

            Write-Host "Trigger stop, Deployment and start step has been completed."
          azPSVersion: 'LatestVersion'
       if: success()