Let’s imagine that we create an Azure Data Factory (ADF) with a pipeline containing a Copy Activity that populates SQL Azure with data from an on premise SQL Server database. If we set the schedule with a short interval, say to run every 15 minutes over a 3 month period, then we will discover that it will generate a large number of executions or slices (around 9,000).
So many slices can be difficult to navigate and manage through the Azure Portal. Currently, the ADF portal has a very basic scheduler and it is not possible to set a schedule executing regularly - but only running Monday and Friday 7am to 7pm - as an example.
To do this you first have to create an AF in the Azure portal and set the trigger with a CRON schedule. You can then enter the code to perform this action by pasting into the portal. What’s the catch you say? Well there is a catch, you will need to register the AF as an app in Azure Active Directory (AD) so that it can authenticate itself before updating the ADF. An excellent article to do this can be found here.
The code below is sufficient to update the ADF start and end dates. Function.json contains the JSON for the AF trigger. In this case we are using a timerTrigger, with the schedule using CRON syntax. Project.json provides references to NuGet packages that will be downloaded on demand. Run.csx is the actual C# code that is compiled on the fly using Roslyn and executed. Note the slightly different syntax for the using statements.
If you have any queries about Azure, or would like to discuss this in further detail please don't hesitate to contact us.
- By Nigel Wardle (Application Architect)
* All links and URLs provided are at the discretion of the author
