Using Azure Batch to Orchestrate and Execute Code at Large-ScaleWritten on March 2nd , 2019 by Steven McLintock
Azure Batch is not a new technology by Microsoft, but for some reason it doesn’t seem to be receiving as much attention as similar Microsoft cloud platforms: Azure Functions and Logic Apps.
Azure Batch has a few benefits over Azure Functions / Logic Apps:
- Reduced cost (Logic Apps are considerably expensive for many executions)
- Avoid outbound connection limit (Azure Functions enforce a strict outbound connection limit)
- Avoid time limit (Azure Functions must complete in a short amount of time)
- Configurable CPU and RAM for compute-intensive workloads
Mark Scurrell of Microsoft can be seen in the walk-through below demonstrating some of the capabilities of Azure Batch.
The first step is to create a batch account in the Microsoft Azure Portal. We will use this account to store the jobs, pools and tasks used for executing our code at large-scale.
Once created, you can navigate to the new batch account and view the ‘Keys’ associated with the account. These credentials are used when opening a new BatchClient in C#, used to access the batch account programmatically.
Create an application package
You can execute both C# and Python code in Azure Batch. For this example I will create a simple console application in C# to multiply 2 integers. This will simulate how Azure Batch is able to accept one or more arguments and execute code at large-scale in a production environment.
The EXE for our application package will contain the C# code block:
Zip the EXE and it’s associated files (including dependencies, configs, etc.)
Navigate to ‘Applications’ within your batch account in the Azure Portal and create a new application package. You will need to provide an Application ID (Multiply), Version (1.0) and upload the ZIP.
Use Azure Batch to Execute
To use Azure Batch in C#, you will need to install the Nuget package ‘Microsoft.Azure.Batch’ into your solution.
To begin the C# code to executing code using Azure Batch, open the BatchClient to access the Azure batch account. You will need to use the credentials you accessed earlier in the Azure Portal.
Within the BatchClient, we will begin to configure the pool of nodes used to execute the job and associated task(s). Below we are configuring a pool of 2 nodes and associating the application package we uploaded to the pool.
One of the benefits of Azure Batch is the flexibility of configuring the VM used in the pool. If you navigate to ‘Pools’ in your batch account and create a new pool, you will be able to view the many VM configuration options (operating system, CPU, RAM, etc.).
To reduce costs, the pool is created programmatically before the job and associated task(s) are created, and deleted once the tasks are complete.
To run the tasks, create a new job and associate it with the pool:
Create a new task and associate it with the job. In this example we will create 1 task to execute, however in a production environment we could use a loop to iterate through a collection of objects and create multiple tasks to run on many nodes.
Once the task is created and added to the job, we wait for the task to complete, applying a timeout of 30 minutes.
When the task is completed (regardless if it resulted in an error), we’ll print out the results to the console window.
Finally, delete the pool (to reduce costs) and terminate the job (to mark it as complete in the Azure Portal).
If the task was successful, you will be able to view the output of the application package, along with any error messages.
How to use Azure Batch Job Schedule
Please see the follow-up article for useful information on Azure Batch job schedule: https://kiltandcode.com/how-to-use-azure-batch-job-schedule/
Download and Install Batch Explorer
Microsoft has built a utility for Windows, Mac and Linux to help debug your Batch applications. You can download it here.
The code in this article is available at github.com/kiltandcode/azure-batch-demo