Azure ARM Virtual Machines, Nested Loops, winRM, and Custom Extensions

What is Azure Resource Manager?

Azure Resource Manager (ARM) is Microsoft Azure’s managed automation hosting service. ARM enables an automation designer to define their design intent, expressed as resource specifications using ARM's declarative, template automation language.

ARM requires designers to express their design intent:

• As the selection, implementation, deployment and provision of computing resource objects, together with
• The assembly sequence of those computing resource objects, that is, the assembly of compute, storage, and network resource objects.

ARM provides a mapping from compute, network or storage resource specifications to resource objects rendered by ARM onto a resource object canvas, located in an automation owner’s Azure subscription namespace, visible through Azure’s secured resource management portal. 

When Is This Automation Used?

This automation is used when an automation designer intends to implement, install, and provision one or more virtual machine assemblies.

This design extends the automation described in the earlier article to include the installation of 1 to n virtual machine assemblies, and to provision winRM, initialize data disks, and configure an application log on each virtual machine. A virtual machine is assembled from a collection of resource objects: compute, storage, network, and the public IP address. 

Design: Virtual Machine Assemblies Requirements

The automation’s design requirements are:

• Install two virtual machine assemblies.
• Attach to each virtual machine two managed disk resource objects, for data.
• Retrieve virtual machine administrator password from a secure location, like the Azure key vault.
• Use existing public IP (pip) address resource objects.
• Use existing network interface card (NIC) resource objects.
• Attach NIC to existing subnet and VNET resource objects.
• Provision each virtual machine with winRM.
• Provision each virtual machine with a Windows Event Viewer application log.
• Initialize each virtual machine’s data managed disks.
• Use a secure private storage location to store nested templates and custom script virtual machine extensions.

Design: Activity Composition

This design uses sequenced activities and a similar collection of loops and nested activities to that used by the automation described in the earlier article. For details about activity composition and the nested control mechanism see that article. 

This design uses two sequenced activities — activity n and n+1, each having an activity composition chain three levels deep — to install a virtual machine assembly and to provision the installed VMAs with winRM, initialized data managed disks, and an application log.

Design: Activity N, Install VM Assemblies

Activity n, install-1toN-vmas-0toN-dmd-kv, installs the virtual machine assemblies. The number of virtual machine assemblies installed is determined using the copy loop count vmNamesCount , at level 1 in the activity composition chain. 

The level 2 and 3 activities, assemble-vma-new-osmd-0toN-dmd-kv and assemble-new-dmd, are repeated for each virtual machine assembly using the copy statement in level 1. 

The level 2 activity has two resource specifications. The first, a link to the level 3 activity, assemble-new-dmd, which is the resource specification for the two managed disk resource objects used for virtual machine for data storage. 

The second resource specification assembles and installs a virtual machine resource object using the resource objects, like the nic, pip, disks, and vm, named in the virtual machine resource specification.

The admin password for each virtual machine — each VM uses the same password in this design — is stored in the automation's key vault and retrieved by ARM when the virtual machine resource object is rendered. For implementation details, see activity n implementation, below. 

Design: Activity N+1, Install VM Extensions

Activity n + 1, install-0toN-vmExts-on-1toN-vms, provisions proprietary 
and custom virtual machine extensions, like bgInfo and ConfigureWinRmAndLog, on each installed virtual machine. 

ARM uses the Microsoft.Compute/virtualMachines/extensions resource object, and naming conventions, to provision both proprietary and custom virtual machine extensions.

This design uses separate resource specifications to provision proprietary and custom script extensions. The activity install-0toN-pVmExts provisions the proprietary bgInfo and antiMalware extensions. And the activity provision-customScript-vmExts provisions the custom script ConfigureWinRmAndLog.ps1 extension. 

For the installation of proprietary custom scripts see the earlier article.

ARM enables only one custom script virtual machine extension to be installed on a virtual machine. When virtual machine provisioning requires multiple scripts either the scripts are chained, as is done below to initialize empty disks, or the first custom script extension must be removed from the virtual machine and a new extension resource object installed.

The custom script ConfigureWinRmAndLog.ps1 installs the application log, configures winRM, and then chains to the script, initialize-emptyDisk-onNomVm.ps1, to initialize the data managed disks on the nominated virtual machine. 

Design: Resource Groups and Secure Private Storage

ARM requires designers to group resource objects in resource group objects. ARM’s guidance to designers is to group resource objects with similar lifecycles in the same resource group object.

This design has two resource groups:

1. tpt-algoR-ops-rg, to group resource objects with long stable lifecycles, for example, the automation’s key vault tpt-algoRigs-kv, and the operations storage account tptalgorigsops10strgacc. The blob collection container, prjtplatesandscripts, used as the private store for automation templates and scripts, is located in the operations storage account.
2. tpt-algoR-vms-rg, to group components for virtual machine assemblies. The component resource objects for each virtual machine assembly are a network interface card, a public IP address, two data managed disks, a virtual machine, and an OS-managed disk.

For the structure of the operations storage resource group see the implementation description below. 

The resource specifications for nested activity levels 2 and 3 are implemented using Azure’s external template syntax described here.

The templates are located in the private storage account tptalgorigsops10strgacc, blobs collection prjtplatesandscripts. A private storage account is secured by a shared access signature, also known as an SAS token.

When ARM renders virtual machine assembly resource objects, a sas token is used to access the task templates at the secure private storage location. For details see, implementation activity n, below.

Implementation: Activity n, Install VM Assemblies

Activity n, implemented by task template install-1toN-vmas-0toN-dmd-kv.json, installs each virtual machine assembly. 

The admin password for each virtual machine is retrieved by ARM, from the key vault resource object tpt-algoRigs-kv, and passed from the level 1 template install-1toN-vmas-0toN-dmd-kv.json as a secure string argument into the level 2 template, assemble-vma-new-osmd-0toN-dmd-kv.json.  

A task template implementing an activity runs ‘in’ a resource group object, as a consequence a reference to a resource object, located in a different resource group tpt-algoR-ops-rg is qualified by its resource group name. 

The key vault is located in the operations resource group, not the virtual machine assemblies’ resource group, so the key vault, referenced using the resource object resource ID, includes the key vault’s resource group: 

"kvId": "[resourceId(variables('kvRgName'), 'Microsoft.KeyVault/vaults', variables('kvName'))]"

The SAS token that secures the private storage containing the nested templates and scripts is passed to task template install-1toN-vmas-0toN-dmd-kv.json, as a secure string argument and subsequently passed to the level two template assemble-vma-new-osmd-0toN-dmd-kv.json. 

A SAS token is retrieved from the Azure portal here:

A virtual machine can be provisioned with winRM using either:

• The Azure DevOps release pipeline available through the Azure DevOps portal, or
• A custom script virtual machine extension as done here, see activity n+1 implementation, below.

To provision winRM using the Azure DevOps portal release pipeline, set the ARM advanced deployment options. 

Note the “VM details for WinRM” is a release pipeline variable name holding the value of the resource group containing the virtual machines.

Implementation: Activity N+1, Install VM Extensions

For a description about provisioning proprietary custom script virtual extensions e.g. bgInfo, see the earlier article.

To provision winRM using a custom script virtual machine extension specification, this design uses the level 2 template provision-customScript-vmExts.json, initiated from level 1. 

Existing installed virtual machines are provisioned by the level 1 activity install-0toN-vmExts-on-1toN-vms implemented by task template install-0toN-vmExts-on-1toN-vms.json. 

The level 1 template install-0toN-vmExts-on-1toN-vms.json receives the vmCustomScriptVmExtNames  string array argument locating and naming PowerShell scripts used by the custom script virtual machine extension: 

"vmCustomScriptVmExtNames": [            
  "https://tptalgorigsops10strgacc.blob.core.windows.net/prjtplatesandscripts/compute/vmExtensions/vmExtScripts/winRM/ConfigureWinRmAndLog.ps1",             
  "https://tptalgorigsops10strgacc.blob.core.windows.net/prjtplatesandscripts/compute/vmExtensions/vmExtScripts/winRM/initialize-emptyDisk-onNomVm.ps1"
]

The vmCustomScriptVmExtNames  strings array is passed to the level 2 template provision-customScript-vmExts.json which implements the custom script virtual machine extension resource specification, Microsoft.compute/virtualMachines/extensions. 

The necessary level 2 extension resource specification properties are:

• " strgAccId ", the template variable value, is a resource Id, qualified by a resource group for the reason described above: 

"strgAccId": "[resourceId(parameters('opsRgName'), 'Microsoft.Storage/storageAccounts', variables('strgAccName'))]"

• “ hostDNSNameScriptArgument ”, the template variable value, 

"hostDNSNameScriptArgument": "[concat (variables('vmName'), '.southeastasia.cloudapp.azure.com')]"

is used as the subject of the self-signed certificate that enables winRM access to each virtual machine, as shown here on an installed VM:

"commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted  -file ./compute/vmExtensions/vmExtScripts/winRM/ConfigureWinRmAndLog.ps1', variables('hostDNSNameScriptArgument'))]"

"storageAccountKey": "[listKeys(variables('strgAccId'), providers('Microsoft.Storage','storageAccounts').apiVersions[0]).keys[0].value]"

All virtual machines in the nominated resource group need to be running when applying a virtual machine extension, even those not targeted for change. The change applies to all virtual machines in the resource group.  Otherwise, this message appears in the log, ‘Cannot modify extensions in the VM when the VM is not running.”

Implementation: Custom Script VM Extension, Pathnames

The file path used by a PowerShell "dot source" statement in a custom script virtual machine extension to locate the script to chain must match the location on the virtual machine to which the custom extension resource copies the fileUris used to provision the virtual machine, as used here in script ConfigureWinRmAndLog.ps1. 

## Initialize empty disks
   Write-EventLog -LogName  $logNameToInstall -Source $logNameSrc -EventId 1000 `
    -Message "Verify, next message confirms extension, initialize-emptyDisk-onNomVm.ps1, ran."
    . ./compute/vmExtensions/vmExtScripts/winRM/initialize-emptyDisk-onNomVm.ps1 

Which is copied to here,

Note that when a custom script extension is not working, some log info is available here:

And here:

Implementation: Storage and Pre-Installed Resource Objects

Some resource objects must be installed prior to those resource objects which are either a) assembled from them, or b) consume their services. For example, a virtual machine’s admin user password, securely stored as a key vault secret in a key vault resource object, must be installed before virtual machine resource objects are installed. Similarly, resource group objects must be installed before virtual machine resource objects are installed. As are the subnet and virtual network resource objects. 

Prior to ARM rendering this design’s resource specifications on the owner’s ARM canvas this automation design requires the two necessary resource group objects to be installed, tpt-algoR-ops-rg and tpt-algoR-vms-rg.

Similar to:

• Automation operations resource group name, tpt-algoR-ops-rg.
  • Key vault name, tpt-algoRigs-kv. 
    • Virtual machine admin password secret name, kvSecretVmAdminPword.
  • Storage account name, tptalgorigsops10strgacc.
    • Blob collection container name, prjtplatesandscripts. The following content needs to be stored at the nominated locations rooted in that container:
      • Nested Azure task templates: prjtplatesandscripts/compute/virtualMachines/
        • assemble-vma-new-osmd-0toN-dmd-kv.json
        • assemble-new-dmd.json
      • prjtplatesandscripts/compute/vmExtensions
        • install-0toN-pVmExts.json
        • provision-customScript-vmExts.json
        • provision-pVmExt-antiMalware.json
        • provision-pVmExt-bgInfo.json
      • Virtual machine extension custom scripts:
        • ConfigureWinRmAndLog.ps1
        • initialize-emptyDisk-onNomVm.ps1
• Virtual machine assembly resource group name, tpt-algoR-vms-rg.
  • Public ip address name, <vmname>-pip.
  • Network interface card name, <vmname>-nic.

Implementation: DevOps Release Pipeline and Activities

The Azure DevOps release pipeline action is install-1toN-vmas-0toN-dmd-ext.

The significant activities implementing the action are:

Notice the (disabled) install-pips and install-nics activities precede, and have installed their resource objects before, the virtual machines and their data disks are installed. The activity install-0toN-vmExts-on-1toN-vms installs the custom script vm extensions after the virtual machines are installed.

Verify: Installed Virtual Machine Assemblies and Provisioning

Verify resource group tpt-algoR-vms-rg, virtual machine assembly components at the ARM portal,

Verify VMs at the ARM portal:

Verify a VM at the ARM portal.

Verify VM extensions through VM activity log at the ARM portal.

Verify VM extensions for a VM at ARM portal:

And:

Verify winRM at the installed VM:

Verify certificate:

Verify disk initialization:

Verify application event log:

Resources

All information in this article is intended to provide general information only.  It does not purport to be comprehensive advice.

An automation’s resource objects can be installed on an owner’s ARM canvas using these Azure tools:

• ARM portal
• Command line interface, aka CLI
• Powershell
• DevOps release pipeline — the method used above. 

The Azure DevOps task type, Azure Resource Group Deployment, creates a resource group if it doesn’t exist. Further guidance is here and here.

Example resource specification templates for use with the above tools e.g. public ip, nic, are here. 

This design’s resource specification templates and scrips, intended for elucidation not for production, are available here, under an MIT license.