As a developer I want an automated mechanism to build, test and deploy everything I do, so when my team and I came to implementing something with AzureML it was the first thing I was challenged on. We have a group of analysts who want to use AzureML to build part of our system without having to translate their requirements into a language that we developers can use to build a system in code. I’d been playing around with AzureML and deployed a few services as part of a proof of concept but hadn’t looked at how we could automated the deployment process and add some control. We didn’t want to have a mechanism that would allow the analysts to build and deploy a new model to production, without some way to check whether what they have built was fit for purpose. After a bit of research I found that we could export both the experiment and the web service as JSON from AzureML. Exporting both the experiment and the web service definitions would allow us to version control the source. We could also import these definitions to allow us to move the experiment and web service to different subscriptions. Exporting and importing the experiment was relatively straight forward using PowerShell.
Export-AmlExperiementGraph & Import-AmlExperiementGraph
The full PowerShell to export and import the experiment is:
Get-AmlWorkspace -ConfigFile .\config-source.json
$exp = Get-AmlExperiment | where Description -eq '<ExperrimentName>' -ConfigFile .\config-source.json
Export-AmlExperimentGraph -ExperimentId $exp.ExperimentId -OutputFile 'C:\experiment.json' -ConfigFile .\config-source.json
Get-AmlWorkspace -ConfigFile .\config-dest.json
Import-AmlExperimentGraph -InputFile 'C:\experiment.json' -ConfigFile .\config-dest.json
Note: This relies on two config files, one to identify the source workspace and the other to identify the destination workspace. These are in the following formats:
{
"Location": "West Europe",
"WorkspaceId": "<WorkspaceID",
"AuthorizationToken": "<AuthorizationToken>"
}
You can find the workspace id and authorization tokens by opening your AzureML workspace then clicking settings.
Exporting the experiment as JSON allows you to add it to source control and version the experiment.
Importing the experiment into a new Workspace will allow you to open the workspace and view/edit the experiment and also to manually deploy the web service. This on its own will give you some control over the web service and allow you to control how and when it gets deployed and will stop the analytics team from accidentally deploying something to production and potentially breaking the system.
It is also possible to export a deployed web service and then import it into a new subscription.
When we came to trying to export the web service we ran into a few issues. Firstly there seemed to be a number of ways to export the web service definition and they seemed to produce different JSON.
https://github.com/ritwik20/AzureML-WebServices
https://github.com/hning86/azuremlps#export-amlwebservicedefinitionfromexperiment
https://docs.microsoft.com/en-us/powershell/module/azurerm.machinelearning/export-azurermmlwebservice?view=azurermps-2.2.0
We settled for the last link but also took some missing information from the first link. The process for exporting is a little more complicated.
Firstly, the web service definition needs to be exported as a JSON file. The web service import uses resource manager and requires a different mechanism to login to the experiment export/import. I used:
Login-AzureRmAccount –SubscriptionId <My Subscription ID>
This use the interactive login so if you want to automate this then you need to use the service principle
To export the web service:
$webService = Get-AzureRmMlWebService -Name "Source Service Name" -ResourceGroupName "Source Resource Group Name"
Export-AzureRmMlWebService -WebService $webService -OutputFile 'C:\wsexport.json'
The JSON then needs editing to add in the new storage account and commitment plan.
The JSON can then be imported into the new subscription using New-AzureRmMlWebService
Changing the JSON to add in the commitment plan id seemed to cause problems and I kept getting the error “Commitment Plan ID must be provided”. This error was confusing as I was including the commitment plan id in the configuration and I thought that I had it in the correct place. If you open the JSON that you export and find the storage account node then you will need to overwrite this with:
"storageAccount":{
"name": "<StorageAccountName>",
"key": "<StorageAccountKey>" },
"commitmentPlan": {
"id": "/subscriptions/0<Subscription-ID>/resourceGroups/<ResourceGroupName>/providers/Microsoft.MachineLearning/commitmentPlans/<CommitmentPlanName>"},
My issue was that I had the commitment id wrong rather than in the wrong place and I found the correct id using:
Get-AzureRmMlCommitmentPlan
Once I had this configured correctly the import worked without any errors using:
New-AzureRmMlWebService -Force -ResourceGroupName "New Resource Group Name" -Name "Web Service Name" -Location "West Europe" -DefinitionFile "C:\wsexport.json"
As we’ve used PowerShell to automate the export and import we could then easily script the config file edits and wire this in to an automated test and deploy process. We can write tests that check the web service parameters have not been changed by the analytics team and that the return data is the correct format, so we can ensure that the deployed service will at least function correctly. It’s more difficult to check whether the actual machine learning process is correct, but we will know when the interfaces are broken. We have also version controlled both the experiment and web service JSON so we can easily roll back if necessary. We decided that we needed both so that the experiment didn’t need to be copied to a new workspace and then automate a web service deploy, we just needed the web service in the new subscription, but we wanted the version control for the experiment too.