Hi all,
today I want to tackle the issue of bulk copying more than one Azure ML experiment at once between different workspaces.
May be you already know that you can partially solve this task by copying an experiment one at a time. But you have to access to both the workspaces with your user. If not, you can simply share a workspace in this way:
Once you can see both the workspaces in your Azure Machine Learning Studio, you can simply select an experiment and than “Copy to workspace”:
and than you can choose the destination workspace:
A you can imagine… you can’t simply select more than one experiment and than copy all them:
Now suppose you have dozens of experiments and simply you don’t want to waste your time coping them all manually, or moreover you can’t have access to a shared workspace for security reasons. Is there a way to bulk copy your experiments? I’ll show you how to do that using few rows of PowerShell.
Azure ML PowerShell Configuration
First of all, you need to download the PowerShell Module for Azure Machine Learning Studio & Web Services from this link:
It allows you to interact with Azure Machine Learning Workspaces, Datasets, Trained Models, Transforms, Custom Modules, Experiments, Web Services and Web Service Endpoints.
Just click on the “%% releases” link:
and download the top most available download from the page:
Then be sure to unpack the two files (.dll and .json) in your preferred folder:
I unpacked them in my C:\Install\AzureMLPS\ folder.
Then clone the json file in two other files: one is used for your source workspace (config_source.json) and the other for your destination workspace (config_destination.json). If you open your config_source.json file in any editor, you can see this code:
{
"Location": "South Central US (or other supported region)",
"WorkspaceId": "",
"AuthorizationToken": ""
}
So you simply have to replace the < ... > tokens and the Location with the ones of your source workspace. In particular you can find these information as shown below:
In the example above I choose the secondary authorization token, but you could use the primary too. The two tokens are equivalent, they are there just to help you in case you have to rotate the keys.
So the content of my config_source.json file become:
{
"Location": "South Central US",
"WorkspaceId": "8******************************8",
"AuthorizationToken": "B**************************************="
}
In the same way, change the content of your config_destination.json file, so that it’ll refer the destination workspace.
Just Use the Scripts: Copy the Experiments between Workspaces
Ok, now it’s time to have fun with few rows of PowerShell script! Simply go to my repository:
and download the zip file:
Unpack the script files it contains in your preferred folder. Now open the Windows PowerShell ISE:
Then open the AzureML-Scripts_CopyExperimentsBetweenWorkspacesGUI.ps1 file just dragging it into the PowerShell ISE workspace (or using the usual Open FIle feature of all the IDEs).
Now make sure to edit the paths of your source and destination workspace config files:
Just push the Run button (the green arrow in the toolbar) or F5 and that’s it! A dazzling grid view windows will pop up. It contains all the experiments of your source workspace. You can filter the content of each column you can see using the Filter textbox and you can select one or more experiments holding down the CTRL or the SHIF key as usual:
Now push the OK button and the copy operation will start:
If your original experiment contains datasets, those are automatically deep-copied into the new workspace. Trained models or transforms are not copied instead. If there are any name collisions with existing assets, they get automatically renamed as well.
Just Use the Scripts: Export the Experiments in JSON Files
Suppose you can’t access to the destination workspace’s authorization tokens due to security policies, but you need to bulk copy some experiments to that workspace. The above mentioned script is not useful to you in this case. Also suppose you are using an Azure ML free account, that has a limited storage of 10 GB, and you are out of space. What if you want to backup your experiments so that you can delete them in the workspace to free up some space? Here a simply solution for both the cases.
Just open the the AzureML-Scripts_ExportExperimentsFromWorkspaceGUI.ps1 file and make sure to edit the path of your source workspace config file and the path of your destination folder that will contain the JSON files to be exported:
Then just push the Run button and a grid view window containing the experiments of the source workspace will pop up. Just select one or more experiment to export and than push the OK button. Your JSON files will be created in your destination folder:
Now you can zip and store them as a backup of your experiments, or you can send the zip file to the destination workspace owner so that he can import the experiments by himself.
Just Use the Scripts: Import the Experiments from JSON Files
If you have to restore some JSON files you created earlier, or if you receive some JSON files to import in your workspace, than you can use the AzureML-Scripts_ImportExperimentFromJsonFiles.ps1 file. As well as the scripts we have seen before, even in this case you have to edit just few parameters. You have to edit the path of the folder containing the source JSON files, the destination workspace config file and a new parameter: a boolean value ( $true or $false in PowerShell) that indicates if you want to assign to the experiments the corresponding JSON file name ( $true ) or the original experiment name ( $false ):
Then you can Run the script and a grid view window containing all the JSON files found in the source folder:
Just select one or more files and press the OK button. Your files will be imported as the apiTestUser author:
After then you can enjoy your experiments on the destination workspace.
Conclusions
Sometimes you have to deal with dozens of experiments to copy to a destination workspace. You can copy them manually using Azure ML Studio, but with some limitations:
- you must have access to the destination workspace after someone has shared it to you;
- then you can copy them one at a time.
Moreover you often need to make a copy of your experiments as a backup, especially if you need to free up some space into your workspace storage.
You can easily solve this cases using few PowerShell scripts I described above.
Enjoy your Azure Machine Learning experiments!
