How to setup VSTS Private Agent to build Windows Server ver 1709 base Docker images

Do you know Windows had breaking changes for their virtualization technologies and I referred about that in Windows Container Version Compatibility | Microsoft Docs. This change will cause an error when you will build Windows Server ver 1709 base Docker images on VSTS build tasks.
Unfortunately, VSTS probably doesn't offer Hosted which are available to build Windows Server ver 1709 base Docker images. As far as I have checked, there are some "Hosted Agent" in Microsoft-hosted agents for VSTS | Microsoft Docs like below.

• Hosted VS2017 if your team uses Visual Studio 2017 - vsts-image-generation/Vs2017-Server2016-Readme.md at master · Microsoft/vsts-image-generation · GitHub
• Hosted Linux if your team uses development tools on Ubuntu.
• Hosted macOS Preview if your team uses development tools on macOS.
• Hosted if your team uses Visual Studio 2013 or Visual Studio 2015.

It was failed when I built my Windows Server ver 1709 base Docker images on VSTS build tasks, so you also need to setup your Private Agent for building Windows Server ver 1709 base Docker images. You can setup the VM following this article!

Step by Step to step Windows Server version 1709 based VM as Private Agent

You need to create new Virtual Machine on Azure Portal. Choose "Windows Server, version 1709 with Containers" for base VM, because it contains "docker.exe" command. But keep in mind the image doesn't contain "docker-compose".

You don't need to add special settings when you create VMs, but don't setup Network Security Group as completely closed to enable accessible VSTS service.

Access the VM using Remote Desktop and install Visual Studio 2017 into the VM, because the VM doesn't contain MSBuild and other commands for VSTS Build/Release processes. Follow below commands.

C:\Users\azureuser>powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\azureuser> curl https://aka.ms/vs/15/release/vs_community.exe -O vs_community.exe
PS C:\Users\azureuser> dir


    Directory: C:\Users\azureuser


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-r---        3/22/2018   4:19 PM                3D Objects
d-r---        3/22/2018   4:19 PM                Contacts
d-r---        3/22/2018   4:19 PM                Desktop
d-r---        3/22/2018   4:19 PM                Documents
d-r---        3/22/2018   4:19 PM                Downloads
d-r---        3/22/2018   4:19 PM                Favorites
d-r---        3/22/2018   4:19 PM                Links
d-r---        3/22/2018   4:19 PM                Music
d-r---        3/22/2018   4:19 PM                Pictures
d-r---        3/22/2018   4:19 PM                Saved Games
d-r---        3/22/2018   4:19 PM                Searches
d-r---        3/22/2018   4:19 PM                Videos
-a----        3/22/2018   4:24 PM        1180608 vs_community.exe

PS C:\Users\azureuser> .\vs_community.exe

I chose below settings in my case, but change the settings for your environment if you need.

After Visual Studio installation has completed, add MSBuild execution folder path into PATH environment variable like below.

PS C:\Users\azureuser> setx /M PATH "%PATH%;C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\B
in"

SUCCESS: Specified value was saved.
PS C:\Users\azureuser> 

Next, you also need to add "docker-compose" to build with it, because this base VM contains only just "docker" command. Follow below commands to install it.

PS C:\> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
PS C:\> Invoke-WebRequest "https://github.com/docker/compose/releases/download/1.20.0/docker-compose-Windows-x86_64.exe" -UseBasicParsing -OutFile $Env:ProgramFiles\docker\docker-compose.exe
PS C:\>

Finally, you need to setup VSTS Private Agent. Refer to How to setup your CentOS VMs as VSTS Private Agent - normalian blog to pick up "access token" for setup Private Agent. Note you need to setup user account of this Private Agent as "NT AUTHORITY\SYSTEM".

PS C:\agent> .\config.cmd

>> Connect:

Enter server URL > https://"your vsts account name".visualstudio.com
Enter authentication type (press enter for PAT) >
Enter personal access token > ****************************************************
Connecting to server ...

>> Register Agent:

Enter agent pool (press enter for default) > "Your Agent Pool Name"
Enter agent name (press enter for VSTSPAVM01) >
Scanning for tool capabilities.
Connecting to the server.
Successfully added the agent
Testing agent connection.
Enter work folder (press enter for _work) >
2018-03-18 05:09:24Z: Settings Saved.
Enter run agent as service? (Y/N) (press enter for N) > Y
Enter User account to use for the service (press enter for NT AUTHORITY\NETWORK SERVICE) > NT AUTHORITY\SYSTEM
Granting file permissions to 'NT AUTHORITY\SYSTEM'.
Service vstsagent.daisami-online.VSTSPAVM01 successfully installed
Service vstsagent.daisami-online.VSTSPAVM01 successfully set recovery option
Service vstsagent.daisami-online.VSTSPAVM01 successfully configured
Service vstsagent.daisami-online.VSTSPAVM01 started successfully
PS C:\agent>

Now, you can choose your Private Agent in your VSTS Build/Release processes.

Note: What will be error messages if you haven't complete docker-compose

You will get "##[error]Unhandled: Failed which: Not found docker: null" message from your VSTS build task.

2018-03-17T21:07:37.4182183Z ##[section]Starting: Build an image
2018-03-17T21:07:37.4186946Z ==============================================================================
2018-03-17T21:07:37.4187316Z Task         : Docker
2018-03-17T21:07:37.4187693Z Description  : Build, tag, push, or run Docker images, or run a Docker command. Task can be used with Docker or Azure Container registry.
2018-03-17T21:07:37.4188244Z Version      : 0.3.10
2018-03-17T21:07:37.4188534Z Author       : Microsoft Corporation
2018-03-17T21:07:37.4188879Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2018-03-17T21:07:37.4189247Z ==============================================================================
2018-03-17T21:07:37.6890544Z ##[error]Unhandled: Failed which: Not found docker: null
2018-03-17T21:07:37.6953107Z ##[section]Finishing: Build an image

Note: What will be error messages if you build Windows Server ver 1709 images with "Hosted VS2017" agent

You will get "The following Docker images are incompatible with the host operating system: [microsoft/aspnet:4.7.1-windowsservercore-1709]. Update the Dockerfile to specify a different base image." message from your VSTS build task.

2018-03-17T21:02:41.2336315Z 
2018-03-17T21:02:41.2336964Z Build FAILED.
2018-03-17T21:02:41.4137038Z 
2018-03-17T21:02:41.4138305Z "D:\a\1\s\Trunk\SFwithASPNetApp\SFwithASPNetApp.sln" (default target) (1) ->
2018-03-17T21:02:41.4139002Z "D:\a\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj" (default target) (3) ->
2018-03-17T21:02:41.4139575Z (DockerComposeBuild target) -> 
<b>2018-03-17T21:02:41.4141190Z   C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.Docker.targets(111,5): error : The following Docker images are incompatible with the host operating system: [microsoft/aspnet:4.7.1-windowsservercore-1709]. Update the Dockerfile to specify a different base image. See http://aka.ms/DockerToolsTroubleshooting for more details. [D:\a\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]</b>
2018-03-17T21:02:41.4142524Z 
2018-03-17T21:02:41.4142996Z     0 Warning(s)
2018-03-17T21:02:41.4143435Z     1 Error(s)
2018-03-17T21:02:41.4143702Z 
2018-03-17T21:02:41.4144142Z Time Elapsed 00:14:00.47
2018-03-17T21:02:43.2578358Z ##[error]Process 'msbuild.exe' exited with code '1'.
2018-03-17T21:02:44.0944814Z ##[section]Finishing: Build solution
2018-03-17T21:02:44.1143215Z ##[section]Starting: Post Job Cleanup
2018-03-17T21:02:44.1300074Z Cleaning any cached credential from repository: US-Crackle-Demo-Projects (Git)
2018-03-17T21:02:44.1413004Z ##[command]git remote set-url origin https://daisami-online.visualstudio.com/_git/US-Crackle-Demo-Projects
2018-03-17T21:02:44.3340613Z ##[command]git remote set-url --push origin https://daisami-online.visualstudio.com/_git/US-Crackle-Demo-Projects
2018-03-17T21:02:44.3757483Z ##[section]Finishing: Post Job Cleanup
2018-03-17T21:02:44.4763369Z ##[section]Finishing: Job

Note: What will be error messages if you setup Private Agent account as "NT AUTHORITY\NETWORK SERVICE"

Your VM can't access " //./pipe/docker_engine: " and the build tasks will be failed.

DockerGetServiceReferences:
docker-compose -f "C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.yml" -f "C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.override.yml" -p dockercompose13733567670188849996 --no-ansi config
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): Error MSB4018: The "GetServiceReferences" task failed unexpectedly.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: The "GetServiceReferences" task failed unexpectedly. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: Microsoft.Docker.Utilities.CommandLineClientException: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.30/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.VisualStudio.Docker.Compose.targets(195,5): error MSB4018: For more troubleshooting information, go to http://aka.ms/DockerToolsTroubleshooting ---> Microsoft.Docker.Utilities.CommandLineClientException: error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.30/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running. [C:\agent\_work\1\s\Trunk\SFwithASPNetApp\docker-compose.dcproj]

References

• Install Docker Compose | Docker Documentation
• Use command-line parameters to install Visual Studio - Visual Studio | Microsoft Docs

Add this comment at 8/8/2018 I found an article below. Refer this if your environment doesn't work well.
stackoverflow.com

How to build ASP.NET Framework Docker images on VSTS build tasks

As you know, Visual Studio Team Services offers to build Docker image tasks and Visual Studio offers Docker support. But there are some tips when you build ASP.NET Framework Docker images with VSTS build tasks. This post shows how to setup that.

You must note your OS versions of Docker images, because you must need to setup Private Agent if your base OS version is Windows Server version 1709. Refer below article at first if you need.
normalian.hatenablog.com

What's happened if you build ASP.NET Framework Docker images with default settings

You can choose "Add - Docker Support" like below if you have setup "Visual Studio Tools for Docker" in your machine. The tool is really useful and it will generate below Dockerfile in your ASP.NET Framework application.

FROM microsoft/aspnet:4.7.1-windowsservercore-1709
ARG source
WORKDIR /inetpub/wwwroot
COPY ${source:-obj/Docker/publish} .

Last "COPY" command line is a little bit tricky, because copy sources directory will change with ${source} variable value. The directory will be ${source} variable value if the value is set, but the directory will be obj/Docker/publish if ${source} variable isn't set.

This Dockerfile should work on your machine, but it won't work on VSTS build tasks like below.

2018-03-20T16:23:53.7207717Z ##[section]Starting: Build an image
2018-03-20T16:23:53.7213905Z ==============================================================================
2018-03-20T16:23:53.7214387Z Task         : Docker
2018-03-20T16:23:53.7214924Z Description  : Build, tag, push, or run Docker images, or run a Docker command. Task can be used with Docker or Azure Container registry.
2018-03-20T16:23:53.7215463Z Version      : 0.3.10
2018-03-20T16:23:53.7215860Z Author       : Microsoft Corporation
2018-03-20T16:23:53.7216376Z Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2018-03-20T16:23:53.7216914Z ==============================================================================
2018-03-20T16:23:54.9185089Z [command]"C:\Program Files\Docker\docker.exe" build -f C:\agent\_work\1\s\Trunk\SFwithASPNetApp\ASPNetApp01\Dockerfile -t xxxxxxxxxxxxxxxxxxxister.azurecr.io/yyyyyyyyyyy-demo-projects:80 C:\agent\_work\1\s\Trunk\SFwithASPNetApp\ASPNetApp01
2018-03-20T16:23:55.0276465Z Sending build context to Docker daemon  3.072kB
2018-03-20T16:23:55.0278113Z 
2018-03-20T16:23:55.0302771Z Step 1/4 : FROM microsoft/aspnet:4.7.1-windowsservercore-1709
2018-03-20T16:23:55.0313682Z  ---> dc3f4d701ead
2018-03-20T16:23:55.0315249Z Step 2/4 : ARG source
2018-03-20T16:23:55.0326718Z  ---> Using cache
2018-03-20T16:23:55.0328908Z  ---> 9a10d9b50bc9
2018-03-20T16:23:55.0329320Z Step 3/4 : WORKDIR /inetpub/wwwroot
2018-03-20T16:23:55.0340016Z  ---> Using cache
2018-03-20T16:23:55.0342052Z  ---> 28d5a9cc0dd0
2018-03-20T16:23:55.0342974Z Step 4/4 : COPY ${source:-obj/Docker/publish} .
2018-03-20T16:23:55.0348449Z COPY failed: GetFileAttributesEx \\?\C:\Windows\TEMP\docker-builder521937930\obj\Docker\publish: The system cannot find the path specified.
2018-03-20T16:23:55.0575409Z ##[error]C:\Program Files\Docker\docker.exe failed with return code: 1
2018-03-20T16:23:55.0588673Z ##[section]Finishing: Build an image

What is workaround of the issue?

You need to setup both Visual Studio and VSTS in this case. Please follow below steps .

Setup in your ASP.NET Framework project
At first you need to setup new pubxml file in your ASP.NET Framework project to ensure output binaries into "obj\Docker\publish" directory, so choose "Publish" from right click menu of your ASP.NET Framework project like below.

Choose "Folder" as target and edit "Choose a folder" as "obj\Docker\publish" like below.

After setup the pubxml file, you can find it in your ASP.NET Framework like below. In this case, the filename is "FolderProfile.pubxml".

Setup in your VSTS tasks
You need to update "Build Solution" tasks at first. Please note to add "/p:PublishProfile=FolderProfile.pubxml" and remove "/p:WebPublishMethod=Package".

• "MSBuild Arguments" - before change

/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"

• "MSBuild Arguments" - after change

/p:DeployOnBuild=true /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\" /p:PublishProfile=FolderProfile.pubxml 

After setup "Build Solution" tasks correctly, you can just add "Build an Image" task and specify "Docker file" correctly like below.

And you also need to add "Push an image" to store your Docker images into some registries. Note to specify as "Push an Image" not "Push images" like below.

You can find your Docker images in your Container registry if the build process on VSTS work correctly.

References

• Does not work out of the box. "The system cannot find the file specified" · Issue #65 · Microsoft/aspnet-docker · GitHub
• Visual Studio Tools for Docker with ASP.NET Core | Microsoft Docs
• docker - Dockerfile COPY {source:-...} - Stack Overflow

What is workaround when you got error messages "There was an error during download.Failed" while downloading container images

As you know, Service Fabric is one of implementations to offer Microservice Architecture provided by Microsoft. Of course, it can be deployed Docker Images both Windows and Linux base images, but you should note "Operating System" of Service Fabric cluster matches with Docker images when you want to deploy Docker images. There might be some reasons of error messages "There was an error during download.Failed" like below when you got the messages while deploying your images.

It's caused by some reasons and it should be one of below.

1. URL of your Docker image is invalid
2. The authentication info of your Docker repository account is invalid
3. There is virtualization mechanism mismatch between base OS of Docker images and operating system version of your Service Fabric cluster

No.1 and No.2 are trivial and not so difficult to fix it, but it's not easy to clarify when you got the message caused by No.3. In this article, I will dig into cause of No.3.

Docker container base images need to match the version of the host its running on. Unfortunately Windows made a breaking change where container images are not compatible across hosts like below article.
docs.microsoft.com
You need to specify your Service Fabric cluster "Operating System" based on your Docker image base OS like below image.

• You must specify "WindowsServer 2016-Datacenter-with-Containers" as Service Fabric cluster Operation System if your base OS is "Windows Server 2016"
• You must specify "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers" as Service Fabric cluster Operation System if your base OS is "Windows Server version 1709"

Example to match OS Versions

It's important to match your Service Fabric cluster "Operating System" and base OS version specified in Dockerfile "FROM" keyword. And I put ServiceManifest and ApplicationManifest.xml just in case.

Example - part of Dockerfile

# This base OS for "WindowsServer 2016-Datacenter-with-Containers"
#FROM microsoft/aspnetcore-build:2.0.5-2.1.4-nanoserver-sac2016 AS base
# This base OS for "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers"
FROM microsoft/aspnetcore:2.0-nanoserver-1709 AS base
WORKDIR /app
EXPOSE 80

# This base OS for "WindowsServer 2016-Datacenter-with-Containers"
#FROM microsoft/aspnetcore-build:2.0.5-2.1.4-nanoserver-sac2016 AS build
# This base OS for "WindowsServerSemiAnnual Datacenter-Core-1709-with-Containers"
FROM microsoft/aspnetcore-build:2.0-nanoserver-1709 AS build
WORKDIR /src
COPY *.sln ./
COPY NetCoreWebApp/NetCoreWebApp.csproj NetCoreWebApp/
RUN dotnet restore
COPY . .
WORKDIR /src/NetCoreWebApp
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "NetCoreWebApp.dll"]

Example - part of ApplicationManifest.xml

  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="GuestContainer1Pkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides />
    <Policies>
      <ContainerHostPolicies CodePackageRef="Code">
        <RepositoryCredentials AccountName="Username of your Container registry" Password="password of your Container registry" PasswordEncrypted="false"/>
        <PortBinding ContainerPort="80" EndpointRef="GuestContainer1TypeEndpoint"/>
      </ContainerHostPolicies>
    </Policies>
  </ServiceManifestImport>

Example - part of ServiceManifest.xml

    <EntryPoint>
      <!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
      <ContainerHost>
        <ImageName>"Username of your Container registry".azurecr.io/sample/helloworldapp:latest</ImageName>
      </ContainerHost>
    </EntryPoint>

References

• There was an error during download.Failed to download container image · Issue #752 · Azure/service-fabric-issues · GitHub
• Create an Azure Service Fabric container application | Microsoft Docs
• Windows Container Version Compatibility | Microsoft Docs

How to pass values generated on VSTS processes into other build/release tasks

When you deploy templates with some linked templates, the linked templates should be stored public or limited access wtih SAS token. But this sometimes makes difficult to setup CI/CD pipeline on Visual Studio Team Service(VSTS). You can understand how to setup this with this article and GitHub - normalian/ARMTemplate-SASToken-InVSTS-Sample.
You can generate SAS token with VSTS task in build process and pass the value with VSTS variables, and you can also override ARM template parameters with VSTS tasks. This is key concepts of this article.

In VSTS Build Process

Create "Azure PowerShell script" task and "Azure Deployment: Create Or Update Resource Group Action" like below.

Azure PowerShell script: inline Script – inline script
Edit "Azure PowerShell script" task like below.

$context = New-AzureStorageContext -StorageAccountName 'your storage account name' -StorageAccountKey 'your storage access key'
$sasUrl = New-AzureStorageContainerSASToken -Container templates -Permission rwdl -Context $context 
Write-Output ("##vso[task.setvariable variable=SasUrl;]$sasUrl")

You can store generated values with VSTS variables like above.

Azure Resource Group Deployment - Override template parameters
Edit "Azure Deployment: Create Or Update Resource Group Action" like below.

-SASToken $(SasUrl)

Part of ARM template

Now you can use SAS token to specify your linked templates like below. Refer this sample if you need.

    "variables": {
      "sharedTemplateUrl": "[concat('https://'your storage account name'.blob.core.windows.net/templates/blank-azuredeploy.json', parameters('SASToken') )]",
      "sharedParametersUrl": "[concat('https://'your storage account name'.blob.core.windows.net/templates/blank-azuredeploy.parameters.json', parameters('SASToken'))]"
    },

References

• GitHub - normalian/ARMTemplate-SASToken-InVSTS-Sample
• Setting VSTS variables from PowerShell
• Build variables | Microsoft Docs

How to setup simple Workflow with Azure Automation

You should read below article before following this article, because this article make a Azure Automation workflow collaborating Runbooks.
normalian.hatenablog.com
Azure Automation offers to collaborate with each Runbooks as Workflok, and you can setup your simple workflow with following this article!

Create your new Runbook as "PowerShell"

Create your new "PowrShell" Runbook under your Azure Automation account and edit it like below. This Runbook output your Azure resources in a location specified by a parameter.

Param
(
    [Parameter (Mandatory = $true)]
    [String] $Location = 'Japan East'
)

# Setup Authentication
$Conn = Get-AutomationConnection -Name AzureRunAsConnection
Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint

Get-AzureRmResourceGroup -Location $Location | ForEach-Object { Write-Output $_.ResourceGroupName }

You can specify parameters with "Param" keyword like above. "PowerShell Workflow" created in next section can call "PowerShell" Runbook, so you have to create your Runbooks as "PowerShell".

Create new Runbook as "PowerShell Workflow"

You can find how to pass your parameters and how to get output with your "PowerShell" Runbook.

workflow workflow-sample
{
    Param
    (
        [Parameter (Mandatory = $true)]
        [String] $Location01 = "West US",
        [Parameter (Mandatory = $true)]
        [String] $Location02 = "West Central US"
    )

    # settings
    $automationAccountName = "mytest-automation"
    $resourceGroupName = "mytest-automation-rg"

    # Setup Authentication
    $Conn = Get-AutomationConnection -Name AzureRunAsConnection
    Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint

    ## backup Runbook
    echo '#1 runbook starts'
    $params = @{ 'Location'=$Location01 }
    $runbookName = 'execute-azure-cmdlet'
    $job = Start-AzureRmAutomationRunbook -AutomationAccountName $automationAccountName -Name $runbookName -ResourceGroupName $resourceGroupName -Parameters $params
    $doLoop = $true
    While ($doLoop) {
        $job = Get-AzureRmAutomationJob –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName
        $status = $job.Status
        if($status -eq "Failed") {
            Write-Error "Error in $runbookName"
            Write-Error $job.Exception
            throw $job.Exception
        }
        $doLoop = (($status -ne "Completed") -and ($status -ne "Suspended") -and ($status -ne "Stopped"))
        Start-Sleep -Seconds 2
    }
    echo '################# output start #################'
    $record = Get-AzureRmAutomationJobOutput –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName –Stream Any | Get-AzureRmAutomationJobOutputRecord
    $record # for example
    echo '                               #################'
    $record | Where-Object { $_.Value.value -NE $null} | ForEach-Object { Write-Output $_.Value.value }
    echo '################# output end #################'
    echo '#1 runbook is ended'

    ## 
    echo '#2 runbook is starts'
    $params = @{ 'Location'=$Location02 }
    $runbookName = 'execute-azure-cmdlet'
    $job = Start-AzureRmAutomationRunbook -AutomationAccountName $automationAccountName -Name $runbookName -ResourceGroupName $resourceGroupName -Parameters $params
    $doLoop = $true
    While ($doLoop) {
        $job = Get-AzureRmAutomationJob –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName
        $status = $job.Status
        if($status -eq "Failed") {
            Write-Error "Error in $runbookName"
            Write-Error $job.Exception
            throw $job.Exception
        }
        $doLoop = (($status -ne "Completed") -and ($status -ne "Suspended") -and ($status -ne "Stopped"))
        Start-Sleep -Seconds 2
    }
    echo '################# output start #################'
    $record = Get-AzureRmAutomationJobOutput –AutomationAccountName $automationAccountName -Id $job.JobId -ResourceGroupName $resourceGroupName –Stream Any | Get-AzureRmAutomationJobOutputRecord
    $record | Where-Object { $_.Value.value -NE $null} | ForEach-Object { Write-Output $_.Value.value }
    echo '################# output end #################'
    echo '#2 runbook is ended'
}

Output logs with Workflow

You can execute your Workflow and find output logs like below.

PSComputerName        : localhost
PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Environments          : {AzureCloud, AzureChinaCloud, AzureUSGovernment}
Context               : Microsoft.Azure.Commands.Profile.Models.PSAzureContext




#1 runbook starts

################# output start #################

PSComputerName        : localhost

PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Value                 : {Environments, Context}
ResourceGroupName     : mytest-automation-rg
AutomationAccountName : mytest-automation
JobId                 : eb19892d-8e2d-4572-862f-9205ca6e89fc
StreamRecordId        : eb19892d-8e2d-4572-862f-9205ca6e89fc:00636563050813081260:00000000000000000001
Time                  : 03/10/2018 18:58:01 +00:00
Summary               : 
Type                  : Output

PSComputerName        : localhost
PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Value                 : {value}
ResourceGroupName     : mytest-automation-rg
AutomationAccountName : mytest-automation
JobId                 : eb19892d-8e2d-4572-862f-9205ca6e89fc
StreamRecordId        : eb19892d-8e2d-4572-862f-9205ca6e89fc:00636563050827143533:00000000000000000002
Time                  : 03/10/2018 18:58:02 +00:00
Summary               : normalian-datacatalog-rg
Type                  : Output

PSComputerName        : localhost
PSSourceJobInstanceId : 256fbcbd-f339-4ce5-b75b-0dc973dd0f2a
Value                 : {value}
ResourceGroupName     : mytest-automation-rg
AutomationAccountName : mytest-automation
JobId                 : eb19892d-8e2d-4572-862f-9205ca6e89fc
StreamRecordId        : eb19892d-8e2d-4572-862f-9205ca6e89fc:00636563050827612512:00000000000000000003
Time                  : 03/10/2018 18:58:02 +00:00
Summary               : sqldb-rg
Type                  : Output

                               #################

normalian-datacatalog-rg

sqldb-rg

################# output end #################

#1 runbook is ended

#2 runbook is starts

################# output start #################

demo-automation-rg

mytest-automation-rg

################# output end #################

#2 runbook is ended

References

• Runbook execution in Azure Automation | Microsoft Docs
• Runbook Output and Messages in Azure Automation | Microsoft Docs
• Runbook input parameters | Microsoft Docs

How to execute Microsoft Azure PowerShell commands on Azure Automation

As you know, Azure Automation is really great feature to automate your schedulable tasks both public cloud an on-premise. There are massive documents to describe how to do that including abstraction. I will introduce how to do that simply with screenshots.

Create your Azure Automation Account

At first, note when you create Azure Automation account. And you must create "Azure Run As account" like below, because it's mandatory to execute your Azure Automation scripts called as "Runbook". This probably need your Azure Active Directory privilege of App Registration.

Create your Runbook

Create a Runbook used to execute your scripts. Choose "Runbook" from left side of you Azure Automation account and click "Add a runbook" like below.

And input your Runbook name and choose "PowerShell" as your Runbook type.

Create your scripts into your Runbook

Open your Runbook and click "Edit" to create your script. Update your script like below.

$Conn = Get-AutomationConnection -Name AzureRunAsConnection
Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint
 
get-azurermresourcegroup | ForEach-Object { $_.ResourceGroupName }

The name of "AzureRunAsConnection" should be created in "'your Azure Automation account name'- Connections". Once again, this is mandatory to execute your script. Confirm it like below if you need.

After updating the script, click "Test pane" to test your script. You can execute your script by clicking "Start" button, so you can take result like below.

Now you can publish your script by clicking "Publish" button to schedule and collaborate with other Runbooks. After publishing that, confirm the status like below.

Schedule your Runbook

Go back to top of your Azure Automation account and choose "Schedule" and click "Add a schedule" like below.

In this example, I setup my schedule as weekly like below.

Finally, you have to associate with your Runbook and Schedule. Go back to your Runbook, choose "Schedule" and click "Add a schedule". Associate your schedule like below.

Now, you can execute your script based on your schedule.

How to revert new deployment to old one in Service Fabric

As you know, Service Fabric is one of services to achieve Microservice architecture. There are two options when you got bad deployments using Service Fabric.

• manual deployment: "Start-ServiceFabricApplicationUpgrade" PowerShell command
• VSTS deployment: create new Release using existing build packages

Revert with "Start-ServiceFabricApplicationUpgrade"

Service Fabric retains old application packages for a while like below. As far as I confirmed, it should retain more than 24 hours.

Meanwhile the retainment, you can revert from new deployment to old one with below PowerShell commands.

Login-AzureRmAccount

$applicationName = 'fabric:/FabricApp01'

$connectArgs = @{  ConnectionEndpoint = "'<your cluster name'".westus.cloudapp.azure.com:19000';  
                   X509Credential = $True;  
                   StoreLocation = "CurrentUser";  
                   StoreName = "My";  
                   ServerCommonName = "'your cluster name'.westus.cloudapp.azure.com";  
                   FindType = 'FindByThumbprint';  
                   # "Client certificates" thumbprint. Pick up this value from "security" item in your cluster on Azure Portal
                   FindValue = "YYYYYYYYYY7e3372bc1ed5cf62b435XXXXXXXXXX"; 
                   # "Cluster certificates" thumbprint.  Pick up this value from "security" item in your cluster on Azure Portal
                   ServerCertThumbprint = "YYYYYYYYYY2E67D7E54647A12B7787XXXXXXXXXX" } 
Connect-ServiceFabricCluster @connectArgs

$app = Get-ServiceFabricApplication -ApplicationName $applicationName
$app 
$table = @{}
$app.ApplicationParameters | ForEach-Object { $table.Add( $_.Name, $_.Value)}
Start-ServiceFabricApplicationUpgrade -ApplicationName $applicationName -ApplicationTypeVersion "1.0.2.52" -ApplicationParameter $table -UnmonitoredAuto

You can watch it progress in Service Fabric Explorer like below.

Revert with new Release using existing build packages

I believe you have already made some build packages for deployment into Service Fabric. You can create new Release in your VSTS using the packages like below.