normalian blog

I will introduce about Microsoft Azure, ASP.NET or Java EE

API Management and Service Fabric Collaboration for Global Scale Applications

As you know, you can achieve Microservice architecture by using Service Fabric, but you might want to need request routing features for your applications for multi languages, cross devices or others. In such a case, you can use API Management for it. In this article, you can learn how to setup API Management with Service Fabric.

Edit ServiceManifest.xml of your Serviec Fabric project

At first, make a REST API application and deploy it into your Service Fabric cluster. Note to edit "ServiceManifest.xml" file in your Service Fabric project not to specify actual port like below. This setup is needed to collaborate API Management and Service Fabric.

    <Endpoint Protocol="http" Name="ServiceEndpoint" Type="Input" />

Download certificate file to access Service Fabric cluster

You probably created the certificate automatically when you made your Service Fabric cluster. Go to your Service Fabric cluster, choose "security" tab and pick up the certificate thumbprint like below.

Next, download the certificate file as pfx into your machine. Go to KeyVault, choose "certificate" tab, select your certificate and choose "Download in PFX/PEM format" like below.

Save thumbprint and pfx file to use ARM Template in later section.

Deploy new API Management instance by using ARM Template

Download apim.json and apim.parameters.json ARM Templates from service-fabric-api-management/apim.json at master · Azure-Samples/service-fabric-api-management · GitHub. And add '"validateCertificateChain": false into apim.json' if you will use self-signed certificate file like below .

            "apiVersion": "2017-03-01",
            "type": "Microsoft.ApiManagement/service/backends",
            "name": "[concat(parameters('apimInstanceName'), '/', parameters('service_fabric_backend_name'))]",
            "dependsOn": [
                "[resourceId('Microsoft.ApiManagement/service', parameters('apimInstanceName'))]",
                "[resourceId('Microsoft.ApiManagement/service/certificates', parameters('apimInstanceName'), parameters('serviceFabricCertificateName'))]"
            "properties": {
                "description": "My Service Fabric backend",
                "url": "fabric:/fake/service",
                "protocol": "http",
                "resourceId": "[parameters('clusterHttpManagementEndpoint')]",
                    "validateCertificateChain": false
                "properties": {
                    "serviceFabricCluster": {
                        "managementEndpoints": [
                        "clientCertificateThumbprint": "[parameters('serviceFabricCertificateThumbprint')]",
                        "serverCertificateThumbprints": [
                        "maxPartitionResolutionRetries": 5

Update apim.parameters.json like below.

    "$schema": "",
    "contentVersion": "",
    "parameters": {
        "apimInstanceName": {
            "value": "sfapim01"
        "subnetName": {
            "value": "API-Subnet"
        "apimPublisherEmail": {
            "value": ""
        "apimSku": {
            "value": "Developer"
        "serviceFabricCertificateName": {
            "value": "Daichi Isami"
        "serviceFabricCertificate": {
            "value": "base64 encoded string of your pfx file. don't insert breaklines"
        "certificatePassword": {
            "value": ""
        "serviceFabricCertificateThumbprint": {
            "value": "your Cluster certificates thumbprint"
        "url_path": {
            "value": "/api/values"
        "clusterHttpManagementEndpoint": {
            "value": "https://'your cluster name'"
            "value": "<policies>\r\n  <inbound>\r\n    <base />\r\n    <set-backend-service backend-id=\"servicefabric\" sf-service-instance-name=\"fabric:/SFApiApp/Web1\" sf-resolve-condition=\"@((int)context.Response.StatusCode != 200)\" />\r\n  </inbound>\r\n  <backend>\r\n    <base />\r\n  </backend>\r\n  <outbound>\r\n    <base />\r\n  </outbound>\r\n  <on-error>\r\n    <base />\r\n  </on-error>\r\n</policies>"
        "policies_policy_name": {
            "value": "policy"
        "apis_service_fabric_app_name": {
            "value": "service-fabric-app"
        "apim_service_fabric_product_name": {
            "value": "service-fabric-api-product"
        "service_fabric_backend_name": {
            "value": "servicefabric"
        "apis_service_fabric_app_name_operation": {
            "value": "service-fabric-app-operation"
        "vnetName": {
            "value": "VNet-sf-sample01-1709cluster"
        "vnetVersion": {
            "value": "2017-03-01"
        "networkSecurityGroupName": {
            "value": "apim-vnet-security-03"
        "networkSecurityGroupVersion": {
            "value": "2017-03-01"

You can put blank for certificatePassword value if you created your certificate file automatically. Refer to commands for base64encode for your certificate below if you need.

$bytes = [System.IO.File]::ReadAllBytes("C:\temp\yourpfxfile.pfx")
$b64 = [System.Convert]::ToBase64String($bytes);

It should takes 30 or 40 minutes to complete this deployment.

Access your Service Fabric application via API Management

Go to "Developer Portal" of your API Management, choose "Service Fabric App" among APIs and click "Try it" button. Now, you can send requests to your API application via API Management like below.

Tips No.1: Troubleshoot - "Service Fabric exception when trying to resolve partition: A Security error has occurred, failed to verify remote certificate"

You might get error messages below if you use self-signed certificate file.

service-fabric-backend (1371 ms)
    "message": "Service Fabric exception when trying to resolve partition: A Security error has occurred, failed to verify remote certificate.",
    "serviceName": {},
    "resourceId": "",
    "managementEndpoint": [

You should forget to update apim.json. Refer to " Deploy new API Management instance by using ARM Template" section in this article.

Tips No.2: Don't use “Client certificates” for API Management

As you know, Service Fabric uses multiple certificates for itself. Note that use "Cluster certificates" not "Client certificates" for API Management.

How to setup CI/CD pipeline with Service Fabric, VSTS and Windows Container

We have tried lots of features to collaborate wtih Service Fabric, VSTS and Docker containers. I have realized it's needed to describe overview of the architecture, so you can learn it following this article.

Overview of Service Fabric, VSTS and Windows Container architecture

At first, refer to the architecture diagram below. Purple lines mean processing flows of the CI/CD pipeline and green lines mean its setting dependencies.

Create below resources to setup this architecture.

  • Service Fabric cluster
  • VSTS Proejct, Build Process and Release Process
  • Azure Container Registry
  • Virtual Machines for VSTS Private Agent

In this article, you can find references to Service Fabric cluster, VSTS and Virtula Machines. But create Azure Container Registry for yourself and it should be quite easy.

a - Setup Private Agent for VSTS Build Definitions

Unfortunately, Windows Docker base image sizes are about 1.5G. It takes much time to download and build Docker images if you don't use Pirvate Agent. By caching the Docker images, its building time can be largely reduced.

3. and 4. Deploy deployment artifacts into Service Fabric cluster and download your Docker images from Azure Container Registry

Create VSTS Release Definitions to deploy Windows Docker images into Service Fabric cluster

This article requires to setup below environment at first. Please refer to them before following this article.

You have to complete all setups of CI/CD cycle using Windows Docker images, Service Fabric cluster and VSTS except for VSTS Release Definitions before following this article. I believe you have already created artifacts using by your VSTS Build definition to deploy into your Service Fabric cluster. Now, you can deploy the artifact by following this article.

Create a Release Definition to use artifacts created by your Build Definition

Choose "Releases" item from the top of VSTS menus, click "+" icon from left side and choose create "Create release definition", so you can find below diagrams.

Next, click "Add artifact", choose "Build" as "Source type" and setup your "Project" and "Source(Build definition)" which you have already created before following this article. Refer to below image if you need.

Next, choose "Add environment" box and choose "Azure Service Fabric Deployment". Note that you need to complete
How to setup Service Fabric connections on VSTS - normalian blog to setup this. Refer to below image and table as you need.

Parameter Name Value note
Application Package $(system.defaultworkingdirectory)/**/drop/applicationpackage -
Cluster Connection sf-sample01-1709cluster You must finish How to setup Service Fabric connections on VSTS - normalian blog to setup this
Publish Profile $(system.defaultworkingdirectory)/**/drop/projectartifacts/**/PublishProfiles/Cloud.xml -

Execute Release definition

After completion to create the Release Definition, click "+Rlease" link on VSTS portal.

Next, you can execute your deployment process by clicking "Deploy" on VSTS portal like below. After execution of the process, you also be able to watch process progress by choosing "Logs" tab on VSTS portal.

This is "Logs" tab on VSTS.

Create Service Fabric Deployment Package with Docker images on VSTS Build Task

This article requires to setup below environment at first. Please refer to them before following this article.

I believe you have already created your own Docker images and pushed them into your Azure Container Registry. Now, you also need to specify the images with Service Fabric setting files to use them by your Service Fabric cluster.

How to setup Build tasks on VSTS

You need to add 5 tasks after your "Push an image" task, but you also need to add "PowerShell Script" task if you might need to use capital letters in your project names or something. Refer to How to override values of environment variables on VSTS tasks - normalian blog to override environment variables.

Now, we will introduce how to setup the tasks.

  • Replace Tokens
  • Build solution
  • Update Service Fabric Manifests
  • Copy Files
  • Publish Build Artifacts
Replace Tokens

You need to update ServiceManifest.xml to specify your Docker image for your Service Fabric cluster.

Parameter Name Value note
Root directory Trunk/SFwithASPNetApp/SFwithASPNetApp Specify as Service Fabric directory
Target files **/*.xml Specify to include ServiceManifest.xml

You need to edit your ServiceManifest.xml like below

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="GuestContainer1Pkg"
    <!-- This is the name of your ServiceType.
         The UseImplicitHost attribute indicates this is a guest service. -->
    <StatelessServiceType ServiceTypeName="GuestContainer1Type" UseImplicitHost="true" />

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
      <!-- Follow this link for more information about deploying Windows containers to Service Fabric: -->
        <ImageName>"your acr account name"{Build.Repository.Name}#:#{Build.BuildId}#</ImageName>
    <!-- Pass environment variables to your container: -->
      <EnvironmentVariable Name="VariableName" Value="VariableValue"/>
Build solution

You need to specify *.sfproj file to build your Service Fabric application like below.

Parameter Name Value note
Solution Trunk/SFwithASPNetApp/SFwithASPNetApp/SFwithASPNetApp.sfproj Specify your Service Fabric cluster *.sfproj file
MSBuild Arguments /t:Package /p:PackageLocation=$(build.artifactstagingdirectory)\applicationpackage Specify to create Service Fabric package
Platform $(BuildPlatform) -
Configuration $(BuildConfiguration) -
Update Service Fabric Manifests

You need to update your ServiceManifest.xml version number by environment variable.

Parameter Name Value note
Update Type Manifest versions -
Application Package $(build.artifactstagingdirectory)\applicationpackage -
Version Value .$(Build.BuildNumber) -
Copy Files

You also need to copy your application xml files.

Parameter Name Value note
Source Folder $(build.sourcesdirectory) -
Contents **\PublishProfiles\*.xml
Publish Build Artifacts

Finally, you can publish your build artifacts and you can use it in your Release process.

Parameter Name Value note
Path to publish $(build.artifactstagingdirectory) -
Artifact name drop -
Artifact publish location Visual Studio Team Services/TFS -

How to confirm build result

You can watch your build result logs in VSTS Build page like below, and you also can find your build number like below. The number is used for Docker images tags.

Replace configuration files with environment variables on VSTS tasks

I believe you definitely want to replace values of some files in your projects with environments variables when you setup Visual Studio Team Services Build/Release processes. There are some ways to replace the values, and I will introduce to use e "Replace Tokens" published in Marketplace.

How to use "Replace Tokens" on VSTS

Input "Replace Tokens" into search box when you add new tasks in your VSTS Build/Release process and click "Install" to initialize it.

After adding "Replace Tokens" task in your process, change "Root directory" and "Target files" to specify which files you want to change. In below example, I specify *.xml files in my "SFwithASPNetApp" project.

And finally refer below a part of Service Fabric ServiceManifest.xml. This xml file uses "Build.Repository.Name" and "Build.BuildId" environment variables to specify Docker image name.

  <CodePackage Name="Code" Version="1.0.0">
      <!-- Follow this link for more information about deploying Windows containers to Service Fabric: -->

The Docker image name will replace from "{Build.Repository.Name}#:#{Build.BuildId}#" into "" in this case.

Note that you must put "}#" not "}" as suffix token.

How to override values of environment variables on VSTS tasks

As you know VSTS can use environment variables in VSTS Build and Release tasks. It's really useful to dynamically change values of build and release process like below, but you should sometimes wants to override even in running tasks.

I have built Windows Docker images on with VSTS build tasks by specifying its name as $(Build.Repository.Name), the actual name is "US-XXXXXX-Demo-Projects", and I store them into Azure Container Registry. But unfortunately, Azure Container Registry stores Docker images as lowercase letters like below.

As this result, you need to change Docker image name from $(Build.Repository.Name), change repository name itself or override environment variable. This article shows how to override the value.

How to override environment variable values on VSTS tasks

You need to add "PowerShell" tasks into your build process, specify its "Type" as "Inline Script" and edit "Inline Script" like below.

$LowerBuildRepositoryName = "$(Build.Repository.Name)".ToLower()
Write-Output ("##vso[task.setvariable variable=Build.Repository.Name;]$LowerBuildRepositoryName")

Write-Host "Build.Repository.Name variable updates"

You can override any variables as you like to edit the inline script.

What is workaround when you got error message "failure in a Windows system call: No hypervisor is present on this system." on Service Fabric Explorer

As you know, we can deploy Docker Container images into Service Fabric clusters, but you need to note when you specify "Instance Size" of Service Fabric when you use Windows Container images. You will get below error messages if you don't specify "Ev3" or "Dv3" for Azure SKUs even though you use "hyperv isolation mode".

There was an error during CodePackage activation.Container failed to start for image:"your container image""your image name":"your tag name". container 4ef6c4ec64f3006db9f5cbe9541dcb77994ccd1ac8d018180b5d08ba0cf95803 encountered an error during CreateContainer: failure in a Windows system call: No hypervisor is present on this system. (0xc0351000) extra info: {"SystemType":"Container","Name":"4ef6c4ec64f3006db9f5cbe9541dcb77994ccd1ac8d018180b5d08ba0cf95803","Owner":"docker","IsDummy":false,"IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\4ef6c4ec64f3006db9f5cbe9541dcb77994ccd1ac8d018180b5d08ba0cf95803","Layers":[{"ID":"1fd99b8d-bc0e-5048-9a10-5eaf17053f7a","Path":"C:\\ProgramData\\docker\\windowsfilter\\b8452e48f79716c4f811a2c80d3f32d4a37c9fb98fb179c6cffce7f0beed1e66"},{"ID":"758bc832-f0e5-55a8-a2ca-8db4d04cb9bd","Path":"C:\\ProgramData\\docker\\windowsfilter\\4e98f3616f260045e987e017b3894dcfa250c7f595997110c9900b02488e05f3"},{"ID":"91a87634-6f3e-59a9-9578-33049cc2ebaa","Path":"C:\\ProgramData\\docker\\windowsfilter\\f18e514d9d6c1d8d892856392b96f939d9f992cc7395d0b2d6f05e22216ac737"},{"ID":"245ef2e6-f122-5f3b-96ed-71d43099508b","Path":"C:\\ProgramData\\docker\\windowsfilter\\715c1bdc318c7b012e2f70d3798f4e1e79a96d2fa165bac377f7540030f1a1a6"},{"ID":"22fbc57d-ea4d-552b-9b3a-87e76e093d2b","Path":"C:\\ProgramData\\docker\\windowsfilter\\80a287a21e7eccba51d5110e25517639351e87f52067a7df382edcdaf312138b"},{"ID":"a451ab3e-a3fd-5db9-8c91-7d69034cd20a","Path":"C:\\ProgramData\\docker\\windowsfilter\\94c121ca13cfde5bf579f36687d5405d0487ad972dd18ff8f598870aa3a72b73"},{"ID":"17dbc8c4-0c4a-548b-857c-5e7928cce5f1","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e9fa399255bb3ba2717dcdd6ca06d493fa189c6d6a0276d0713f2a524dd3d2a"},{"ID":"20a857ef-13bb-55cf-8a67-2e18591c66bc","Path":"C:\\ProgramData\\docker\\windowsfilter\\12e7f9bda0627587e24bb4d5a3fb91c3ed9b6b6943ac1d35244afac547796dd1"},{"ID":"6d4958bb-c89f-5390-9420-5ddaff8ef0ca","Path":"C:\\ProgramData\\docker\\windowsfilter\\a4f9f68812499ffdaed2e84a6b84d8d878286f4d726b91bb7970b618f1d8dd65"},{"ID":"9067a7bd-7072-5844-8e54-f91888468462","Path":"C:\\ProgramData\\docker\\windowsfilter\\7efa7258bfec74a13dc8bbd93d4d0bc108ac6443ae0691198ff6a9a692c703f7"},{"ID":"edf15855-a46a-593c-a2c2-e476ccd00f3f","Path":"C:\\ProgramData\\docker\\windowsfilter\\38b937d97d3e4149be3671ae411441b9e839e2e0e40c5f626479357a0de8da00"},{"ID":"b4b0754d-1b7f-5413-ad1c-63616d0c72ff","Path":"C:\\ProgramData\\docker\\windowsfilter\\8753c8e07b95e6d80767dd39ace29e6e1108992ec72e46140ba990287272f418"},{"ID":"11cfb709-b618-529b-9165-d6d12db7330c","Path":"C:\\ProgramData\\docker\\windowsfilter\\d4a3ef41985c9ed1d7308f1f8603e8527154104bd7b308fed9f731f863d29314"}],"HostName":"4ef6c4ec64f3","MappedDirectories":[{"HostPath":"d:\\svcfab\\log\\containers\\sf-15-aa4017f4-24d0-437a-b7b5-a48db8717a13_3b9b217e-374e-48ea-8ee5-ccf597e47d34","ContainerPath":"c:\\sffabriclog","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0},{"HostPath":"d:\\svcfab\\_fronttype_0\\fabric","ContainerPath":"c:\\sfpackageroot","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0},{"HostPath":"c:\\program files\\microsoft service fabric\\bin\\fabric\\fabric.code","ContainerPath":"c:\\sffabricbin","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0},{"HostPath":"d:\\svcfab\\_app\\sfwithaspnetapptype_app15","ContainerPath":"c:\\sfapplications\\sfwithaspnetapptype_app15","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0}],"SandboxPath":"C:\\ProgramData\\docker\\windowsfilter","HvPartition":true,"EndpointList":["a118be34-14d8-48df-87be-2abaf0f40160"],"HvRuntime":{"ImagePath":"C:\\ProgramData\\docker\\windowsfilter\\8753c8e07b95e6d80767dd39ace29e6e1108992ec72e46140ba990287272f418\\UtilityVM"},"Servicing":false,"AllowUnqualifiedDNSQuery":true,"DNSSearchList":"SFwithASPNetApp"}


Refer to Create an Azure Service Fabric container application | Microsoft Docs and watch the note. Next, confirm your ApplicationManifest.xml file in your Service Fabric project. It should be specified as "hyperv" isolation mode in attribute of "ContainerHostPolicies " tag if you have hyperv role enabled.

    <ServiceManifestRef ServiceManifestName="GuestContainer1Pkg" ServiceManifestVersion="1.0.5" />
    <ConfigOverrides />
      <ContainerHostPolicies CodePackageRef="Code" Isolation="hyperv">
        <RepositoryCredentials AccountName="your account name" Password="your password" PasswordEncrypted="false"/>
        <PortBinding ContainerPort="80" EndpointRef="GuestContainer1TypeEndpoint"/>

This error is caused by "you are running hyperV container on a machine that does not have hyperv role enabled", so you probably need to recreate your Service Fabric cluster as "Ev3" or "Dv3" instance size, because it might not be able to change SKU into different series.