normalian blog

Let's talk about Microsoft Azure, ASP.NET and Java!

Setup tips for SQL DB auto export PowerShell scripts

SQL Database offered to backup SQL Database instances with their build-in features, but it was expired now. You can choose below options.

In this post, I will introduce setup tips for the scripts. Please read README of "Automate export PowerShell script with Azure Automation" to setup this script at first.

Add SQL DB instances into single script

You can add other databases to add them into “$databaseServerPairs” in below code.

And please use other credentials if you use other SQL Database servers.

Export error when SQL DB instances so large

Please read this section when you get below error.
f:id:waritohutsu:20180218102309p:plain

The error message is caused by below line.
- https://github.com/Microsoft/sql-server-samples/blob/master/samples/manage/azure-automation-automated-export/AutoExport.ps1#L115

The error is caused by below, so it seems to take too long time to copy DB data.

  if((-not $? -and $global:retryLimit -ile $dbObj.RetryCount) -or ($currentTime - $dbObj.OperationStartTime).TotalMinutes -gt $global:waitInMinutes)

Please change variable “$waitInMinutes = 30;” from 30 minutes to long time.

In order to execute the runbook do I need to have the automation account to have the ability to “Run As account”?

“Azure Run As account” is needed, because we can't execute Runbook scripts without this. It needs to enable Azure Active Directory to register applications.
https://docs.microsoft.com/en-us/azure/automation/automation-create-aduser-account#create-an-automation-account-in-the-azure-portal

" 429 Too many requests" error in Runbook Job log when exporting large SQL Database instances

You will get below error when you execute long jobs.

Get-AzureSqlDatabaseImportExportStatus : A task was canceled.
At line:181 char:11
+ ...    $check = Get-AzureSqlDatabaseImportExportStatus -Request $dbObj.Ex ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AzureSqlDatabaseImportExportStatus], TaskCanceledException
    + FullyQualifiedErrorId : 
Microsoft.WindowsAzure.Commands.SqlDatabase.Database.Cmdlet.GetAzureSqlDatabaseImportExportStatus

The error is caused by frequent requests using “Get-AzureSqlDatabaseImportExportStatus”, so it need to insert “Start-Sleep” in the script to reduce Azure Management API calls internally.

How to setup Service Fabric connections on VSTS

Visual Studio Team Service, VSTS, is really powerful tool to achieve your CI/CD pipeline. Before setting up Service Fabric connections, you need to create a *.pfx file to register as "Client Admin" certificate into your Service Fabric cluster. Please refer
Step by step how to setup Service Fabric Explorer on Azure - normalian blog if you have registered no *.pfx files as "Admin Client" certificates yet.

Create BASE64 string from your *.pfx file

Create BASE64 string for registering on VSTS portal to setup Service Fabric cluster connections.

PS C:\Users\normalian> [System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes("D:\temp\yourpfxfile.pfx"))
MIIJ+gIBAzCCCbYGCSqGSIb3DQEHAaCCCacEggmjMIIJnzCCBgAGCSqGSIb3DQEHAaCCBfEEggXtMIIF6TCCBeUGCyqGSIb3DQEMCgECoIIE9jCCBPIwHAYKKoZIh
"omission"
OBBRKwq7BWPo3ZdSGscBgAYKIhP8yGwICB9A=

Pick up and save the BASE64 string.

Setup on VSTS portal

Go to your VSTS project page and choose right side icon and "Services" item like below.
f:id:waritohutsu:20180216091954p:plain

Click "New Service Endpoint" and choose "Service Fabric" like below.
f:id:waritohutsu:20180216092050p:plain

Input your info into "Add new Service Fabric Connection" wizard like below. Input *.pfx file password into "Password" section.
f:id:waritohutsu:20180216093140p:plain

Now, you can use your Service Fabric cluster in your VSTS project.

How to setup your CentOS VMs as VSTS Private Agent

VSTS is really powerful tool and you can use compute resources from cloud, but you will sometimes want to use your own custom libraries or executable files in you build processes. You can use Private Agent for such cases. In this post,
I will setup private agent with CentOS VM, but please note VSTS offers some platforms agents like below and CentOS isn't supported right now.

Step by Step to setup Private Agent

Follow below three sections.

  1. Create new pool in VSTS portal
  2. Create new “Personal access tokens” in VSTS portal
  3. Setup agent in your VM

1. Create new pool in VSTS portal

Go to “Agent Pools” tab in your VSTS and click “New pool…”.
f:id:waritohutsu:20180216004316j:plain

Input Agent pool name as you need. I recommend to name for each uses.
f:id:waritohutsu:20180216004328j:plain

Download agent package from your agents pool like below.
f:id:waritohutsu:20180216004337j:plain

2. Create new “Personal access tokens” in VSTS portal

Choose “Security” from your account setting.
f:id:waritohutsu:20180216004343j:plain

Create new “Personal access tokens”.
f:id:waritohutsu:20180216004352j:plain

Keep access token from VSTS portal. The value never show without at the time.
f:id:waritohutsu:20180216004407j:plain

3. Setup agent in your VM

Transfer agent package into your VM and extract data. I executed below commands.

# be root
sudo su -

# I have to install below pakcages into your CentOS VM, because VSTS agent offers RedHat but not CentOS
yum install centos-release-dotnet.noarch
yum install rh-dotnetcore11-dotnetcore.x86_64

# setup agent
mkdir /opt/agent
mv vsts-agent-rhel.7.2-x64-2.123.0.tar.gz /opt/agent
tar zxvf vsts-agent-rhel.7.2-x64-2.123.0.tar.gz
chown -R azureuser /opt/agent/

cd /opt/agent/
./config.sh
./run.sh

Your instance will be registered after “./config.sh” execution. You have to execute “./run.sh” to maintain “Online” status.
f:id:waritohutsu:20180216004433j:plain

Service Fabric deployment tips - always return “Failed to submit updates for certificate" on Azure Portal

As you know, Service Fabric uses some various certificates to manage their clusters.

  • Cluster certificate : Client to node security, e.g. Management Endpoints such as Service Fabric Explorer or PowerShell
  • Server certificate: Server (node) to clients, and server (node) to a server (node).
  • Client certificates : Role-Based Access Control (RBAC) – used to limit access to certain cluster operations for different groups of users, e.g. Admin vs User.

You need to register your own certificate into Azure Portal for browsing your cluster and deploy your applications using Visual Studio, and you can register your certificate with Azure Portal like below.
f:id:waritohutsu:20180210091235j:plain

Almost all cases above step will work well, but there is possibility to get below error “Failed to submit updates for certificate" on Azure Portal.
f:id:waritohutsu:20180210091334j:plain

This issues is caused by wrong version API on Azure Portal. Please use below PowerShell command to avoid the error.

Login-AzureRmAccount
Add-AzureRmServiceFabricClientCertificate -ResourceGroupName "your resource group name" -Name "your service fabric cluster name" -Thumbprint "your thumbprint" -Admin

Service Fabric deployment tips - Deployment on Azure stops when scripts log "Copying application to image store"

When you try to deploy ASP.NET Core Stateless applications into Service Fabric on Azure in like below environment.

The deployment might stop when scripts log "Copying application to image store" like below.

C:\Users\xxxxxxxx\source\repos\FabricApp01\Web1\bin\Debug\netcoreapp2.0\win7-x64\Web1.dllWeb1 -> C:\Users\xxxxxxxx\source\repos\FabricApp01\Web1\obj\Debug\netcoreapp2.0\win7-x64\PubTmp\Out\FabricApp01 -> C:\Users\xxxxxxxx\source\repos\FabricApp01\FabricApp02\pkg\Debug-------- Package: Project: FabricApp01
succeeded, Time elapsed: 00:00:18.7823627 --------3>Started executing script
'Deploy-FabricApplication.ps1'.3>powershell -NonInteractive -NoProfile
-WindowStyle Hidden -ExecutionPolicy Bypass -Command ".
'C:\Users\xxxxxxxx\source\repos\FabricApp01\FabricApp01\Scripts\Deploy-FabricApplication.ps1'
-ApplicationPackagePath
'C:\Users\xxxxxxxx\source\repos\FabricApp01\FabricApp01\pkg\Debug'
-PublishProfileFile
'C:\Users\xxxxxxxx\source\repos\FabricApp01\FabricApp01\PublishProfiles\Cloud.xml'
-DeployOnly:$false -ApplicationParameter:@{} -UnregisterUnusedApplicationVersionsAfterUpgrade
$false -OverrideUpgradeBehavior 'None' -OverwriteBehavior
'SameAppTypeAndVersion' -SkipPackageValidation:$false -ErrorAction Stop" 3>Copying application to image store.

This issue is already known in Copy-ServiceFabricApplicationPackage hangs forever · Issue #813 · Azure/service-fabric-issues · GitHub. You need to remove your "Local Cluster" when you deploy your applications into your cluster on Azure.
f:id:waritohutsu:20180209092112p:plain

After removing Local Cluster, you can deploy your applications into Service Fabric on Azure.

Step by step how to setup Service Fabric Explorer on Azure

This article introduce how to setup the environment on Azure. Service Fabric offers Microservices and containerized architecture on Microsoft Azure, and it's secure with multiple certificates for the clusters.

  • Create Key Vault
  • Create Service Fabric
  • Create Certificate and install into your computer
  • Register your certificate into your Service Fabric
  • Access Service Fabric Explorer

Create Key Vault

You need to setup a "Key vault" instance like below and it's OK to use existing one if you have already your "Key vault" instance, because Service Fabric depends on Key Vault
f:id:waritohutsu:20180209081056p:plain

Create Service Fabric

Create your Service Fabric cluster following below steps. You need to choose "Operating system" adjusting to your applications.
f:id:waritohutsu:20180209081308p:plain

"Node types" is similar with "Cloud Service Roles". VM Scale sets will be created as a number of "Node type count", and specify your "Node type name" and instance type for your "Node Type".
f:id:waritohutsu:20180209081443p:plain

Setup up your cluster security. Please note "Click to hide advanced access policies" at first, and add "Enable access ..." as your Access policies on your Key Vault instance, because it's mandatory to setup. Input your "Certificate name" for your internal cluster communication.
f:id:waritohutsu:20180209081534p:plain

Verify your cluster info and execute to create it.
f:id:waritohutsu:20180209081803p:plain

Create Certificate and install into your computer

You need to create and register new certificate to communicate between your client machine and your cluster. At first, execute below command to create new certificate and you need to input some passwords for the certificates.

# I tried "C:\Program Files (x86)\Windows Kits\10\bin\10.0.16299.0\x64\makecert and PVK2PFX", but its path depends on your computer platform
makecert -sv yourprivatekeyfile.pvk -n "CN=Daichi Isami" yourcertfile.cer -b 02/14/2018 -e 12/31/2020 -r
PVK2PFX -pvk yourprivatekeyfile.pvk -spc yourcertfile.cer -pfx yourpfxfile.pfx -po <your password>

After creating your certificate, just double click it and install certificate into your compute with clicking below button. you need to input password when you create this certificate.
f:id:waritohutsu:20180216070141p:plain

Next, execute "mmc" on PowerShell. Choose "File" -> "Add/Remove Snap-in..." and add "Certificates" as "My user account" like below.
f:id:waritohutsu:20180216070515p:plain

Choose "Certificates - Current User" -> "Personal" in left side and chose your certificate among certificates in central box. Now, you can pick up "Thumbprint" of you certificate like below.
f:id:waritohutsu:20180216070951p:plain

Register your certificate into your Service Fabric

Before proceed this step, you need to confirm your cluster status as "Ready". It might wait more than 20 minute. You might get “failed to submit updates for certificate” error if you proceed this step before that.
Register your certificate into your Service Fabric cluster in Azure Portal. Choose "Security" tab and "Add.." button.
f:id:waritohutsu:20180209082342p:plain

Select "Authentication type" as Admin client to deploy and upgrade your cluster. Input your saved thumbprint into "Certificate thumbprint" and save it.
f:id:waritohutsu:20180209083110p:plain

After this, it takes about 30 minutes to complete this step.
f:id:waritohutsu:20180209083121p:plain

Access Service Fabric Explorer

Access Service Fabric Explorer URL like https://"your cluster name"."your region".cloudapp.azure.com:19080/Explorer/index.html#/, and choose proper certificate installed into your compute.
f:id:waritohutsu:20180216072725p:plain

You can watch Service Fabric Explorer on your cluster
f:id:waritohutsu:20180209083302p:plain

Quick introduction for portal sites of Microsoft Azure

Do you know how many portal sites Microsoft Azure has? Almost all users access only "Microsoft Azure Portal". "Microsoft Azure Portal" manages all Azure resources such like VMs, App Service, SQL Database and others. In addition the portal site, Microsoft Azure also offers two other different portal sites as "Enterprise Azure Portal" and "Azure Account Portal".

  • "Enterprise Azure Portal" manages Azure subscriptions, subscription administrators and billing reports. This portal is mainly used by billing administrator in your company.
  • "Azure Account Portal" manages all Azure resources. This portal is mainly used by developers.
  • "Microsoft Azure Portal" is mainly used for create, transfer, cancel subscriptions. This portal is mainly used by developers.

f:id:waritohutsu:20171231160454p:plain

You don't need to use "Enterprise Azure Portal" if there are no EA contract with Microsoft in your company. Almost all users who use only "Pay-As-You-Go" Azure subscriptions need to use only "Azure Account Portal" and "Microsoft Azure Portal".