normalian blog

Let's talk about Microsoft Azure, ASP.NET and Java!

Tips to utilize Windows Server containers on AKS

Microsoft has announced that Azure Kubernetes Service (AKS) supports Windows Server containers as GA. This is quite useful and essential feature to containerize your ASP.NET Framework applications. In this article, you can acquire tiny tips to utilize Windows Server containers on AKS.

Enable Azure CNI (advanced) for Windows Server Container

Note that AKS requires to enable " Azure CNI (advanced) network plugin" to utilize Windows Server Containers. Choose "Advanced" as Network configuration like below when you try to create AKS clusters.
f:id:waritohutsu:20200511055511p:plain

You can confirm your AKS clusters are enabled Azure CNI on Azure Portal.
f:id:waritohutsu:20200511055548p:plain

Next, you need to create node pools as Windows OS type to deploy your Windows Server Container applications like below.
f:id:waritohutsu:20200511055835p:plain

Windows Server Container size

Windows Server Container requires huge capacity than Linux images. I have just pushed a simple hello world ASP.NET application into my Azure Container Repository(ACR) but it uses 1.08GB on my ACR. It will take a much time to upload your container images first time, so please note your network bandwidth not only ACR capacities when you push your container images into ACRs.
f:id:waritohutsu:20200511060031p:plain

Manage authorization for your application with user account attributes

Azure AD offers quite useful features to manage accessibilities for your applications. I believe most Azure developers has already utilized user groups to assign privilege easily, but I guess many people don’t know “Dynamic User” user group. This user group enable to authorize users with user account attributes.

Let's setup to manage accessibilities with job title by using Dynamic User group. Here are accounts which are verified by the group.
f:id:waritohutsu:20200509043431p:plain
f:id:waritohutsu:20200509043445p:plain

How to create Dynamic User group

Let’s go to Azure Portal, choose Azure Active Directory, and click to “New Group” at first .
f:id:waritohutsu:20200509043119p:plain

You can choose “Dynamic User” as membership type like below.
f:id:waritohutsu:20200509043133p:plain

Click “Add dynamic query” to setup query to authorize users. This sample authorize users who contain “Principal” for their job title. It’s also possible to create complex queries to meet your business requirements.
f:id:waritohutsu:20200509043143p:plain

Click “Validate Rules (Preview)” like below. You can confirm your queries will works well or not.
f:id:waritohutsu:20200509043152p:plain

Reduce AKS clusters cost by setup zero node count for user mode node pools

Here is interesting article - Release Release 2020-04-13 · Azure/AKS · GitHub. You can find that "AKS now allows User nodepools to scale to 0" in the article. This feature enables to reduce AKS cost in your environments. I believe you would try to change node count by using az command, but it won't work well at this time - 5/1/2020. Please note this setting is possible for only User mode node pools not System mode.

$subcriptionId = "YOUR SUBSCRIPTION ID"
$rg = "YOUR RESOURCE GROUP"
$clustername = "YOUR AKS CLUSTER NAME"
$poolname = "YOUR NODE POOL NAME"
$count = 0
az aks scale --resource-group $rg --name $clustername --node-count $count --nodepool-name $poolname

f:id:waritohutsu:20200502043921p:plain

This issue is caused that az command doesn't support to setup zero node count for user mode node pools at this time. There are two options to achieve this setting here.

Change node count on https://resources.azure.com/

Open https://resources.azure.com/ and find your user mode node pool of your AKS clusters. Put "Edit" button to enable to change Azure resources setting and edit value of "count" as zero.
f:id:waritohutsu:20200502044622p:plain

Please note this setting is possible only User mode node pools. It will fail to change node count into zero for System mode node pools.
f:id:waritohutsu:20200502044934p:plain

Use REST API to change node count

You can REST API by using az command. Here is example to setup zero node count for user mode node pools.

$subcriptionId = "YOUR SUBSCRIPTION ID"
$rg = "YOUR RESOURCE GROUP"
$clustername = "YOUR AKS CLUSTER NAME"
$poolname = "YOUR NODE POOL NAME"
$count = 0

$body = "{  \`"properties\`": {    \`"count\`": ${count} } }"
$header = "{\`"Content-Type\`": \`"application/json\`"}"
az rest -u "https://management.azure.com/subscriptions/${subcriptionId}/resourceGroups/${rg}/providers/Microsoft.ContainerService/managedClusters/${clustername}/agentPools/${poolname}?api-version=2020-03-01" --method put --headers $header --body $body

You can confirm this setting on Azure Portal.
f:id:waritohutsu:20200502045455p:plain

Spot node pool limitation for AKS

Azure Kubernetes Service(AKS) recently offers spot node pool feature as preview - 4/30/2020 right now. This feature enable Azure developers to reduce VM costs by using spot VMs for AKS clusters.
Refer to Preview - Add a spot node pool to an Azure Kubernetes Service (AKS) cluster - Azure Kubernetes Service | Microsoft Docs. Here is a sample command to execute on Azure Portal, but you need to enable this preview feature by following the article before running commands below. You can execute this command successfully.

az aks nodepool add \
    --resource-group YOUR-RESOURCE-GROUP \
    --cluster-name YOUR-AKS-CLUSTER-NAME \
    --name spotnode01 \
    --priority Spot \
    --node-vm-size Standard_DS2_v2  \
    --node-count 1 \
    --eviction-policy Delete \
    --spot-max-price -1 \
    --no-wait

I guess you also want to save cost by using burstable instances named "B series". You can execute a command to specify B series instances but it will be failed .

az aks nodepool add \
    --resource-group YOUR-RESOURCE-GROUP \
    --cluster-name YOUR-AKS-CLUSTER-NAME \
    --name spotnode02 \
    --priority Spot \
    --node-vm-size Standard_B2ms \
    --node-count 1 \
    --eviction-policy Delete \
    --spot-max-price -1 \
    --no-wait

Here is a screenshot for commands above. You can execute commands to add node pools by using spot VMS, but provisions will fail.
f:id:waritohutsu:20200501144554p:plain

This is limitation of spot VMs. Refer to Use Azure Spot VMs - Azure Windows Virtual Machines | Microsoft Docs. B-series and Promo versions of any size (like Dv2, NV, NC, H promo sizes) are not supported at this time.

How Hybrid Runbook Worker work on Azure Automation in practice

I believe many Azure developers have already utilized Azure Automation to automate your management, operation and other tasks to avoid human effort. Azure Automation is fully PaaS feature on Azure, but some cases you might need to integrate its workflow with on-premise or other cloud VMs. You can utilize Hybrid Runbook Worker feature on Azure Automation to integrate Azure Automation built-in environment and other platforms.
docs.microsoft.com

Enable Hybrid Runbook Worker

You can enable both Windows and Linux platform into Hybrid Runbook Worker, but I will talk about only Windows in this post. Please refer to Azure Automation Linux Hybrid Runbook Worker | Microsoft Docs if you need.

At first, prepare your Windows Server 2012 or later machine at first, and follow steps Azure Automation Windows Hybrid Runbook Worker | Microsoft Docs.

I have followed the simplest way to setup Hybrid Runbook Worker. You need to download "New-OnPremiseHybridWorker.ps1" script from PowerShell Gallery | New-OnPremiseHybridWorker 1.6 and execute a command below as administrator on your Windows Server machine. It will take a few minutes to complete.

PS C:\Users\xxxxuser> Install-Script -Name New-OnPremiseHybridWorker	

Next, you execute commands below. This will also take a few minutes.

PS C:\Users\xxxxuser> New-OnPremiseHybridWorker.ps1 -AutomationAccountName <NameofAutomationAccount> -AAResourceGroupName <NameofResourceGroup> -OMSResourceGroupName <NameofOResourceGroup> -HybridGroupName <NameofHRWGroup>  -SubscriptionId <AzureSubscriptionId> -WorkspaceName <NameOfLogAnalyticsWorkspace>
Importing necessary modules...
     Required version 6.13.1 of AzureRM is installed...
Pulling Azure account credentials...
Connecting with the Following Parameters
Accessing Azure Automation Account named demo-automation in region southcentralus...
Referencing existing OMS Workspace named automaiton-demo-workspace in region westus...
Warning: Your Automation account and OMS workspace are in different regions and will not be compatible for future linking.
Downloading and installing the Microsoft Monitoring Agent...
Waiting for agent registration to complete...
Registering the hybrid runbook worker...

WorkspaceName and OMSResourceGroupName are optional parameters for Log Analytics and create them automatically if you don't specify them, but you need specify them if Log Analytics is unavailable in Azure Automation account region. You will get error messages below if you try to enable Hybrid Runbook Worker without putting WorkspaceName and OMSResourceGroupName in Analytics unavailable regions.

PS C:\Users\xxxxuser> New-OnPremiseHybridWorker.ps1 -AutomationAccountName <NameofAutomationAccount>  -OMSResourceGroupName <NameofOResourceGroup> -HybridGroupName <NameofHRWGroup>  -SubscriptionId <AzureSubscriptionId>
Importing necessary modules...
     Successfully installed version 6.13.1 of AzureRM...
Pulling Azure account credentials...
Connecting with the Following Parameters
Accessing Azure Automation Account named demo-automation in region southcentralus...
Creating new OMS Workspace named hybridWorkspace6163 in region westcentralus...
New-AzureRmOperationalInsightsWorkspace : HTTP Status Code: BadRequest
Error Message: New workspaces cannot be created in this region
Request Id: 28545988-a1b4-4b3e-b9bc-a0076b3bd05a
Timestamp (Utc):10/06/2019 19:03:53
At C:\Program Files\WindowsPowerShell\Scripts\New-OnPremiseHybridWorker.ps1:300 char:18
+ ... Workspace = New-AzureRmOperationalInsightsWorkspace -Location $OmsLoc ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [New-AzureRmOperationalInsightsWorkspace], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.OperationalInsights.NewAzureOperationalInsightsWorkspaceCommand

You can find your hybrid work group like below after completion the command.
f:id:waritohutsu:20191007054704p:plain

Run Runbooks on a Hybrid Runbook Worker

Refer to Run runbooks on Azure Automation Hybrid Runbook Worker | Microsoft Docs. As example, I have created new Runbook on Azure Automation like below.

$pwd = pwd
write-output $pwd 

$data = Get-Content -Path "C:\opt\localfile-01.txt" -Encoding UTF8
write-output $data 

Next, I create a new text file at C:\opt\localfile-01.txt on Azure VM enabled Hybrid Runbook Worker like below.
f:id:waritohutsu:20191007055049p:plain

Run this runbook on Azure Automation on Azure Portal. You can choose your hybrid worker as "Run Settings" like below.
f:id:waritohutsu:20191007055224p:plain

As a result, you can confirm the outputs like below.
f:id:waritohutsu:20191007055322p:plain

This means your runbook scripts will be executed at temporary folder and it's possible to utilize on-premise assets.

How to get started with Azure Lighthouse to manage Azure resources across Azure AD tenants

I believe some of Azure users might worry about how to manage Azure resources across Azure AD tenants. It should be an important topic especially Azure CSP users, because CSP subscription management is on the assumption of Azure AD tenants per customer.
Today, you can retrieve Azure resources across Azure AD tenants by using Azure Lighthouse.

Overview to enable Azure Lighthouse

Here are steps to enable Azure Lighthouse

  • Common Azure AD Tenant Owner:
    • Step1 : Create common Azure AD tenant – optional
    • Step2 : Create user group on common Azure AD tenant and add user accounts into the user group
  • Subscription Owner:
    • Step3 : modify ARM Template
    • Step4 : Run PowerShell command to enable Azure Lighthouse
  • Common Azure AD Tenant Owner:
    • Step 5: Retrieve Azure resources across Azure AD tenants
  • Subscription Owner:
    • Step 6 : Delete offer

f:id:waritohutsu:20191002022252p:plain

You," Common Azure AD Tenant Owner", need to create a user group into the "common Azure AD tenant" - most of cases this Azure AD tenant should be owned by System Integrators or others. User accounts in the user group can retrieve Azure resources across Azure AD tenants via common Azure AD tenant.
In next, you, Subscription Owner, need to update an ARM template and execute powershell commands to enable Azure Lighthouse.
Let's get into details each steps.

Step1 : Create common Azure AD tenant – optional

This step is optional if you want to utilize existing Azure AD tenant as common Azure AD tenant. Refer to
Create an Azure Active Directory tenant | Microsoft Docs to create new Azure AD tenant.
You need to pick up "Directory ID" by reference to an image below.
f:id:waritohutsu:20190922091703p:plain

Step2 : Create user group on common Azure AD tenant and add user accounts into the user group

Open your common Azure AD tenant on Azure portal. Choose "Groups" from left side menus and push create "New group" button. Now, you can create new user group on Azure AD tenant by reference below.
f:id:waritohutsu:20190922090826p:plain
You need to pick up "Object ID" of this user group by following below. This ID will be used to enable Azure Lighthouse.
f:id:waritohutsu:20190922091036p:plain

You can complete this step by adding users to relish Azure Lighthouse like below.
f:id:waritohutsu:20190922091423p:plain

Step3 : modify ARM Template

You need to update an ARM Template file with picked up IDs in previous steps - Object ID and Directory ID like below. You can find RBAC Role IDs with Built-in roles for Azure resources | Microsoft Docs. You can choose "Contributor", "Reader" or others depending on requirements of subscription owners.
f:id:waritohutsu:20190922092544p:plain

Save texts below as a JSON file and update parameters by following instructions above.

{
    "$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "mspName": {
            "type": "string",
            "metadata": {
                "description": "Specify the Managed Service Provider name"
            }
        }        
    },
    "variables": {
        "mspRegistrationName": "[guid(parameters('mspName'))]",
        "mspAssignmentName": "[guid(parameters('mspName'))]",
        "mspOfferDescription": "Field Test Offer",
        "managedByTenantId": "<common Azure AD tenant ID>",
        "authorizations": [
                {
                    "principalId": "<user group ID in your common Azure AD tenant ID>",
                    "roleDefinitionId": "<RBAC role ID>",
		    "principalIdDisplayName": "My auth"
                },
            ]
    },
    "resources": [
        {
            "type": "Microsoft.ManagedServices/registrationDefinitions",
            "apiVersion": "2019-06-01",
            "name": "[variables('mspRegistrationName')]",
            "properties": {
                "registrationDefinitionName": "[parameters('mspName')]",
                "description": "[variables('mspOfferDescription')]",
                "managedByTenantId": "[variables('managedByTenantId')]",
                "authorizations": "[variables('authorizations')]"
            }
        },
        {
            "type": "Microsoft.ManagedServices/registrationAssignments",
            "apiVersion": "2019-06-01",
            "name": "[variables('mspAssignmentName')]",
            "dependsOn": [
                "[resourceId('Microsoft.ManagedServices/registrationDefinitions/', variables('mspRegistrationName'))]"
            ],
            "properties": {
                "registrationDefinitionId": "[resourceId('Microsoft.ManagedServices/registrationDefinitions/', variables('mspRegistrationName'))]"
            }
        }
    ],
    "outputs": {
        "mspName": {
            "type": "string",
            "value": "[concat('Managed by', ' ', parameters('mspName'))]"
        },
        "authorizations": {
            "type": "array",
            "value": "[variables('authorizations')]"
        }
    }
}

Step4 : Run PowerShell command to enable Azure Lighthouse

Make sure that you have already installed "Azure PowerShell module". Install "Azure PowerShell module" with reference to Install Azure PowerShell with PowerShellGet | Microsoft Docs if you haven't installed it.

Now, you can enable Azure Lighthouse by execution commands below. Please change "subscription id" and ARM Template json file name depending on your environments.

Connect-AzAccount
Get-AzSubscription 

$subscription = Get-AzSubscription -SubscriptionId "your subscription ID"
Select-AzSubscription -Subscription $subscription

New-AzDeployment -name "Towboat" -mspName "$env:USERNAME TowboatProj" -Location EastUS -TemplateUri C:\Users\myuser\Desktop\azurelighthousesetup.JSON -Verbose

Here is an image when I have tried to run the commands. It takes a few minutes to complete the command and it takes about 10 or 20 minutes to enable Azure Lighthouse on Azure Portal.
f:id:waritohutsu:20190922093533p:plain

Step 5: Retrieve Azure resources across Azure AD tenants

Find "My customers (Azure Lighthouse)" menu on Azure Portal and choose "customers" from left side menu. Now, you can find subscriptions in other Azure AD tenants like below.
f:id:waritohutsu:20190922093950p:plain

As an example to enjoy an feature of Azure Lighthouse, you can enable to retrieve Azure VMs across Azure AD tenants like below.
f:id:waritohutsu:20190922094308p:plain

Step 6: Delete offer

Choose "Service providers (Azure Lighthouse) " on Azure Portal with "Subscription owner" account. Select "Service provider offers" from left side menus. Delete offers by clicking trash box icons.
f:id:waritohutsu:20191002023110p:plain

Whose accounts you can invite into EA Portal as administrator role?

As you know, you can utilize Azure EA Portal to create new Azure subscriptions to charge from your EA contract. Microsoft will send an invitation mail to an account which your company has own. The account will be invited as Enterprise Administrator. Refer to an article below if you need to confirm each role on EA Portal.
normalian.hatenablog.com

Next, you need to invite accounts as new administrator roles. You have to confirm which account types are available to be invited, because there are two account types below.

  • Microsoft Account
    • This account type is also called as "Personal Account"
    • Microsoft manages this account type
    • This account type isn't one of Microsoft Azure resources
    • You could create Microsoft Account by using your corporate mail address such like yyyy@normalian.xyz in past, but you can do that no longer
  • Work or School Account

You need to utilize these accounts on EA Portal. You can find "Auth Level" menu on EA Portal below and choose a proper option to invite new administrators.
f:id:waritohutsu:20190602074000p:plain

Difference of Auth Level on EA Portal

There are four types of "Auth Level" you can choose. Assume your organization has "normalian.xyz" Azure Active Direcotry tenant and here are diagrams which account types you can invite.
f:id:waritohutsu:20190602074345p:plain

f:id:waritohutsu:20190602074450p:plain

f:id:waritohutsu:20190602074608p:plain

f:id:waritohutsu:20190602074640p:plain

Other tips

#1 You will find error messages below if you choose wrong Auth Level. Change proper Auth Level option to invite your accounts.
f:id:waritohutsu:20190602080002p:plain

#2 Wizard to add new administrators will be changed like below.
f:id:waritohutsu:20190602080152p:plain

#3 There is no effect for your existing administrator accounts even if you will change "Auth Level" after inviting the accounts.

#4 It seems to take a little bit time to reflect change of Auth Level, because it sometimes failed to invite new accounts just after changing Auth Level.