What is API Gateway? Need of API Gateway in modern software architectures

I work with very large size non-ITes and ITes organizations. When I talk with them about modern software architectures such as Serverless, Micro-services, Event driven; we inevitably talk about mysterious word - “API Gateway”.

Most of the time I have seen zero awareness about “API Gateway” being an important part of new modern software architectures. Many senior TDMs [Technical Decision Makers] completely ignore this important aspect of overall API centric approach for all of their applications as an Organization strategy. There are few senior architects who understand need of “API Gateway” but the number is really really less.

In last 4 months, after 7 customer visits, approx. 10 deep dive modern architecture discussions at various level in organizations, I felt there is a necessity of writing “Simple yet effective” blog post that will focus on need of “API Gateway” in today’s software architectures when we talk about “Digital Transformation journey” with big organizations.

And here I am, writing a post on “What is API Gateway? Why it matters? And how should you choose the right API Gateway for yourself”.

Let’s go!

Realizing the concept of API Gateway

Today every organization is trying to provide service based offerings. For example, Gmail provides “Email as a service”, O365 provide “productivity solutions as a service”.

When you think of providing services based offerings for your customers then inevitably large portion of your offerings will be built based on “APIs”. In today’s world it will be based on “REST API” and in many cases legacy APIs as well build on XML based services. So your APIs will essentially consist of main business logic/ critical intellectual property of your service. So it is really important for you to make sure that you PROTECT these APIs. Therefore API Gateway is an important architecture strategy.

If you are “solution architect” and working on API based solution architecture; API Gateway is a must for you. Let’s understand why we need API gateway Or what benefits we get by using API Gateway.

Understanding the need of an API Gateway?

In today’s world REST APIs usually consist of main business logic/ Intellectual property/ critical exposing layer for your sensitive data. So directly exposing your actual REST API to rest of the world is not a good idea. It has to have a protection layer which monitors every coming requests. Sees if incoming request is a valid, legitimate request and then allow to reach to actual API. This middle man/ protection layer/ wrapper around your actual API is called as “API Gateway”.

An API Gateway is wrapper around your actual REST APIs or any type of APIs for that matter. When you say wrapper; means you don’t expose your actual REST API to the outside world rather you expose it through API Gateway. This has number of benefits –

•          Security for your actual API
•         Manage API lifecycle
•         Routing, protocol transformation
•         API monitoring, analytics
•         Logging the every request hitting your API

Below is the conceptual diagram of API Gateway and where it resides –

As you can see in above diagram, all types of applications who consume your API will pass through the API Gateway Layer. So basically now all the common aspects of security for accessing your actual API; can now be “centralized” at one place and that is your API Gateway layer. So essentially you “Avoid” duplication of same work for each of the APIs common requirements such as security, monitoring, analytics etc.

Benefits of using API Gateway

How to create service principal or App registration in Azure AD

Abstract

Azure AD is the centralized authentication and authorization mechanism for Azure. Any administration operation on Azure environment can be performed only if you are part of Azure AD.

The common questions I get are –

1. How do I authenticate to perform Azure management operations without using actual User credentials?
2. How do I authenticate Azure Resource Manager Request?
3. To call management REST APIs of Azure, how do I generate authentication token using Azure AD?
4. To call management REST API of Azure, how do I generate and pass authentication token from my application?

Answer to all above questions is – Azure AD Service Principal or App registration.

This blog post explains

• -        why you need Azure AD service principal,
• -        how can you create your azure AD service principal,
• -        what can you do with Azure AD service principal.

Why you need Azure AD service principal?

Example, if you want to create a VM in Azure from portal; you first must be part of Azure AD as a user. Then, Azure subscription always belong to Azure AD; so your user id should have enough rights on Azure subscription.

However, to perform such administrative operation you can’t use actual user credentials/ authorization. There are numerous scenarios where you want rights on Azure subscription but not as a user; rather as an application. For example, provisioning infra on Azure using “Infrastructure as Code” approach. Or changing the pricing tier of VM/ or a service on Azure using an application and by not using Azure portal. This is where we need Azure Service Principal AD.

Leap back in history – what is Azure AD service principal?

The service principal is an entity that powers Logic apps to perform an administrative action against azure account. But, what is service principal?

Last year I wrote a detailed blog on making azure automation account powerful enough to perform administrative actions against azure account using service principal. Please read the same to know more about Service principal and how to create the same in Azure using “Azure AD App registration” - 

http://sanganakauthority.blogspot.com/2017/05/run-login-azurermaccount-to-login.html.

I am assuming that you are not lazy and must have gone through what is service principal. The service principal mentioned in that blog is the one that gets created automatically when you create an automation account. In this article how can I create app registration manually and then use the same to generate authentication token to perform wonders in Azure administrative operations.

How to create an Azure service principal?

There are two ways. One using traditional way of app registration on Azure AD; second is using v2.0 endpoint. As of today [18^th Apr 2019] there are limitations on using v2.0 endpoint based app registrations. Refer to below document to decide on whether you need 1.0 or 2.0 endpoint - https://docs.microsoft.com/en-us/azure/active-directory/develop/azure-ad-endpoint-comparison .This may change in future. Therefore we will be using traditional way of app registration and it works best.

Azure AD v1.0 endpoint based app registration

For registering app you may not need to be Azure AD admin. However when it comes to providing the permissions to an app about what it can do; does require admin rights for Azure AD/ Azure subscription owner access.. Therefore it is always best to get it done from your IT team.

Open Azure portal and open Azure AD instance in the portal. You should have similar page as below –

If you observe above screenshot there are total 2 options for app registration. Select the one where “Preview” is not written. Click on “New Application Registration” option. Enter the values as shown below –

Record Tenant ID, application Id and secret key

After successful application registration in Azure AD you will land on the screen as below. Then click on “Settings” -> “keys”

Make sure you copy the application id and keep it safe. We will require it for generating token.

Then provide the information as below and click “Save”. On successful save a key will be generated and visible only until you close the window. Once you close the window the generated key is never displayed again. So keep it safe. Based on the expiration setup; the key will become invalid. For example if you selected 1 year as validity then key will expire after a year from the date of generation.

Tenant id means unique Id of your Azure Active Directory. It is available under “Properties” option under Azure active directory as shown below. Record it and keep it safe.

Assign correct permissions to Azure AD App

How to receive an email on Azure Network Security Group Rule changes

Abstract

Microsoft Azure Portal already gives a capability to receive an email alert when new Azure Network Security Group  (NSG) is added or existing is deleted. However there is no option today to receive an email when individual NSG security rules are added, deleted or modified. This post will provide the solution to receive emails on Azure NSG security rules changes which isn’t offered by Azure Portal.

Why do I need it?

If you are chief security officer of the company, then you definitely understand why do you care to receive an alert when NSG rules are changed.

NSG’s are fundamental to restrict/ allow access in Azure IaaS VM deployments. They offer controlled access using source and destination port, protocol and IP. So as a security best practice any Azure VM (Network Interface Card) NIC or Subnet in VNET should have NSG associated to it.

Having said that, maintaining rules in NSG is critical. Hence many times Azure portal administrators, CISO staff, IT head, Security head will always love to receive an email in Inbox to verify if the NSG security rule added/ modified/ deleted is after appropriate approval or no.

What do I need?

Creating alert is possible from Azure Monitor services. For example, if I want to create alert of NSG creation or deletion then below is the screenshot which shows how exactly you can configure alert.

As you can see in the above screenshot, there is no resource type available for NSG Security Rules. So, you may get under impression that “email alert on NSG security rule change can’t be configured”; which is wrong. The rule of thumb for Microsoft Azure I follow is

“If any functionality in not achievable from the Azure Portal then try it using Azure PowerShell or Azure ARM Templates.”

So, email alert on NSG security rule change can’t be configured from Portal however it is possible to configure using ARM Template.

Also, we need to create an “Action Group” on Azure portal so as to receive the email. So as a summary we will need below artifacts from Azure  -

1.      Azure ARM template to create Alert

2.      Action group to send emails

3.      Resource group which will contain the alert and action group.

So let’s get started.

Create Action Group

Creation an action group to send emails as per the steps mentioned in the link - https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-action-groups

I have created an action group named as AdminsActionsGroup as shown below with Email as Action type –

After successful creation action group, you will receive an email about welcome as shown below –

Copy the resource ID for future use from overview tab as shown below –

Azure ARM Template to create NSG rule add/modify email alert

A sample ARM script to create an Activity Log alert using ARM is available here - https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-create-activity-log-alerts-with-resource-manager-template

Out of the base ARM template present in above link, we need to replace the operationName for NSG rules Write operation as shown below –

Then search “templates” store at the top in Azure portal. Click on “add”, then provide the suitable name and description for the template. Copy the ARM template we created in above step. After adding the template it will be visible as below –

Complete template download is available at the end of this post.

Let’s deploy!

Click on the Deploy button as highlighted in above screenshot. Provide the action group resource id copied in above steps. Then click on “accept terms and condition” and then click on “Purchase” to complete deployment.

You can view the created alert as shown below –

Modify the security rule of any NSG present in Azure subscription and you should receive an email.

Email on Delete NSG Rule Operation

The approach is same. We need to create another alert for delete operation of NSG rules. Only the operation name will change as below –

Hope this helps.

Download complete script - https://gallery.technet.microsoft.com/Receive-an-email-on-Azure-6ebdd9a5

Backup NSG

I have seen many people asking about backing up NSG and rules. One way you can export the rules using PowerShell. 

If you want readymade solution then, to Backup NSG in your Azure Subscription I found one of the Marketplace solution. I have tried this at one of my customer and works good - https://azuremarketplace.microsoft.com/en-us/marketplace/apps/bowspritconsultingopcprivatelimited1596291408582.nsgbackup?tab=Overview

Start Stop Multiple Azure VMs on schedule using Azure Automation

Abstract

I searched through many, many, many…articles written about Azure VM stop(de-allocate) to save cost. Every article only targets single Azure VM to shutdown (de-allocate). None of the articles target start and Shutdown of multiple Azure VMs using automation. I am going to address this using Azure powershell in this article.

What all Cool features of azure VM start/ stop my script offers?

If you have multiple Azure VMs present in your azure subscription; and you want to start or stop few of them on schedule then you can use below azure PowerShell scripts. It will have below features –

è  Add/ remove the VM names from the list of start/ stop schedule easily. [You don’t need to have PowerShell knowledge]

è  Use same script in multiple different Azure runbooks, with different Azure VM names to start and stop; that too with different schedules.

è  In case any wrong/ nonexistent VM name added to the list, script takes care of giving relative user-friendly message and do not throw error. It continues with other remaining set of Azure VMs and perform start or stop.

è  Can be used with single VM as well if required.

è  You start and stop azure VMs based on schedule hence obviously you optimize or save Azure cost.

è  Script is available for download; hence you can own it to change in future. [that’s cool 😊]

I can say all above feature are nothing but the “Key differentiator” or “Value Proposition” from other Azure VM shut down/ start automation blogs/ scripts. [These marketing terms; I hear them a lot 😊]

Fundamentals of Azure VM cost optimization

Azure VM are made of Compute and Storage. Compute is nothing but Core (number of cores/ CPUs) and RAM you get when you provision Azure VM. Storage is nothing but the C drive or other letter drives; except D drive for windows and Sdb drive for Linux]. These drives are nothing but disks or .vhd files present in Azure storage account [if VM is unmanaged] else disks or .vhd files will be part of your subscription [if VM is created using managed disks].

When we say you stop Azure VM from portal or using PowerShell command “Stop-AzureRMVm” basically Compute part of VM is released (or de-allocated). It is worth to mention here that “STORAGE PART REMAINS”.

So, when you think that if VM is in de-allocated state then I am not charged, YOU ARE WRONG. You are not charged for compute/Cores part; for storage capacity you are charged. However that is too cheap, so don’t worry. Cores are costly and that is where you save cost.

Also if you shutdown Azure VM after taking RDP/ SSH, compute is not released so you are charged for compute and storage both in that case. Therefore, it is recommended that you stop azure vm either using portal or PowerShell command. Refer the FAQ section from link for more details.

Let’s go back to script part now.

Stop multiple Azure VMs script

#define the VM names which are required to be shutdown

$vmNamesToBeShutDown = "vm1", "vm2", "vm3"

foreach($vmName in $vmNamesToBeShutDown)

{

    $resource = Find-AzureRmResource -ResourceNameEquals $vmName -ResourceType "Microsoft.Compute/virtualMachines"

    if($resource -ne $null)

    {  

        Write-Output "Stopping virtual machine..." + $vmName

        Stop-AzureRmVM -ResourceGroupName $resource.ResourceGroupName -Name $vmName -Force

    }   

    else

    {

        Write-output "Virtual machine not found:" + $vmName

    }

}

The variable $vmNamesToBeShutDown is very important. This is where you can add as many VMs as you want to de-allocate them. The only requirement is to add VM name in double quotes and comma separated.

The code then traverse through each of the VM names, checks its existence and if found then fires command to de-allocate the azure vm.

Start multiple Azure VMs script

#define the VM names which are required to be shutdown

$vmNamesToBeStarted = "vm1","vm2", "vm3", "vm4"

foreach($vmName in $vmNamesToBeStarted)

{

    $resource = Find-AzureRmResource -ResourceNameEquals $vmName -ResourceType "Microsoft.Compute/virtualMachines"

    if($resource -ne $null)

    {  

        Write-Output "Starting virtual machine..." + $vmName

        Start-AzureRmVM -ResourceGroupName $resource.ResourceGroupName -Name $vmName

    }   

    else

    {

        Write-output "Virtual machine not found:" + $vmName

    }

}

The structure of the start Azure VM script is same as that of Stop except the command used here to start VM is Start-AzureRmVM.

Here also you will only add VM names comma separated and they will get automatically started once you schedule script running.

Automate and schedule the start/stop Azure VM scripts

Azure automation allows you to create PowerShell script based runbook which can run automatically on pre-defined schedule.

The guidance here is copy paste above scripts in a runbook in PowerShell. When you plan to run the PowerShell as runbook make sure that you are using service principal authentication code at the start of runbook as depicted in this link – http://sanganakauthority.blogspot.in/2017/05/run-login-azurermaccount-to-login.html. The authentication code should be added before any other PowerShell code in runbook. After creating runbook, publish it. Without publish runbook will not execute.

Then create schedule in automation account as per desired schedule. Screenshot below –

Once schedule is created, attach to runbook as below –

Similarly you can create as many as runbook and schedule them at your will and it should work.

Hope this helps.

Happy automating!!

You may be interested in – 

Domain join Azure virtual machine automatically using Azure Automation and DSC - http://sanganakauthority.blogspot.in/2017/01/domain-join-azure-vm-using-azure.html