What is API Gateway? Need of API Gateway in modern software architectures

I work with very large size non-ITes and ITes organizations. When I talk with them about modern software architectures such as Serverless, Micro-services, Event driven; we inevitably talk about mysterious word - “API Gateway”.

Most of the time I have seen zero awareness about “API Gateway” being an important part of new modern software architectures. Many senior TDMs [Technical Decision Makers] completely ignore this important aspect of overall AI centric approach for all of their applications as an Organization strategy. There are few senior architects who understand need of “API Gateway” but the number is really really less.

In last 4 months, after 7 customer visits, approx. 10 deep dive modern architecture discussions at various level in organizations, I felt there is a necessity of writing “Simple yet effective” blog post that will focus on need of “API Gateway” in today’s software architectures when we talk about “Digital Transformation journey” with big organizations.

And here I am, writing a post on “What is API Gateway? Why it matters? And how should you choose the right API Gateway for yourself”.

Let’s go!

Realizing the concept of API Gateway

Today every organization is trying to provide service based offerings. For example, Gmail provides “Email as a service”, O365 provide “productivity solutions as a service”.

When you think of providing services based offerings for your customers then inevitably large portion of your offerings will be built based on “APIs”. In today’s world it will be based on “REST API” and in many cases legacy APIs as well build on XML based services. So your APIs will essentially consist of main business logic/ critical intellectual property of your service. So it is really important for you to make sure that you PROTECT these APIs. Therefore API Gateway is an important architecture strategy.

If you are “solution architect” and working on API based solution architecture; API Gateway is a must for you. Let’s understand why we need API gateway Or what benefits we get by using API Gateway.

Understanding the need of an API Gateway?

In today’s world REST APIs usually consist of main business logic/ Intellectual property/ critical exposing layer for your sensitive data. So directly exposing your actual REST API to rest of the world is not a good idea. It has to have a protection layer which monitors every coming requests. Sees if incoming request is a valid, legitimate request and then allow to reach to actual API. This middle man/ protection layer/ wrapper around your actual API is called as “API Gateway”.

An API Gateway is wrapper around your actual REST APIs or any type of APIs for that matter. When you say wrapper; means you don’t expose your actual REST API to the outside world rather you expose it through API Gateway. This has number of benefits –

•          Security for your actual API
•         Manage API lifecycle
•         Routing, protocol transformation
•         API monitoring, analytics
•         Logging the every request hitting your API

Below is the conceptual diagram of API Gateway and where it resides –

As you can see in above diagram, all types of applications who consume your AI will pass through the API Gateway Layer. So basically now all the common aspects of security for accessing your actual API; can now be “centralized” at one place and that is your API Gateway layer. So essentially you “Avoid” duplication of same work for each of the APIs common requirements such security, monitoring, analytics etc.

Benefits of using API Gateway

Language agnostic –

Consider a scenario – You have 10 API’s developed in Java and .NET. All of them need to implement modern OAuth 2.0 based authentication scenario for protection of your APIs. So essentially, you will have learn and code about implementation of OAuth 2.0 using your “Identity service (such as Auth0, Azure AD, Active Directory, Apache Directory and so on)” in both .NET and Java. Tomorrow you embark on the journey of Python for writing you Machine Learning APIs then you will have to same thing in Python as well. This is duplication of same work in different languages. Here API Gateway can rescue you.

No matter in which language your API is developed (Java, NodeJS, C#, Python etc.); what you need is a common API description format [example, Swagger] to onboard in API Gateway. Hence API gateway makes onboarding your organization wide APIs independent of the language in which they are written. This is huge benefit. So now you write your OAuth 2.0 implementation only with respect to “API Gateway” layer and you are done. All your APIs are now OAuth 2.0 authentication enables without writing/ changing a Single line of code in actual APIs.

So API Gateway is Language Agnostic.

Compute and hosting platform Agnostic

To import your APIs in API Management it is not necessary to host your APIs on VM/ or physical machine or cloud. Your APIs can be anywhere and can be onboarded into API Gateway. You can have your actual APIs hosted on-premises, any cloud, Serverless, CDN hosted, VM hosted, PaaS anything. Doesn’t matter. The process of onboarding APIs inside API Gateway remains same and easy.

Centralized SSL implementation

Making sure that “Encryption during transit” of your API call is of most importance. This is where you use SSL certificates. When there is HTTPS is involved, payload (data sent in requests like heard and Body both) is encrypted during transit. This can be configured in API Gateway itself. Hence avoiding the same configuration involved on each of the server hosting your APIs.

Centralized Security configuration

API Gateway provides important features such as IP Filter, Validate Tokens configurations. This helps in building important security for all your APIs at one place.

Centralized Caching Framework

Many API Gateways has inbuilt caching mechanism which helps in caching GET method responses. This provides huge performance benefits on exposing APIs without API Gateway layer. In one of the trial of API Gateway I have seen without caching the response was taking around 300ms and after caching it tool 7ms. This is amazing. Also most of API Gateway products these days coming with support for external caching products like Redis Cache. This way is you the size of API Gateway caching is limited you can extend to external cache seamlessely and have more storage for caching the GET method responses.

So this way you avoid caching implementation at each of the individual APIs.

Note – Here some smart people may think why I am talking about only GET method response caching why not other methods such Post, Put, Delete? Well only GET method can be cached. Technically you can cache all methods but not recommended. Caching should be done for GET methods only. This is commonsense in Architecture design principals.

Transformation

REST based APIs development approach is used for new API development however there is a large portion of existing APIs that deal in XML. So many times to make them work in new digital world and various new modern technologies you need to convert request and response from XML to JSON or JSON to XML, Transform XML using XSLT, replace string in body, set query string parameter and so on. All of this kind of transformations can be achieved using API gateway without writing a single line of code.

Versioning of APIs

This is another important benefit of API Gateway. You can expose multiple versions of the same API depending on the need. So this helps in updating the APIs to latest features without downtime because you can multiple versions of the same API exposed at the same time using API Gateway.

Policies

This is by far one of the most important benefit you get by using API Gateway. Policies bring life of your APIs in balance. There are tons of policies API Gateway provides that makes API lifecycle management extremely easy and manageable. There are various categories of policies provided such as  -

•         Access restriction
•         Authentication
•         Cross domain
•         Caching
•         Transformation
•         Trace
•         Control Flow
•         Error handling and many more…

Supercool: Making money out of your APIs

I have seen lot of startup companies in recent time who only creates APIs. Then they sell these APIs, charge them per call to various companies and make money out of it. If you have such a vision in your mid then you can use API Gateway various features such as Grouping of APIs in logical structure and make it open to your customers. For example you can create FREE group which allows calling of your APIs only 10 calls/ min and max 1000 in a day. Then you can create BASIC pricing plan and allow 10 calls/ sec and charge them. Similarly you can create various plan such standard, premium and so on. Example Sendgrid, Twilio. These companies offer email, SMS APIs and they charge their customers based on such type of pricing plans. Building such type of money making model becomes super easy using “API Gateway layer”.

So by now I hope you are convinced why it makes sense to include API Gateway layer in your modern application architectures. As API gateway helps you to manage the lifecycle of APIs; it is also called as “API Management solution”.

So, what is Azure API Management?

In essence API Gateway is a Concept. Azure API Management is an implementation/ product of API Gateway concept. Similarly, other companies also provide API Gateway concept-based product; like Apigee, AWS API Gateway and so on.

If you are seriously looking into Azure cloud; API Management is a hero service that you can’t ignore. You should use API Management if you are betting on Azure platform. It is one of the versatile service in Azure and a complete solution to your “API Gateway” needs.

Conclusion

Hope this post gave you an idea why we should include API Gateway layer. This is really needed and important component to include in your organization wide applications architectures.

Well, API Gateway is going to be mission critical for you over a time period once you start on it and hence it is really important to configure DR (Disaster recovery) for the same. In next few posts I will show how can we achieve DR for Azure API Management solution.

Happy APIsation!!

How to create service principal or App registration in Azure AD

Abstract

Azure AD is the centralized authentication and authorization mechanism for Azure. Any administration operation on Azure environment can be performed only if you are part of Azure AD.

The common questions I get are –

1. How do I authenticate to perform Azure management operations without using actual User credentials?
2. How do I authenticate Azure Resource Manager Request?
3. To call management REST APIs of Azure, how do I generate authentication token using Azure AD?
4. To call management REST API of Azure, how do I generate and pass authentication token from my application?

Answer to all above questions is – Azure AD Service Principal or App registration.

This blog post explains

• -        why you need Azure AD service principal,
• -        how can you create your azure AD service principal,
• -        what can you do with Azure AD service principal.

Why you need Azure AD service principal?

Example, if you want to create a VM in Azure from portal; you first must be part of Azure AD as a user. Then, Azure subscription always belong to Azure AD; so your user id should have enough rights on Azure subscription.

However, to perform such administrative operation you can’t use actual user credentials/ authorization. There are numerous scenarios where you want rights on Azure subscription but not as a user; rather as an application. For example, provisioning infra on Azure using “Infrastructure as Code” approach. Or changing the pricing tier of VM/ or a service on Azure using an application and by not using Azure portal. This is where we need Azure Service Principal AD.

Leap back in history – what is Azure AD service principal?

The service principal is an entity that powers Logic apps to perform an administrative action against azure account. But, what is service principal?

Last year I wrote a detailed blog on making azure automation account powerful enough to perform administrative actions against azure account using service principal. Please read the same to know more about Service principal and how to create the same in Azure using “Azure AD App registration” - 

http://sanganakauthority.blogspot.com/2017/05/run-login-azurermaccount-to-login.html.

I am assuming that you are not lazy and must have gone through what is service principal. The service principal mentioned in that blog is the one that gets created automatically when you create an automation account. In this article how can I create app registration manually and then use the same to generate authentication token to perform wonders in Azure administrative operations.

How to create an Azure service principal?

There are two ways. One using traditional way of app registration on Azure AD; second is using v2.0 endpoint. As of today [18^th Apr 2019] there are limitations on using v2.0 endpoint based app registrations. Refer to below document to decide on whether you need 1.0 or 2.0 endpoint - https://docs.microsoft.com/en-us/azure/active-directory/develop/azure-ad-endpoint-comparison .This may change in future. Therefore we will be using traditional way of app registration and it works best.

Azure AD v1.0 endpoint based app registration

For registering app you may not need to be Azure AD admin. However when it comes to providing the permissions to an app about what it can do; does require admin rights for Azure AD/ Azure subscription owner access.. Therefore it is always best to get it done from your IT team.

Open Azure portal and open Azure AD instance in the portal. You should have similar page as below –

If you observe above screenshot there are total 2 options for app registration. Select the one where “Preview” is not written. Click on “New Application Registration” option. Enter the values as shown below –

Record Tenant ID, application Id and secret key

After successful application registration in Azure AD you will land on the screen as below. Then click on “Settings” -> “keys”

Make sure you copy the application id and keep it safe. We will require it for generating token.

Then provide the information as below and click “Save”. On successful save a key will be generated and visible only until you close the window. Once you close the window the generated key is never displayed again. So keep it safe. Based on the expiration setup; the key will become invalid. For example if you selected 1 year as validity then key will expire after a year from the date of generation.

Tenant id means unique Id of your Azure Active Directory. It is available under “Properties” option under Azure active directory as shown below. Record it and keep it safe.

Assign correct permissions to Azure AD App

This is an important step. This grants permissions to Azure Ad app to perform administrative operations against your Azure resources. Open your Azure AD app -> Settings -> Required Permissions -> Add as shown below –

Then click on Select an API -> Windows Azure service management API -> Select. And then select delegated permissions as shown below –

This gives all the necessary permissions for our Azure Ad app. Now by using this app we can generate an Azure AD token. Then we can use the token to invoke any Azure REST api to perform an operation. For example, create backup and recovery of Azure API Management using its REST API and by passing this AD token. Similarly there can be numerous such use case which can be accomplished now.

Code base for generating token

Clone/ download the repository of github - https://github.com/kunalchandratre1/AzureADTokenGenerator. Replace the values of Application ID, Secret and Tenant ID and run the application. This should generate the Azure AD token as shown below –

Once you have this project ready calling the REST APIs of Azure becomes a piece of cake.

Conclusion

I hope this post have helped you to save you from very repetitive tasks of token generation for Azure authentication. Please provide your valuable comments. Good news is its free!!

Keep Authenticating!!

How to receive an email on Azure Network Security Group Rule changes

Abstract

Microsoft Azure Portal already gives a capability to receive an email alert when new Azure Network Security Group  (NSG) is added or existing is deleted. However there is no option today to receive an email when individual NSG security rules are added, deleted or modified. This post will provide the solution to receive emails on Azure NSG security rules changes which isn’t offered by Azure Portal.

Why do I need it?

If you are chief security officer of the company, then you definitely understand why do you care to receive an alert when NSG rules are changed.

NSG’s are fundamental to restrict/ allow access in Azure IaaS VM deployments. They offer controlled access using source and destination port, protocol and IP. So as a security best practice any Azure VM (Network Interface Card) NIC or Subnet in VNET should have NSG associated to it.

Having said that, maintaining rules in NSG is critical. Hence many times Azure portal administrators, CISO staff, IT head, Security head will always love to receive an email in Inbox to verify if the NSG security rule added/ modified/ deleted is after appropriate approval or no.

What do I need?

Creating alert is possible from Azure Monitor services. For example, if I want to create alert of NSG creation or deletion then below is the screenshot which shows how exactly you can configure alert.

As you can see in the above screenshot, there is no resource type available for NSG Security Rules. So, you may get under impression that “email alert on NSG security rule change can’t be configured”; which is wrong. The rule of thumb for Microsoft Azure I follow is

“If any functionality in not achievable from the Azure Portal then try it using Azure PowerShell or Azure ARM Templates.”

So, email alert on NSG security rule change can’t be configured from Portal however it is possible to configure using ARM Template.

Also, we need to create an “Action Group” on Azure portal so as to receive the email. So as a summary we will need below artifacts from Azure  -

1.      Azure ARM template to create Alert

2.      Action group to send emails

3.      Resource group which will contain the alert and action group.

So let’s get started.

Create Action Group

Creation an action group to send emails as per the steps mentioned in the link - https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-action-groups

I have created an action group named as AdminsActionsGroup as shown below with Email as Action type –

After successful creation action group, you will receive an email about welcome as shown below –

Copy the resource ID for future use from overview tab as shown below –

Azure ARM Template to create NSG rule add/modify email alert

A sample ARM script to create an Activity Log alert using ARM is available here - https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-create-activity-log-alerts-with-resource-manager-template

Out of the base ARM template present in above link, we need to replace the operationName for NSG rules Write operation as shown below –

Then search “templates” store at the top in Azure portal. Click on “add”, then provide the suitable name and description for the template. Copy the ARM template we created in above step. After adding the template it will be visible as below –

Complete template download is available at the end of this post.

Let’s deploy!

Click on the Deploy button as highlighted in above screenshot. Provide the action group resource id copied in above steps. Then click on “accept terms and condition” and then click on “Purchase” to complete deployment.

You can view the created alert as shown below –

Modify the security rule of any NSG present in Azure subscription and you should receive an email.

Email on Delete NSG Rule Operation

The approach is same. We need to create another alert for delete operation of NSG rules. Only the operation name will change as below –

Hope this helps.

Download complete script - https://gallery.technet.microsoft.com/Receive-an-email-on-Azure-6ebdd9a5

Start Stop Multiple Azure VMs on schedule using Azure Automation

Abstract

I searched through many, many, many…articles written about Azure VM stop(de-allocate) to save cost. Every article only targets single Azure VM to shutdown (de-allocate). None of the articles target start and Shutdown of multiple Azure VMs using automation. I am going to address this using Azure powershell in this article.

What all Cool features of azure VM start/ stop my script offers?

If you have multiple Azure VMs present in your azure subscription; and you want to start or stop few of them on schedule then you can use below azure PowerShell scripts. It will have below features –

è  Add/ remove the VM names from the list of start/ stop schedule easily. [You don’t need to have PowerShell knowledge]

è  Use same script in multiple different Azure runbooks, with different Azure VM names to start and stop; that too with different schedules.

è  In case any wrong/ nonexistent VM name added to the list, script takes care of giving relative user-friendly message and do not throw error. It continues with other remaining set of Azure VMs and perform start or stop.

è  Can be used with single VM as well if required.

è  You start and stop azure VMs based on schedule hence obviously you optimize or save Azure cost.

è  Script is available for download; hence you can own it to change in future. [that’s cool 😊]

I can say all above feature are nothing but the “Key differentiator” or “Value Proposition” from other Azure VM shut down/ start automation blogs/ scripts. [These marketing terms; I hear them a lot 😊]

Fundamentals of Azure VM cost optimization

Azure VM are made of Compute and Storage. Compute is nothing but Core (number of cores/ CPUs) and RAM you get when you provision Azure VM. Storage is nothing but the C drive or other letter drives; except D drive for windows and Sdb drive for Linux]. These drives are nothing but disks or .vhd files present in Azure storage account [if VM is unmanaged] else disks or .vhd files will be part of your subscription [if VM is created using managed disks].

When we say you stop Azure VM from portal or using PowerShell command “Stop-AzureRMVm” basically Compute part of VM is released (or de-allocated). It is worth to mention here that “STORAGE PART REMAINS”.

So, when you think that if VM is in de-allocated state then I am not charged, YOU ARE WRONG. You are not charged for compute/Cores part; for storage capacity you are charged. However that is too cheap, so don’t worry. Cores are costly and that is where you save cost.

Also if you shutdown Azure VM after taking RDP/ SSH, compute is not released so you are charged for compute and storage both in that case. Therefore, it is recommended that you stop azure vm either using portal or PowerShell command. Refer the FAQ section from link for more details.

Let’s go back to script part now.

Stop multiple Azure VMs script

#define the VM names which are required to be shutdown

$vmNamesToBeShutDown = "vm1", "vm2", "vm3"

foreach($vmName in $vmNamesToBeShutDown)

{

    $resource = Find-AzureRmResource -ResourceNameEquals $vmName -ResourceType "Microsoft.Compute/virtualMachines"

    if($resource -ne $null)

    {  

        Write-Output "Stopping virtual machine..." + $vmName

        Stop-AzureRmVM -ResourceGroupName $resource.ResourceGroupName -Name $vmName -Force

    }   

    else

    {

        Write-output "Virtual machine not found:" + $vmName

    }

}

The variable $vmNamesToBeShutDown is very important. This is where you can add as many VMs as you want to de-allocate them. The only requirement is to add VM name in double quotes and comma separated.

The code then traverse through each of the VM names, checks its existence and if found then fires command to de-allocate the azure vm.

Start multiple Azure VMs script

#define the VM names which are required to be shutdown

$vmNamesToBeStarted = "vm1","vm2", "vm3", "vm4"

foreach($vmName in $vmNamesToBeStarted)

{

    $resource = Find-AzureRmResource -ResourceNameEquals $vmName -ResourceType "Microsoft.Compute/virtualMachines"

    if($resource -ne $null)

    {  

        Write-Output "Starting virtual machine..." + $vmName

        Start-AzureRmVM -ResourceGroupName $resource.ResourceGroupName -Name $vmName

    }   

    else

    {

        Write-output "Virtual machine not found:" + $vmName

    }

}

The structure of the start Azure VM script is same as that of Stop except the command used here to start VM is Start-AzureRmVM.

Here also you will only add VM names comma separated and they will get automatically started once you schedule script running.

Automate and schedule the start/stop Azure VM scripts

Azure automation allows you to create PowerShell script based runbook which can run automatically on pre-defined schedule.

The guidance here is copy paste above scripts in a runbook in PowerShell. When you plan to run the PowerShell as runbook make sure that you are using service principal authentication code at the start of runbook as depicted in this link – http://sanganakauthority.blogspot.in/2017/05/run-login-azurermaccount-to-login.html. The authentication code should be added before any other PowerShell code in runbook. After creating runbook, publish it. Without publish runbook will not execute.

Then create schedule in automation account as per desired schedule. Screenshot below –

Once schedule is created, attach to runbook as below –

Similarly you can create as many as runbook and schedule them at your will and it should work.

Hope this helps.

Happy automating!!

You may be interested in – 

Domain join Azure virtual machine automatically using Azure Automation and DSC - http://sanganakauthority.blogspot.in/2017/01/domain-join-azure-vm-using-azure.html