How to delete or purge Azure Application Insights data

Abstract

English is not my mother tongue. So often it takes a bit of time for myself to understand few English statements; if it does not contain regular words. Honestly “PURGE” was one of such word where I had to search for dictionary.

//PURGE//

Means: To purify, getting rid of whatever that is impure or undesirable and most important: when you want to start fresh.

How this is related in blog post? Well read on…

What is an APM tool?

One of my customers wanted to do application monitoring during load test. They started configuring monitoring for their Virtual machines hosting applications and database for CPU, Memory, network I/O, disk I/O and so on. Configuration of their monitoring showed them all metrics of VMs like CPU, memory etc. within 70% however application was failing consistently in load test.

Then they called me and explained the same scenario and said application is failing but there is no trace we are getting why it is failing. So I told them, what you are doing currently is good however you are doing only underlying “Infrastructure” monitoring. This is good but it can’t tell always why application is failing. So you need “Application performance monitoring (APM)” tool for application monitoring.

Folks, application monitoring and infrastructure monitoring are two different things. If your VM is not healthy then for sure your application will not work however if your application is not working still your Virtual Machine might be healthy. Got it?

So when you say I am monitoring CPU, memory for application you are actually monitoring only underlying infrastructure capacity/ performance and not the application behavior. To monitor application means you want to understand  -

1. Application failures
2. Exceptions and error raised
3. User telemetry and usage
4. Availability of application
5. Live metrics
6. Response, time, latency and application page load time
7. Application load time in client browser
8. Dependencies, slowest operation performed and so on.

This has nothing to do with your underlying infrastructure performance metrics monitoring.

So you need a tool that can show all above mentioned things in an application and this is where Application performance monitoring APM tools are used.

There are many APM tools in the market like NewRelic, Dynatrace, App Dynamics and “Azure Application Insights”.

So APM is generic concept and Application Insights is a product from Azure for APM.

Customer Scenario

Some customer can be real pain in the …heart [I know what you were thinking].

Customer configured application insights agent and SDK both to monitor various aspects of their few java and .NET based applications for POC. After running the POC for 15 days they were quite satisfied by application insights features. They wanted to start using the same application insights instance in production. However they wanted to start a fresh and hence wanted to delete all the data captured by application insights till now. At the same time they also wanted to retain the application insights service and post data deletion they wanted to start capturing production data in the same service instance of application insights.

Options for deleting application insights data

What is API Gateway? Need of API Gateway in modern software architectures

I work with very large size non-ITes and ITes organizations. When I talk with them about modern software architectures such as Serverless, Micro-services, Event driven; we inevitably talk about mysterious word - “API Gateway”.

Most of the time I have seen zero awareness about “API Gateway” being an important part of new modern software architectures. Many senior TDMs [Technical Decision Makers] completely ignore this important aspect of overall API centric approach for all of their applications as an Organization strategy. There are few senior architects who understand need of “API Gateway” but the number is really really less.

In last 4 months, after 7 customer visits, approx. 10 deep dive modern architecture discussions at various level in organizations, I felt there is a necessity of writing “Simple yet effective” blog post that will focus on need of “API Gateway” in today’s software architectures when we talk about “Digital Transformation journey” with big organizations.

And here I am, writing a post on “What is API Gateway? Why it matters? And how should you choose the right API Gateway for yourself”.

Let’s go!

Realizing the concept of API Gateway

Today every organization is trying to provide service based offerings. For example, Gmail provides “Email as a service”, O365 provide “productivity solutions as a service”.

When you think of providing services based offerings for your customers then inevitably large portion of your offerings will be built based on “APIs”. In today’s world it will be based on “REST API” and in many cases legacy APIs as well build on XML based services. So your APIs will essentially consist of main business logic/ critical intellectual property of your service. So it is really important for you to make sure that you PROTECT these APIs. Therefore API Gateway is an important architecture strategy.

If you are “solution architect” and working on API based solution architecture; API Gateway is a must for you. Let’s understand why we need API gateway Or what benefits we get by using API Gateway.

Understanding the need of an API Gateway?

In today’s world REST APIs usually consist of main business logic/ Intellectual property/ critical exposing layer for your sensitive data. So directly exposing your actual REST API to rest of the world is not a good idea. It has to have a protection layer which monitors every coming requests. Sees if incoming request is a valid, legitimate request and then allow to reach to actual API. This middle man/ protection layer/ wrapper around your actual API is called as “API Gateway”.

An API Gateway is wrapper around your actual REST APIs or any type of APIs for that matter. When you say wrapper; means you don’t expose your actual REST API to the outside world rather you expose it through API Gateway. This has number of benefits –

•          Security for your actual API
•         Manage API lifecycle
•         Routing, protocol transformation
•         API monitoring, analytics
•         Logging the every request hitting your API

Below is the conceptual diagram of API Gateway and where it resides –

As you can see in above diagram, all types of applications who consume your API will pass through the API Gateway Layer. So basically now all the common aspects of security for accessing your actual API; can now be “centralized” at one place and that is your API Gateway layer. So essentially you “Avoid” duplication of same work for each of the APIs common requirements such as security, monitoring, analytics etc.

Benefits of using API Gateway

How to create service principal or App registration in Azure AD

Abstract

Azure AD is the centralized authentication and authorization mechanism for Azure. Any administration operation on Azure environment can be performed only if you are part of Azure AD.

The common questions I get are –

1. How do I authenticate to perform Azure management operations without using actual User credentials?
2. How do I authenticate Azure Resource Manager Request?
3. To call management REST APIs of Azure, how do I generate authentication token using Azure AD?
4. To call management REST API of Azure, how do I generate and pass authentication token from my application?

Answer to all above questions is – Azure AD Service Principal or App registration.

This blog post explains

• -        why you need Azure AD service principal,
• -        how can you create your azure AD service principal,
• -        what can you do with Azure AD service principal.

Why you need Azure AD service principal?

Example, if you want to create a VM in Azure from portal; you first must be part of Azure AD as a user. Then, Azure subscription always belong to Azure AD; so your user id should have enough rights on Azure subscription.

However, to perform such administrative operation you can’t use actual user credentials/ authorization. There are numerous scenarios where you want rights on Azure subscription but not as a user; rather as an application. For example, provisioning infra on Azure using “Infrastructure as Code” approach. Or changing the pricing tier of VM/ or a service on Azure using an application and by not using Azure portal. This is where we need Azure Service Principal AD.

Leap back in history – what is Azure AD service principal?

The service principal is an entity that powers Logic apps to perform an administrative action against azure account. But, what is service principal?

Last year I wrote a detailed blog on making azure automation account powerful enough to perform administrative actions against azure account using service principal. Please read the same to know more about Service principal and how to create the same in Azure using “Azure AD App registration” - 

http://sanganakauthority.blogspot.com/2017/05/run-login-azurermaccount-to-login.html.

I am assuming that you are not lazy and must have gone through what is service principal. The service principal mentioned in that blog is the one that gets created automatically when you create an automation account. In this article how can I create app registration manually and then use the same to generate authentication token to perform wonders in Azure administrative operations.

How to create an Azure service principal?

There are two ways. One using traditional way of app registration on Azure AD; second is using v2.0 endpoint. As of today [18^th Apr 2019] there are limitations on using v2.0 endpoint based app registrations. Refer to below document to decide on whether you need 1.0 or 2.0 endpoint - https://docs.microsoft.com/en-us/azure/active-directory/develop/azure-ad-endpoint-comparison .This may change in future. Therefore we will be using traditional way of app registration and it works best.

Azure AD v1.0 endpoint based app registration

For registering app you may not need to be Azure AD admin. However when it comes to providing the permissions to an app about what it can do; does require admin rights for Azure AD/ Azure subscription owner access.. Therefore it is always best to get it done from your IT team.

Open Azure portal and open Azure AD instance in the portal. You should have similar page as below –

If you observe above screenshot there are total 2 options for app registration. Select the one where “Preview” is not written. Click on “New Application Registration” option. Enter the values as shown below –

Record Tenant ID, application Id and secret key

After successful application registration in Azure AD you will land on the screen as below. Then click on “Settings” -> “keys”

Make sure you copy the application id and keep it safe. We will require it for generating token.

Then provide the information as below and click “Save”. On successful save a key will be generated and visible only until you close the window. Once you close the window the generated key is never displayed again. So keep it safe. Based on the expiration setup; the key will become invalid. For example if you selected 1 year as validity then key will expire after a year from the date of generation.

Tenant id means unique Id of your Azure Active Directory. It is available under “Properties” option under Azure active directory as shown below. Record it and keep it safe.

Assign correct permissions to Azure AD App

How to receive an email on Azure Network Security Group Rule changes

Abstract

Microsoft Azure Portal already gives a capability to receive an email alert when new Azure Network Security Group  (NSG) is added or existing is deleted. However there is no option today to receive an email when individual NSG security rules are added, deleted or modified. This post will provide the solution to receive emails on Azure NSG security rules changes which isn’t offered by Azure Portal.

Why do I need it?

If you are chief security officer of the company, then you definitely understand why do you care to receive an alert when NSG rules are changed.

NSG’s are fundamental to restrict/ allow access in Azure IaaS VM deployments. They offer controlled access using source and destination port, protocol and IP. So as a security best practice any Azure VM (Network Interface Card) NIC or Subnet in VNET should have NSG associated to it.

Having said that, maintaining rules in NSG is critical. Hence many times Azure portal administrators, CISO staff, IT head, Security head will always love to receive an email in Inbox to verify if the NSG security rule added/ modified/ deleted is after appropriate approval or no.

What do I need?

Creating alert is possible from Azure Monitor services. For example, if I want to create alert of NSG creation or deletion then below is the screenshot which shows how exactly you can configure alert.

As you can see in the above screenshot, there is no resource type available for NSG Security Rules. So, you may get under impression that “email alert on NSG security rule change can’t be configured”; which is wrong. The rule of thumb for Microsoft Azure I follow is

“If any functionality in not achievable from the Azure Portal then try it using Azure PowerShell or Azure ARM Templates.”

So, email alert on NSG security rule change can’t be configured from Portal however it is possible to configure using ARM Template.

Also, we need to create an “Action Group” on Azure portal so as to receive the email. So as a summary we will need below artifacts from Azure  -

1.      Azure ARM template to create Alert

2.      Action group to send emails

3.      Resource group which will contain the alert and action group.

So let’s get started.

Create Action Group

Creation an action group to send emails as per the steps mentioned in the link - https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-action-groups

I have created an action group named as AdminsActionsGroup as shown below with Email as Action type –

After successful creation action group, you will receive an email about welcome as shown below –

Copy the resource ID for future use from overview tab as shown below –

Azure ARM Template to create NSG rule add/modify email alert

A sample ARM script to create an Activity Log alert using ARM is available here - https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-create-activity-log-alerts-with-resource-manager-template

Out of the base ARM template present in above link, we need to replace the operationName for NSG rules Write operation as shown below –

Then search “templates” store at the top in Azure portal. Click on “add”, then provide the suitable name and description for the template. Copy the ARM template we created in above step. After adding the template it will be visible as below –

Complete template download is available at the end of this post.

Let’s deploy!

Click on the Deploy button as highlighted in above screenshot. Provide the action group resource id copied in above steps. Then click on “accept terms and condition” and then click on “Purchase” to complete deployment.

You can view the created alert as shown below –

Modify the security rule of any NSG present in Azure subscription and you should receive an email.

Email on Delete NSG Rule Operation

The approach is same. We need to create another alert for delete operation of NSG rules. Only the operation name will change as below –

Hope this helps.

Download complete script - https://gallery.technet.microsoft.com/Receive-an-email-on-Azure-6ebdd9a5

Backup NSG

I have seen many people asking about backing up NSG and rules. One way you can export the rules using PowerShell. 

If you want readymade solution then, to Backup NSG in your Azure Subscription I found one of the Marketplace solution. I have tried this at one of my customer and works good - https://azuremarketplace.microsoft.com/en-us/marketplace/apps/bowspritconsultingopcprivatelimited1596291408582.nsgbackup?tab=Overview