Thursday, June 6, 2019

How to delete or purge Azure Application Insights data


Abstract


English is not my mother tongue. So often it takes a bit of time for myself to understand few English statements; if it does not contain regular words. Honestly “PURGE” was one of such word where I had to search for dictionary.

//PURGE//

Means: To purify, getting rid of whatever that is impure or undesirable and most important: when you want to start fresh.

How this is related in blog post? Well read on…

What is an APM tool?


One of my customers wanted to do application monitoring during load test. They started configuring monitoring for their Virtual machines hosting applications and database for CPU, Memory, network I/O, disk I/O and so on. Configuration of their monitoring showed them all metrics of VMs like CPU, memory etc. within 70% however application was failing consistently in load test.

Then they called me and explained the same scenario and said application is failing but there is no trace we are getting why it is failing. So I told them, what you are doing currently is good however you are doing only underlying “Infrastructure” monitoring. This is good but it can’t tell always why application is failing. So you need “Application performance monitoring (APM)” tool for application monitoring.

Folks, application monitoring and infrastructure monitoring are two different things. If your VM is not healthy then for sure your application will not work however if your application is not working still your Virtual Machine might be healthy. Got it?

So when you say I am monitoring CPU, memory for application you are actually monitoring only underlying infrastructure capacity/ performance and not the application behavior. To monitor application means you want to understand  -

  1. Application failures
  2. Exceptions and error raised
  3. User telemetry and usage
  4. Availability of application
  5. Live metrics
  6. Response, time, latency and application page load time
  7. Application load time in client browser
  8. Dependencies, slowest operation performed and so on.

This has nothing to do with your underlying infrastructure performance metrics monitoring.
So you need a tool that can show all above mentioned things in an application and this is where Application performance monitoring APM tools are used.

There are many APM tools in the market like NewRelic, Dynatrace, App Dynamics and “Azure Application Insights”.

So APM is generic concept and Application Insights is a product from Azure for APM.

Customer Scenario


Some customer can be real pain in the …heart [I know what you were thinking].

Customer configured application insights agent and SDK both to monitor various aspects of their few java and .NET based applications for POC. After running the POC for 15 days they were quite satisfied by application insights features. They wanted to start using the same application insights instance in production. However they wanted to start a fresh and hence wanted to delete all the data captured by application insights till now. At the same time they also wanted to retain the application insights service and post data deletion they wanted to start capturing production data in the same service instance of application insights.

Options for deleting application insights data



There 3 options to delete the application insights data –

Option 1 – Delete application insights


I know this is one of the crude ways of doing it. When you delete application insights everything contained inside of it will be deleted. This option would not fit into our scenario described above. When you create a new application insights unfortunately you will have to redo the changes of application insights key and setting again in the application wherever it is referenced. My customer did not want to redo the settings. Customer is a lazy…heart [I know what you were thinking].

Option 2 – Automatic deletion of data


This is more of configuring the data retention policy in such a way that it deletes all the data over a time period. Default duration of data retention is 90 days. So post 90 days all the data stored in application insights is automatically removed. Also Application insights is charged as per the amount of storage in GBs you are consuming. So it makes sense to store captured data only for limited period and then delete it automatically. If you want to store data longer than 90 days then one way is to do continuous export of application insights data to Azure Blob Storage. Storage cost is cheaper however you will not be able to query it using application insights UI. It will be just a dump of data collected. So good for showing audit and compliance teams that you are maintaining the data but it is not of much use.

The other option to retain beyond 90 days is preview feature Microsoft is working on. Through this feature you would be able to even store only till 30 days to 730 days. Refer  - https://feedback.azure.com/forums/357324-application-insights/suggestions/17454031

In our above scenario of our customer this will not help; as to move into production customer will have to wait for 90 days.

Option 3 - PURGE


Microsoft Azure launched REST APIs for application insight that can help you to delete/ remove/clear/ Purge data stored in application insights based on filters you specify. This will exactly fill into our scenario.  Let us build those apis and use it for deleting the application insights data.

Authorization token and Postman


For calling PURGE REST API I am going to use Postman. “Purge application insights data” is a management operation. Hence we will need proper authorized token for calling the APIs. This is the time where you should subscribe to my blog. Because recently I wrote a blog post on how to use Azure AD service principal for invoking the management API of Azure. Refer - https://sanganakauthority.blogspot.com/2019/04/how-to-create-service-principal-or-app.html

PURGE APIs


This is post method. Add the body as below –

{
"table": "customEvents",
"filters": [
              {
                "column": "timestamp",
                "operator": ">",
                "value": "2019-01-01T00:00:00"
              }
]
}

In the above body we are specifying the table name and timestamp filter. Means all the data collected after 01/01/2019 in customEvents table will be deleted. So for deleting data from all tables you will need run the API for those many tables with required date filter. Refer below screenshots for setting up Params, Headers and Body for PURGE operation in postman –





Where I can find other application insights tables?


Open your application insights in Azure portal. Then click on “Analytics” option as shown below –



Left hand side blade shows all the tables of application insights –




Status of PURGE operation


The above API starts the application insights data deletion operation. The operation returns a GUID as shown below –




This id can be used to know status of PURGE operation. The delete of application insights data will take upto 72 hours. To know the status of PURGE operation refer to below REST API –


This is GET operation and will also require Authorization header token. After 72 hours you will find that the data has been deleted.

Conclusion


I hope this post have helped you to purge azure application insights data.

Please provide your valuable comments.

Keep Purging!!



2 comments: