How to delete or purge Azure Application Insights data


Abs07act


English is not my mother tongue. So often it takes a bit of time for myself to understand few English statements; if it does not contain regular words. Honestly “PURGE” was one of such word where I had to search for dictionary.

//PURGE//

Means: To purify, getting rid of whatever that is impure or undesirable and most important: when you want to start fresh.

How this is related in blog post? Well read on…

What is an APM tool?


One of my customers wanted to do application monitoring during load test. They started configuring monitoring for their Virtual machines hosting applications and database for CPU, Memory, network I/O, disk I/O and so on. Configuration of their monitoring showed them all me07ics of VMs like CPU, memory etc. within 70% however application was failing consistently in load test.

Then they called me and explained the same scenario and said application is failing but there is no 07ace we are getting why it is failing. So I told them, what you are doing currently is good however you are doing only underlying “Infras07ucture” monitoring. This is good but it can’t tell always why application is failing. So you need “Application performance monitoring (APM)” tool for application monitoring.

Folks, application monitoring and infras07ucture monitoring are two different things. If your VM is not healthy then for sure your application will not work however if your application is not working still your Virtual Machine might be healthy. Got it?

So when you say I am monitoring CPU, memory for application you are actually monitoring only underlying infras07ucture capacity/ performance and not the application behavior. To monitor application means you want to understand  -

    Application failures
  1. Exceptions and error raised
  2. User teleme07y and usage
  3. Availability of application
  4. Live me07ics
  5. Response, time, latency and application page load time
  6. Application load time in client browser
  7. Dependencies, slowest operation performed and so on.

This has nothing to do with your underlying infras07ucture performance me07ics monitoring.
So you need a tool that can show all above mentioned things in an application and this is where Application performance monitoring APM tools are used.

There are many APM tools in the market like NewRelic, Dyna07ace, App Dynamics and “Azure Application Insights”.

So APM is generic concept and Application Insights is a product from Azure for APM.

Some customer can be real pain in the …heart [I know what you were thinking].


Customer configured application insights agent and SDK both to monitor various aspects of their few java and .NET based applications for POC. After running the POC for 15 days they were quite satisfied by application insights features. They wanted to start using the same application insights instance in production. However they wanted to start a fresh and hence wanted to delete all the data captured by application insights till now. At the same time they also wanted to retain the application insights service and post data deletion they wanted to start capturing production data in the same service instance of application insights.

There 3 options to delete the application insights data –


I know this is one of the crude ways of doing it. When you delete application insights everything contained inside of it will be deleted. This option would not fit into our scenario described above. When you create a new application insights unfortunately you will have to redo the changes of application insights key and setting again in the application wherever it is referenced. My customer did not want to redo the settings. Customer is a lazy…heart [I know what you were thinking].


This is more of configuring the data retention policy in such a way that it deletes all the data over a time period. Default duration of data retention is 90 days. So post 90 days all the data stored in application insights is automatically removed. Also Application insights is charged as per the amount of storage in GBs you are consuming. So it makes sense to store captured data only for limited period and then delete it automatically. If you want to store data longer than 90 days then one way is to do continuous export of application insights data to Azure Blob Storage. Storage cost is cheaper however you will not be able to query it using application insights UI. It will be just a dump of data collected. So good for showing audit and compliance teams that you are maintaining the data but it is not of much use.


The other option to retain beyond 90 days is preview feature Microsoft is working on. Through this feature you would be able to even store only till 30 days to 730 days. Refer  - https://feedback.azure.com/forums/357324-application-insights/suggestions/17454031

In our above scenario of our customer this will not help; as to move into production customer will have to wait for 90 days.

Microsoft Azure launched REST APIs for application insight that can help you to delete/ remove/clear/ Purge data stored in application insights based on filters you specify. This will exactly fill into our scenario.  Let us build those apis and use it for deleting the application insights data.


For calling PURGE REST API I am going to use Postman. “Purge application insights data” is a management operation. Hence we will need proper authorized token for calling the APIs. This is the time where you should subscribe to my blog. Because recently I wrote a blog post on how to use Azure AD service principal for invoking the management API of Azure. Refer - https://sanganakauthority.blogspot.com/2019/04/how-to-create-service-principal-or-app.html


https://management.azure.com/subscriptions/YourSubscriptionID/resourceGroups/YourResourceGroup/providers/Microsoft.Insights/components/ApplicationInisghtsservice/purge?api-version=2015-05-01


This is post method. Add the body as below –

{
"table": "customEvents",
"filters": [
              {
                "column": "timestamp",
                "operator": ">",
                "value": "2019-01-01T00:00:00"
              }
]
}

In the above body we are specifying the table name and timestamp filter. Means all the data collected after 01/01/2019 in customEvents table will be deleted. So for deleting data from all tables you will need run the API for those many tables with required date filter. Refer below screenshots for setting up Params, Headers and Body for PURGE operation in postman –





Open your application insights in Azure portal. Then click on “Analytics” option as shown below –