This is the fifth post of my series of posts about Log Analytics and Application Insights. The previous post talked about adding custom logging to you code using Application Insights. Now you’ve got your logging into Application Insights you can run log analytics queries and build dash boards, alerts etc. Sometime though you want to use this data in other systems and it would be useful if you could export the data and use it else where. This post will show you how you can regularly export the data from Application Insights into Azure Storage. Once it is in Storage it can easily be moved into other systems or used else where such as PowerBI. This can be achieved using the Continuous Export feature of Application Insights
To enable Continuous Export, login to the Azure management portal and navigate to Application Insights. Click on the instance you want Continuous Export enabled. The scroll down the options on the left until you find the Configure section and click on Continuous Export
To use Continuous Export you will need to configure a storage account. Click Add:
Click Data types to export:
I was only interested in the logs I emitted from my custom logging, so selected Custom Event, Exception and Trace then clicked OK
Next pick a storage location. Make sure you use one where your Application Insights instance is located otherwise you will be charged egress fees to move the data to a new datacentre.
You can now pick an existing storage account or create a new one. Upon selecting a storage account you can pick an existing container or create a new one.Once a blob container is selected click OK.
Continuous Export is now configured. You will not see anything in the storage container until the next set of logs are sent to Application Insights.
My log analytics query shows the following logs have been generated:
If you look at the continuous Export configuration page you will see that the last updated date has changed.
Now look in blob storage. You should see a folder that is<ApplicationInsights_ServiceName>_<ApplicationInsights_InstrumentationKey
Click through and you will see a number of folders. One for each of the logs that I enabled when setting up Continuous Export. Click through one of them and you will see a folder for the date and then a folder for the hour of the logs. Then a file containing the logs for that hour.
You will get a row of json data for each row output for the log query. Note the whole logs emitted will be in each of the folders.
e.g.
{
"event": [
{
"name": "Some Important Work Completed",
"count": 1
}
],
"internal": {
"data": {
"id": "a guid is here",
"documentVersion": "1.61"
}
},
"context": {
"data": {
"eventTime": "2020-03-22T18:12:30.2553417Z",
"isSynthetic": false,
"samplingRate": 100.0
},
"cloud": {},
"device": {
"type": "PC",
"roleInstance": "yourcomputer",
"screenResolution": {}
},
"session": {
"isFirst": false
},
"operation": {},
"location": {
"clientip": "0.0.0.0",
"continent": "Europe",
"country": "United Kingdom",
"province": "Nottinghamshire",
"city": "Nottingham"
},
"custom": {
"dimensions": [
{
"CustomerID": "4df16004-2f1b-48c0-87d3-c1251a5db3f6"
},
{
"OrderID": "5440d1cf-5d06-4b0e-bffb-fad522af4ad1"
},
{
"InvoiceID": "a7d5a8fb-2a2e-4697-8ab4-f7bf8b8dbe18"
}
]
}
}
}
As the data is now out of Application Insights you can move it where ever you need it. You will also need to manage the blob storage data too otherwise you will end up with the logs stored in two places and the storage costs will be doubled.
One example of subsequent usage is exporting the data to Event Hub. As the data is in blob storage you can use a function with a blob trigger to read the blob in a row at a time and publish the data onto Event Hub:
[FunctionName("ContinuousExport")]
public static async void Run([BlobTrigger("logs/{name}", Connection = "ConitnuousExportBlobSetting")]Stream myBlob, string name,
[EventHub("logging", Connection = "EventHubConnectionAppSetting")] IAsyncCollector<string> outputEvents,TraceWriter log)
{
log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
StreamReader textReader = new StreamReader(myBlob);
while (!textReader.EndOfStream)
{
string line = textReader.ReadLine();
log.Info(line);
await outputEvents.AddAsync(line);
}
}
Note: This is an example, so will need additional code to make sure that you don’t exceed the Event Hub maximum message size
So with Continuous Export you can extract your log data from Application insights and move it to other systems for processing.