How to clear out the diagnostics logs in Azure Table Storage?

Question

I have Trace logging working in my Azure roles and it's writing to table storage correctly. I can see the entries in the VS explorer plus other tools.

How do I remove entries from it? It's so large that it takes a while for me to load the whole thing and go to the bottom to see my more recent entries.

Is there a way to get VS to load the most recent thousand records first instead of loading them in ascending order?

I installed the MMC but I can't get it to filter by Timestamp.

Do I have to delete the WADLogsTable?

I assume that doing that while I have running services is a bad idea. Can I suspend my services and delete it? Will it run my startup code and recreate the tables when I restart them?

Does the table just keep building, or will the older entries age out eventually? How long before they age out? Is there a way to modify that?

Thanks,
RobinDotNet

---

Click here to visit my ClickOnce blog!
Microsoft MVP, Client App Dev

Answer 1

Deleting your table and re-initializing your role (making your DiagnosticsMonitor.Start run again), should do the trick, yes.

Answer 2

@Sam --> I would be careful about deleting the table. Deleting the WAD* table is a viable option if you don't have too much data in it. What happens when you delete a table is that it is not deleted at that very moment however it is marked for deletion and some background process actually deletes that table. If you (or the diagnostics process) try to create the same table you would get an error that "Table is being deleted".

@RobinDotNet --> [With due apologies for shameless product plugin :)] May I recommend that you take a look at our product Azure Diagnostics Manager which has a feature to purge logs data. Take a look at this demo which demonstrates how you can purge logs data using Azure Diagnostics Manager: http://cerebrata.blob.core.windows.net/onlinedemos/azurediagnosticsmanager/purgelogsdemo/purgeeventlogsdemo.html . However please keep in mind that purging through this tool first downloads the data and then sends the data for deletion. Also if you're interested in seeing the trace logs or other diagnostics data, I believe Azure Diagnostics Manager can help. It is a tool built to visualize and manage diagnostics data only.

To answer your other questions:

I installed the MMC but I can't get it to filter by Timestamp.

I've not used MMC so I can't comment but here is what you can do: In all diagnostics table, the value of the PartitionKey is higher end bits of DateTime.Ticks and I believe the way PartitionKey is constructed is that for every minute worth of data, there will be one PartitionKey value. So if you can query your table data through MMC, you should get the ticks for the date/time range you're interested in, prepend a 0 in front of that value and query your logs table based on that. Also make sure that when you take date/time, you would need to convert them to UTC.

Do I have to delete the WADLogsTable?

I assume that doing that while I have running services is a bad idea. Can I suspend my services and delete it? Will it run my startup code and recreate the tables when I restart them?

Please see my response above.

Does the table just keep building, or will the older entries age out eventually? How long before they age out? Is there a way to modify that?

Yes. The data in this table will keep on growing and the older entities will not age out. One way to handle this is to ensure that you configure the diagnostics in such a way that only the data that you want to persist gets persisted in Azure Storage.

Hope this helps.

Thanks

Gaurav Mantri

Cerebrata Software

http://www.cerebrata.com

Answer 3

Hello, as Gaurav pointed out above, you can use Cerebrata's Azure Diagnostics Manager. Alternatively, if you only want to use Visual Studio, you can use the following trick to filter the records:

When opening the table, you will see a message "Loading entities. Click here to pause." So click "click here".

Then you can filter the data using partition key. The partition key of the diagnostics table is the tick. So you can write some code to calculate a tick:

DateTime dt = new DateTime(2010, 10, 1);
long l = dt.Ticks;

The result looks like 634214880000000000. Then you can filter the partition key by typing:

PartitionKey gt '0634214880000000000'

Click the Execute button, and you'll get all data recorded after 2010/10/1.

Note PartitionKey is a string, and it usually begins with a 0. Also note time is stored as UTC, which you may want to take into consideration.

Alternatively, you can filter by Timestamp. But since PartitionKey is indexed, you'll get better performance.

Visual Studio doesn't allow you to delete any data. But you can write a simple console application that deletes the data. Just check if PartitionKey is less than a particular tick, and delete the old data.

---

Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.