DiagnosticMonitor / Transferring logs to storage

Question

Hi!

I use an automatic transfer of Event Log on schedule (for example, every 10 minutes). My problem is the log transfer in "OnStop" method. I want logs that have not been transferred yet (to the table storage) to be copied there nevertheless. Ideally it would be wonderful just to call a method like "Flush". But for some reason there is not such a way.

I tried to use a technique described here:
http://msdn.microsoft.com/en-us/library/windowsazure/hh508990.aspx
(Chapter "Performing On-Demand Transfers")

However, afterwards BeginOnDemandTransfer nothing happens. Records in the table do not appear, a message does not appears (queue.GetMessage() == null everytime).

Thanks,

Sergey.

Answer 1

Hi,

I assume that, you have errors logged in Event Viewer and you want to transfer all of them to Azure Table Storage. Following is the code. Place it in OnStart of your role

var diagConfig = DiagnosticMonitor.GetDefaultInitialConfiguration();

            int logInterval = Convert.ToInt16(RoleEnvironment.GetConfigurationSettingValue("LogIntervalInSeconds"));//read from config

            diagConfig.Logs.ScheduledTransferLogLevelFilter = LogLevel.Information;
            diagConfig.Logs.ScheduledTransferPeriod = TimeSpan.FromSeconds(logInterval);

            diagConfig.WindowsEventLog.DataSources.Add("Application!*[System[Provider[@Name='ASP.NET 4.0.30319.0'] and (Level=2 or Level=3)]]");
            diagConfig.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromSeconds(logInterval);

            DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString",
              diagConfig);

http://sanganakauthority.blogspot.in/2011/12/azure-diagnostics-creating-windows.html

Hope it helps!!

---

Mark As Answer if it helps you | My Blog

Answer 2

Hi Kunal,

Yes, I am doing like that. The problem is some logs can be lost when an instance is restarted. I've found the similar topic, but there is no the clear answer there:

http://social.msdn.microsoft.com/Forums/en/windowsazuretroubleshooting/thread/25fcd08b-5cbc-49c7-bf6f-d98bc8e14c94

Answer 3

Hi,

I can not find any offcial documentaion says the logs will be lost when instances is restarted, as far as i know, the logs is stored in Local Storage and it will be upload to Blob storage if you configure the Diagnostics, i think Local Storage won't be cleared when instances is restarted and your logs can be uploaded after a period of time because upload isn't an immediate operation. Unless your instances was crashed, the logs may lost, but i think the OnStop method also can not be executed when instance was crashed, so "On Demand Transfer " does not works for this situation.

Hope this helps.

---

Please mark the replies as answers if they help or unmark if not. If you have any feedback about my replies, please contact [e-mail] Microsoft One Code Framework

Answer 4

Hi,

> i think Local Storage won't be cleared when instances is restarted

There is the flag "Clean on role recycle" for a local storage. I set this flag. Maybe I should try to disable cleaning. Thanks for this suggestion!

>  i think the OnStop method also can not be executed when instance was crashed, so "On Demand Transfer " does not works for this situation.

As far as I know there are situations when Azure infrastructure wants to renew the machine for an instance (it closes the old machine and start a new machine for this instance). In this situation the local storage will be lost anyway, even if I disable cleaning for it. But the OnStop method should be called.

Answer 5

Yes, If Azure Platform update the instace (Stop old instances and re-create a new instance), the OnStop method will be invoked, but when more serious problem cause instances crashed, OnStop method may not be invoked.

I think read queue by application is not a recommended way, if your application get the queues, Diagnostices can not get them twice because every queue could be read for once.

---

Please mark the replies as answers if they help or unmark if not. If you have any feedback about my replies, please contact [e-mail] Microsoft One Code Framework

Answer 6

> There is the flag "Clean on role recycle" for a local storage. I set this flag.

I was wrong. DiagnosticMonitor uses it's own predefined local storage, namely "DiagnosticStore":

<DefaultLocalStorage name="DiagnosticStore" sizeInMB="4096" cleanOnRoleRecycle="false"/> (this line is from one of Azure SDK files)

And the flag "cleanOnRoleRecycle" is false by default (I do not change it).

But the initial question is still actual in 2 situations:

1) A role instance is moved to a different virtual machine

2) An instance is deleted (for example when scaling down is performed)

When the instance is crashed nothing helps of course.

Answer 7

Yes, I suggest you use "On Demand Transfer" with these 2 situations.

---

Please mark the replies as answers if they help or unmark if not. If you have any feedback about my replies, please contact [e-mail] Microsoft One Code Framework

Answer 8

> Yes, I suggest you use "On Demand Transfer" with these 2 situations.

I tried but failed as I wrote in the first post.

Answer 9

I do assume the OnStop Method returns before the OnDemandTransfer happens, since the demand transfer is initiated asyncronously to run in the background.

What you may need is to wait in your OnStop method until the RoleInstanceDiagnosticManager does not have any active Transfer ( via .GetActiveTransfers() ) anymore. From a quick look in documentation seems there is no WaitHandle or so available so you'll have to write some code around.

Answer 10

I think the following now. The "BeginOnDemandTransfer" function call really changes only the configuration file in blob storage (in wad-control-container). Once a minute Diagnostic Monitor updates its configuration. This is enough time because 5 minutes are given for OnStop. But when does  Monitor begin the log transfer? My guess is Monitor waits when the scheduled transfer ends (i.e. it can wait several minutes), and only after that it begins OnDemand transfer.

GetActiveTransfers() is used for OnDemand transfers only. Before the "BeginOnDemandTransfer" call GetActiveTransfers() == 0, and after that it's > 0.

Answer 11

When OnStop begins, the system sends the shutdown notification to DiagnosticsAgent.exe. Probably after that Diagnostics Monitor stops to update its configuration once a minute. This can be the cause of OnDemand transfer failing.

Answer 12

Hi,

 Seems like Kunal's input is correct..can you pls paste youe code here which would help us to understand in better way.

---

cpsingh