none
Turning On Diagnostics eats up transactions ( MACommand.xml ) - a serious issue !!!

    Question

  • We were just trying out the Azure Storage Analytics Service, and something very unusual caught our attention. 

    The transaction count for the diagnostics storage account ( the account to which the Diagnostics Service writes it's data) was extremely high. We are talking about 600~ transaction per hour, all of which are GetBlob() operations, and all of them ended with error ( ClientOtherError is equal to the total number of operations ). Further investigation revealed that each instance running which has Diagnostics turned on, produces 300~ transactions per hour ( we has 2 instances, thus the 600). Continuing the investigation, looking at the $logs that the Analytics Service is producing revealed what really going on :

    The log is filled with lots of calls to an xml file that's not exists. The log file itself is very cluttered  but it's very clear that most of the calls are searching for  

    https://*****.blob.core.windows.net/mam/MACommand.xml  and also /mam/MACommandb.xml and /mam/MACommandb.xml

    all those calls have an error of 404.

    This issue is a real problem for us, and we have no idea what causing it.

    Has anyone encountered this issue ?

    • (edit: Forgot to mention, the Diagnostics Service is not logging anything - scheduledTransferPeriod is zero for all the categories)

     

    • (edit: here is some part of the log file, sensitive information is replaced with ****

     

    1.0;2012-01-06T19:38:21.5182981Z;GetBlob;ClientOtherError;404;4;4;authenticated;****;****;blob;"https://****.blob.core.windows.net/mam/****/****/TestApp.Web/TestApp.Web_IN_0/MACommand.xml";"/****/mam/****/****/TestApp.Web/TestApp.Web_IN_0/MACommand.xml";2608d6f1-1129-4abd-bbb6-2fcc1f7f9315;0;213.199.129.92:49581;2009-09-19;373;0;145;225;0;;;;;;"MAHttpClient";;

    1.0;2012-01-06T19:38:21.5482951Z;GetBlob;ClientOtherError;404;4;4;authenticated;****;****;blob;"https://****.blob.core.windows.net/mam/****/****/TestApp.Web/TestApp.Web_IN_0/MACommanda.xml";"/****/mam/****/****/TestApp.Web/TestApp.Web_IN_0/MACommanda.xml";9387eed2-9065-42af-8b7b-142458d405a1;0;213.199.129.92:49581;2009-09-19;374;0;145;225;0;;;;;;"MAHttpClient";;

    1.0;2012-01-06T19:38:21.5772922Z;GetBlob;ClientOtherError;404;5;5;authenticated;****;****;blob;"https://****.blob.core.windows.net/mam/****/****/TestApp.Web/MACommanda.xml";"/****/mam/****/****/TestApp.Web/MACommanda.xml";0f1e19ae-7796-4fdb-b575-3d115c65ef07;0;213.199.129.92:49581;2009-09-19;357;0;145;225;0;;;;;;"MAHttpClient";;

    • (edit: The issue happens regardless of the role type WebRole/WorkerRole)

     



    • Edited by Yaron Levi Sunday, January 08, 2012 12:15 PM
    Friday, January 06, 2012 2:10 AM

Answers

  • After a week trying to figure this out, with the help of  Farida Bharmal from the azure support team, we managed solve the issue. Those calls are an expected behavior, and happens from SDK 1.6.
    Taken from a response Farida got from the product team :

    "I also got a response from the product team. There is a change in the SDK 1.6. These calls are expected. There are about 20 calls per 5 minutes for every role instance. Which means that there will be 6000 calls a day. This will cost around a penny every day for every role instance. (Storage Transactions = $0.01 per 10,000 transactions)"
    The product team also said that they looking into optimizing this behavior.

     
    • Marked as answer by Yaron Levi Tuesday, January 10, 2012 9:57 PM
    Tuesday, January 10, 2012 9:56 PM

All replies

  • Hi,

    Can you post your configuration and steps in order to reproduce the issue?


    If you found this post useful, Please "Mark as Answer" or "Vote as Helpful". Best Regards.
    Friday, January 06, 2012 11:03 AM
  • I have a simple, empty WebRole.

    I have enabled Diagnostics like so :

    I am deploying to the staging environment.

    Other than this, no special config I made. (I have no WebRole.cs with OnStart() ).

    The Diagnostics config file is created automatically in  /wad-control-container/<deploymentID>/<rolename>/<roleinstance>.

    Friday, January 06, 2012 12:37 PM
  • Not sure if this would help, but I ran into this thread here: http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/638e9945-0017-4e8c-a24d-f613002fc369. Guy who posted this thread was also facing problem that you're facing.

    Hope this helps.

    Thanks

    Gaurav Mantri

    Cerebrata Software

    http://www.cerebrata.com

     

    Friday, January 06, 2012 4:09 PM
  • Thank you, but this is not exactly the same problem. Although his log shows something about the enigmatic MACommand.xml, his problem is related to the emulator. My problem is, as I mentioned, when the WebRole is live and running in staging. 
    Friday, January 06, 2012 7:49 PM
  • Hi Yaron,

    You mean you just create a empty WebRole with enable the diagnostics? Do you add some code?

    If you can provide more information for your issue, i think it will help us for reproduce this situation.


    Please mark the replies as answers if they help or unmark if not. If you have any feedback about my replies, please contact msdnmg@microsoft.com Microsoft One Code Framework
    Monday, January 09, 2012 9:14 AM
  • Yes, just create a simple solution (Happens both in Worker and Webrole - Worker produces a more simple solution), and enable Diagnostics through the VS ui.

    Not a single line of code added.

    Also enable Storage Analytics Service for that diagnostics storage account to see the log files (in $log container).

     

    Tuesday, January 10, 2012 8:38 AM
  • After a week trying to figure this out, with the help of  Farida Bharmal from the azure support team, we managed solve the issue. Those calls are an expected behavior, and happens from SDK 1.6.
    Taken from a response Farida got from the product team :

    "I also got a response from the product team. There is a change in the SDK 1.6. These calls are expected. There are about 20 calls per 5 minutes for every role instance. Which means that there will be 6000 calls a day. This will cost around a penny every day for every role instance. (Storage Transactions = $0.01 per 10,000 transactions)"
    The product team also said that they looking into optimizing this behavior.

     
    • Marked as answer by Yaron Levi Tuesday, January 10, 2012 9:57 PM
    Tuesday, January 10, 2012 9:56 PM
  • This is a very bad situation for trial accounts! They will be locked after a few days... :S
    If you found this post useful, Please "Mark as Answer" or "Vote as Helpful". Best Regards.
    Tuesday, January 10, 2012 10:19 PM
  • YUP.

    LOCKED. Disabled Trial Account here. THREE DAYS.

     

    How much does DataSync Utilize?

     

    Wednesday, January 11, 2012 12:37 AM