locked
Exceptional slow batch operation for Azure Table RRS feed

  • Question

  • I have a VM Medium with an Azure Storage.

    • West Europe
    • Same Affinity Group
    • Different storage account for VM and for my application

    I noted that today, during the afternoon, my storage became very very slow all of a sudden.
    I was happily uploading my batches with the rate of 2 MB/S, and now, suddenly this afternoon, I started to get better than 300KB/S.

    I noticed that my network and CPU activity were very low.
    I started Fiddler, and seen that my batches of 100 KB approximately took 14 second to upload !

    I tested the same program on a VM in a different subscription with different affinity group, the same exact same batch took 200ms approximately.

    What can do enjoy my full speed again ? Seems like a mouse is eating the cable of my rack.





    Friday, January 9, 2015 12:44 AM

All replies

  • Hi Nicolas,

    Thanks for your feedback.

    Firstly, I suggest you could check your regions speed via this page(http://azurespeedtest.azurewebsites.net/ ).

    And at the same time, I will investigate this issue and post back for you later. Thanks for your understanding.

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, January 9, 2015 10:57 AM
  • Not very good.... I launch it on directly on the VM.

    Friday, January 9, 2015 12:34 PM
  • On the other hand, the latency I get from the TCP Connection list, seems fine (less than 12 ms)

    So only batch are unbelievably slow, I'll try using another storage account...

    Friday, January 9, 2015 1:59 PM
  • So after further tests, the slowdown happens only on a particular Azure Table. (which name is "transactions")

    I get expected throughput if I am using a new table.

    Why would a table make my batch throughput slower ?

    Same storage account, different table : everything is fine.

    I remember that at the beginning my slow table was not slow. In the past, insert were always fast regardless of the size of partition. Have you introduced some changes into azure storage that broke that ?




    Friday, January 9, 2015 2:09 PM
  • I have an idea : maybe my partitions are too crowded. It surprise me because I never had any problem with that before.

    What is the impact of partition size on insert time ?

    Please do not respond me with the blog post on how to scale with partitions. I read them all, multiple time, and no one talk about the impact of insert speed depending on partition size.

    My comprehension is that each partitions have performance targets on query, which I am very far from reaching.
    However nobody speak about how insert time evolve depending on partition size. Can you tell me more ?
    Having a 100KB batch insert taking 20 seconds is very weird. It is like if the performance of insert grew linearly with the size of the partition.
    I think you are using binary trees internally to make range query efficient, so I expect to be O(LogN).

    Again, this problem is very recent, and I had no problem before.

    Friday, January 9, 2015 5:10 PM
  • Update :

    I eventually it the threshold with a new table, my insert on the other table are starting to take more than 2 seconds to process, and it goes up with time.

    So, insert is not constant, but depend on partition's size. Is it normal ?

    Niranjan Nilakantan of Microsoft said

    If the key for your logical data model has more than 1 property, you should default to (multiple partitions, multiple rows for each partition).

    If the key for your logical data model has only one property, you would default to (multiple partitions, one row per partition).

    We have two columns in the key to separate what defines uniqueness(PartitionKey and RowKey) from what defines scalability(just PartitionKey). 
    In general, write and query times are less affected by how the table is partitioned.  It is affected more by whether you specify the PartitionKey and/or RowKey in the query.

    If this is true, there is a real problem with my storage node.


    Saturday, January 10, 2015 11:57 AM
  • Since I can reproduce the problem, and that it clearly comes from azure storage, I just reposted the repro and data on this other post.
    Saturday, January 10, 2015 7:25 PM
  • Just proposed an answer in your related post. Please check it out. Hopefully it helps.
    Monday, March 9, 2015 9:09 PM