none
Cosmo db partition storage metric graph is showing one bar where data + index is 18GB. Should it have partition when the 10GB limit is reached? RRS feed

  • Question

  • I've inserted enough data so that the data+index is now 18GB. There is only one bar in the graph and  "No partition keys with significant size were detected for partitioning"

    I've also tried to do a few queries while inserting data to max out the RU to 50K throughput, as my throughput is set to 2500RU.

    The partition key is a guid, so it is well distributed. I thought it would have hit the 10GB logical partition limit and so would have created at least 2 physical partitions.

    Based on what I'm seeing, have I mis-read the graph and it is like a partition collection of logical partitions, and the partition storage limit is higher than the 10GB logical partition limit.

    Maybe my request hasn't reached the  N = T/t partitions limits? (Issue 7248Stackoverflow)

    Is there some way I can make it partition, or do I keep on inserting data until some partitioning limit is hit.

    Out of the box Azure FHIR cosmos has a 10K RU limit. I'm concerned about the performance if it does partition, as the RU will need to be shared across the partitions.


    • Edited by kphung Sunday, September 15, 2019 1:12 PM
    Sunday, September 15, 2019 1:11 PM

All replies

  • Hi kphung,

    I believe there is compression on disk and/or you have not hit the 10Gb logical partition limit. I am looking for an older discussion that will be helpful here. I believe that would be why. Because this is entirely managed by the database engine you will need to keep inserting data until the threshold has been reached.

    Please let us know if you have additional questions.

    Regards,

    Mike

    Thursday, September 19, 2019 12:08 AM
    Moderator
  • Hi Mike,

    Following from your advice, I've inserted more data. It's now 21.7GB.

    I googled around about compression, but it didn't seem to fit my case. If compression was turned on, I thought the storage size should be based on the compressed size. There is no mention of this in the graph.

    The msdn doc (Partition-Data) suggests: "A physical partition supports the maximum amount of storage and request units (RUs). "

    Would this be the maximum for a partition collection, which in earlier versions of cosmos db was 250GB (Partition Collection). Not sure what it is now.

    The feedback section from the msdn article from @rkalivarapu commented:

    "Currently, we have this 10K RU/10GB ratio, this might change in future. Partitions are an internal concept and are transient. Cosmos DB will automatically scale partitions beyond 100 GB at some point as your storage grows without any action from you. "

    This means I have a long way to go before I reach 100GB. This would be more than enough for what we are doing. We might just hit 50GB. However, I need to be mindful that cosmos might also partition, to spread the load.  For now, I'm using an msdn subscription for the proof of concept, the allocated budget won't allow me test these limits. 


    Thanks again for your response.
    Khuong 




    • Edited by kphung Friday, September 20, 2019 5:26 AM
    Friday, September 20, 2019 2:18 AM
  • Thank you for this additional detail, kphung. You are free to reach-out to the PG directly: AskCosmosDB as they are keen on hearing directly from the customer with regard to this topic. 

    Please post any relevant feedback provided by the Product Group that is relevant to your inquiry, as it is helpful to others who are looking for the same information.

    Regards,

    Mike

    Monday, September 23, 2019 5:53 PM
    Moderator