What is the best practise to do partitoning data in Azure Redis RRS feed

  • Question

  • I am planning to use Redis cache in my application and the data size may vary from 10KB - 500 MB. If I use the large key/value entries from Redis, Reding and Saving data might get performance issue, it means I have to transfer more data over the network from the server to the client and I might get below consequences. (Got below from @stackoverflow)

    1. It takes more time to transfer the data, so client may need to have a higher timeout value configured to allow for this additional transfer time.
    2. Requests made to the server can get stuck behind the big transfer and cause other requests to timeout.
    3. Network buffers used to transfer this data can impact available memory on the client or server, which can aggravate the available memory issues described around fragmentation.
    4. If these large key/value items are accessed frequently, this magnifies the impacts if we repeatedly transferring this data over and over again.
      So I would like to do partitioning in Redis cache.

    Example: I have an endpoint(GetEmployees(Some computation happens to get all employees)) which returns around 500 records with size 400 MB ( Example). I get the same data until I add any new employee so I want to have cache mechanism so that I can store these records and get it from the cache instead of doing calculations again and again.

    If I store all records in the single shard of Redis, will face above mentioned issues.

    Key/Value : GetEmplyeesTodayDate/ {List Of Records}

    So, I would like to split the data and across shards and get all records when reading from Redis.

    What is the best way to store all 500 records in Redis with same/multiple keys (s) and retrieve them?

    Friday, April 5, 2019 9:35 AM

All replies

  • Hello Balu,

    Your use case is best served by a no sql store like Cosmos Db and is more cost effective. 

    Redis cache is not the ideal backend for storing high volume transactional entities like employees, orders, customers etc. Some of the use cases for Redis are documented here

    Please let us know if you have further questions.

    Friday, April 5, 2019 12:01 PM
  • Just checking in to see if the above answer helped.
    If this answers your query, do click “Mark as Answer” and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
    Sunday, April 7, 2019 4:30 PM
  • Adding link to same question asked in Stack Overflow
    Thursday, April 25, 2019 7:09 PM
  • Thanks Kalyan for the info. 

    I have a question in my mind, Though we use cosmoDb or RedisCache, We may get the above mentioned issues if we store the large amount of data. 

    If we store the data in different partitions, we can fetch the data from all the partitions, compute and then sent to the client. If we have a good mechanish to store data as partitions, we might resolve few blockers. Can you please comment it?

    Friday, April 26, 2019 5:29 AM
  • In the example you mentioned above, do you need to return all 500 records in the GetEmployees call at once?

    You should be implementing Pagination in your API with feilds like nextPage or Next50 to get the next set of results.

    The client (caller) can then decide if it needs to make a call to get the next batch.

    Cosmos DB offers excellent scale with large volumes of data. You will only need to ensure that you partition your data correctly in a way that there are no 'Hot Spots'. 

    Please refer to this link to get started.

    Do let us know if you have specific questions.

    Monday, April 29, 2019 4:23 AM