none
Cache connections throttled where they shouldn't

    Question

  • Hi,

    I've set up a 1GB AppFabric cache. Per the documentation, I should support 40 concurrent connections. However - I'm getting throttled, almost all the time. I make (at most) two concurrent calls, I have MaxConnectionsToServer set to 10, and there are two copies of the factory, which means I theoretically shouldn't go over 20 connections anyway, right? Not that I'm exceeding two right now as is.

    Any ideas on why this could be happening? This is the exception:

    Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0009>:There is a temporary failure. Please retry later. (The request failed, because you exceeded quota limits for this hour. If you experience this often, upgrade your subscription to a higher one). Additional Information : Throttling due to resource : Connections. ---> Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0009>:There is a temporary failure. Please retry later. (The request failed, because you exceeded quota limits for this hour. If you experience this often, upgrade your subscription to a higher one). Additional Information : Throttling due to resource : Connections.

    Friday, May 27, 2011 6:20 PM

Answers

  • "So if we have a web role running 1 website, we run a minimum of 2 instances for availability, even if we set DataCacheFactoryConfiguration.MaxConnectionToServer to 1, that is 2 connections for the cache session provider and 2 connections to access the cache for other data via a DataCacheFactory, that is going to count as 4 connections?" -- That is correct.

    "Then is you are lucky enough to have a traffic spike and bring up a 3rd instance, would that require 6 connections?" -- That is correct too.

    "If your web role runs more than one website, and they all need to access the same cache, will you have to multiply this by the number of sites (that brings us up to 18 on just one web role)? " -- Not necessarily. -- Your web role could have a static instance of DataCacheFactory which can be used by all the websites.


    "Does setting DataCacheFactoryConfiguration.MaxConnectionToServer to 1 mean that there will only be 1 actual connection to the cache from the web role that can only service one request at a time?" -- This means that there will be only one connection for each CacheFactory instance that you make. Different factories don't share connections.

    Friday, June 03, 2011 11:50 AM

All replies

  • Hi Mark,

    > Throttling due to resource : Connections.

    A blog says that there will be three possible reasons causing this issue:

    a. DataCacheFactoryConfiguration.MaxConnectionToServer is set to value of more than 5 : In this case, whenever you instantiate a DataCache object from a unique DataCacheFactory, your DataCache is going to open more than 5 connections to Azure AppFabric Cache service. That would mean that you are exceeding the allowed count.

    Solution: Try setting the value to 1 and try again.

    b. Several instances of DataCaches created from different DataCacheFactory: If in your code you are initializing several DataCacheFactories and do a GetDefaultCache() from each one of them, then each instance of DataCache you create is going to have it’s own connections to the Azure cache service.

    Solution: Create only a single instance DataCacheFactory in your code and re-use it as much as possible. If you really need to have several instances of DataCacheFactories, please ensure to Dispose() them off, once they are not needed.

    c. Several instances of application running with Windows Azure: It might also be a case that your application has several instances deployed and each one of them is using the same cache. In such a case you are exhausting the connections since each instance is creating connections to Azure cache service.

    Solution: Upgrade your cache subscription.

    If the issue is still there after trying these solutions, may ask you for a sample project for reproducing purpose? This will very helpful for resolving the issue.

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Monday, May 30, 2011 6:10 AM
  • Hi Wenchao,

    So just to clarify connections are the total number of instances * DataCacheFactoryConfiguration.MaxConnectionToServer * the number of different DataCacheFactory per instance.

    So if we have a web role running 1 website, we run a minimum of 2 instances for availability, even if we set DataCacheFactoryConfiguration.MaxConnectionToServer to 1, that is 2 connections for the cache session provider and 2 connections to access the cache for other data via a DataCacheFactory, that is going to count as 4 connections? 

    Then is you are lucky enough to have a traffic spike and bring up a 3rd instance, would that require 6 connections?

    If your web role runs more than one website, and they all need to access the same cache, will you have to multiply this by the number of sites (that brings us up to 18 on just one web role)? 

    Does setting DataCacheFactoryConfiguration.MaxConnectionToServer to 1 mean that there will only be 1 actual connection to the cache from the web role that can only service one request at a time?

    Thanks

    Paul

    Wednesday, June 01, 2011 2:01 AM
  • Hi Wengchao,

    Unfortunately, I've already seen that blog, and it hasn't solved my issue. I have a 1GB cache, which means I support 40 connections. I do recreate my factory, but its properly disposed after each use. There is one instance of my code running at any given time, with at most two concurrent calls - which means that even with two calls, I shouldn't go over my limit of 40 calls. Note that lowering my MaxConcurrentCalls results in the same error.

    My code is fairly complex so I can't really post it as-is, I'll see if I can pull out the necessary code to something simpler.

    Thursday, June 02, 2011 4:17 PM
  • Hi Mark,

     

    You could contact  https://support.microsoft.com/oas/default.aspx?gprid=14924&st=1&wfxredirect=1&sd=gn for any cache related issues. Also, it would be very useful in detecting what's going wrong if you could get the caching logs as described here http://msdn.microsoft.com/en-us/library/gg618006.aspx and send them across to :akshar<[at]>live.in.

    Thanks,
    Akshat

    Friday, June 03, 2011 10:34 AM
  • "So if we have a web role running 1 website, we run a minimum of 2 instances for availability, even if we set DataCacheFactoryConfiguration.MaxConnectionToServer to 1, that is 2 connections for the cache session provider and 2 connections to access the cache for other data via a DataCacheFactory, that is going to count as 4 connections?" -- That is correct.

    "Then is you are lucky enough to have a traffic spike and bring up a 3rd instance, would that require 6 connections?" -- That is correct too.

    "If your web role runs more than one website, and they all need to access the same cache, will you have to multiply this by the number of sites (that brings us up to 18 on just one web role)? " -- Not necessarily. -- Your web role could have a static instance of DataCacheFactory which can be used by all the websites.


    "Does setting DataCacheFactoryConfiguration.MaxConnectionToServer to 1 mean that there will only be 1 actual connection to the cache from the web role that can only service one request at a time?" -- This means that there will be only one connection for each CacheFactory instance that you make. Different factories don't share connections.

    Friday, June 03, 2011 11:50 AM
  • Hi,

    I would like to check up on the status of this issue. I seem to be encountering the exact same problem, that is a throttling due to number of connections.

    My scenario is one of the simplest, two web roles with one web application that use the cache session provider and one instance (each) of a DataCacheFactory. That would have my concurrent connections reach a maximum of four connections at a time and shouldn't be a problem with my currect cache subscription (that is a 128MB cache with 5 connections). I have read almost every post there is over this issue and still can't understand why I am getting this error.

    Furthermore, this always happens when the application is trying to access the session data and not when trying to fetch data from a DataCacheFactory. That is really serious as it's not just a matter of not getting cached data but not being able to get session data and thus crashing my application.

    I would be happy to provide any data necessary to solve this issue.

    Thanx in advance,

    Kostas

    Wednesday, August 24, 2011 3:49 PM
  • Anything on this?
    Sunday, September 04, 2011 11:19 PM
  • We are experiencing the same issue when trying to Remove an object from the cache.  Is there any update on this ?
    Tuesday, September 27, 2011 6:45 PM
  • Please note this behavior has been changed in recent versions of the SDK. From November 2011, the Windows Azure SDK Caching has the ability to pool connections. In your case this means, using the default configuration, every Windows Azure Instance (running a single website) only requires a single connection to the Azure backend if you share the same dataCacheClient.

    "Beginning with the November 2011 release of the Windows Azure SDK, Caching has the ability to pool connections. When connection pooling is configured, the same pool of connections is shared for a single application instance. The number of shared connections is determined by the maxConnectionsToServer configuration setting (the default is 1)."

    For more information: Understanding and Managing Connections in Windows Azure


    • Edited by Djon Kleine Wednesday, January 18, 2012 8:58 PM
    • Proposed as answer by Djon Kleine Wednesday, January 18, 2012 8:58 PM
    Wednesday, January 18, 2012 8:55 PM