Redis cache, RedisSessionStateProvider, MS ReportViewer == performance problems

Question

Hi all,we are currently migrating from the old Azure Managed Cache (which will be shut down shortly) to the Redis Cache. All is going well except of our reporting project (it's a website project in visual studio). Only there we have session turned on via web.config like this:

        <add name="RedisSessionProvider"
             type="Microsoft.Web.Redis.RedisSessionStateProvider"
             settingsClassName="Common.Configuration.ConfigurationSettings"
             settingsMethodName="GetRedisCacheConnectionString"
             databaseId="1"
             applicationName="reporting"
             retryTimeoutInMilliseconds="5000" />

And there is one aspx page with report viewer:

        <rsweb:ReportViewer id="ReportViewer" runat="server" width="100%" height="100%" SizeToReportContent="true" OnReportRefresh="ReportViewer_ReportRefresh" KeepSessionAlive="true" />

The report works for small data but when we open the report on larger data (which worked on the Managed Cache previously), it is only loading and loading and never finishing. After waiting for like half an hour, there is an error in our logs:

Unhandled exception occurred in Reporting System.TimeoutException: Timeout performing EVAL, inst: 0, queue: 38, qu: 14, qs: 24, qc: 0, wr: 1, wq: 1, in: 0, ar: 0, clientName: QicsUnity.Web_IN_0, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=4,Free=32763,Min=2,Max=32767), Local-CPU: 7% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags)
   at Microsoft.Web.Redis.StackExchangeClientConnection.<>c__DisplayClass4.<Eval>b__3()
   at Microsoft.Web.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation)
   at Microsoft.Web.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation)
   at Microsoft.Web.Redis.StackExchangeClientConnection.Eval(String script, String[] keyArgs, Object[] valueArgs)
   at Microsoft.Web.Redis.RedisConnectionWrapper.TryUpdateAndReleaseLockIfLockIdMatch(Object lockId, ISessionStateItemCollection data, Int32 sessionTimeout)
   at Microsoft.Web.Redis.RedisSessionStateProvider.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem)
   at System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

I know that ReportViewer component heavily uses session. I spent whole day searching and trying things but to no avail. Our Redis cache is of size C1, memory usage is around 350 MB, CPU usage 10%. Redis server is in the same region as web role (currently we have only one web role that is used for testing). Versions of StackExchange.Redis and Microsoft.Web.Redis.RedisSessionStateProvider are the latest. While the report is "being opened", other clients can work with the cache without problems. Slowlog only shows some EVALSHA commands around 100ms each.

Interesting fact: monitor mode using redis-cli reveals the same commands being repeated over and over again (with around 0.5s between them):

1473775865.429354 [1 52.178.30.43:14509] "EVALSHA" "71fe2e7962348b06aa8ce3e244cdb3f774b4f549" "3" "{reporting_5aklth5kts5jssinoouw5wjd}_Write_Lock" "{reporting_5aklth5kts5jssinoouw5wjd}_Data" "{reporting_5aklth5kts5jssinoouw5wjd}_Internal" "636093726662138908" "3600"
1473775865.429408 [1 lua] "SETNX" "{reporting_5aklth5kts5jssinoouw5wjd}_Write_Lock" "636093726662138908"
1473775865.429422 [1 lua] "GET" "{reporting_5aklth5kts5jssinoouw5wjd}_Write_Lock"
1473775865.429433 [1 lua] "HGET" "{reporting_5aklth5kts5jssinoouw5wjd}_Internal" "SessionTimeout"
1473775865.429447 [1 lua] "EXPIRE" "{reporting_5aklth5kts5jssinoouw5wjd}_Data" "1200"
1473775865.429457 [1 lua] "EXPIRE" "{reporting_5aklth5kts5jssinoouw5wjd}_Internal" "1200"

Tried to raise timeouts such as:

retryTimeoutInMilliseconds="1000" operationTimeoutInMilliseconds = "60000" connectionTimeoutInMilliseconds = "10000"

which did not help.

What can we do about this ReportViewer+RedisSessionStateProvider behavior so that it works without timeouts?

Answer 1

Hello Mek7,

Did you ever solve your problems with ReportViewer, the MS RedisSessionStateProvider, and Azure Redis Cache? 

I am having a different problem, where I get a dataset error if I use configure SessionState in my Redis Cache, and it works if I don't (InProc).   I don't get a timeout, but an error accessing a dataset (see below).  I am running the report in local mode, not remote mode.   I'm going to look to see if I should adjust timeouts, but other usage of SessionState seems to be working, just not sure why ReportViewer is not in some instances.  Perhaps there is a class that we need to make serializable, but we don't know which one, the error is not specific enough to know if that is it or not.

The error I see in the ReportViewer is:

• An error occurred during local report processing.
  • An error has occurred during report processing.
    • Cannot create a data reader for dataset 'DataSet1'.

If I switch back to InProc SessionState in my web.config, it works fine.  All other processing of SessionState is normal in my app (as far as I know yet).

Any idea how to get more detail on the error from ReportViewer?

Just wondering what you learned.

Thanks in advance,

Bo

Answer 2

I found another link that suggested that in local mode, to set AsyncRendering = false, and that solved this problem.  Makes some sense, but not exactly.  I consider this a workaround.

Answer 3

Hello,

good that you solved your problem. As for my original problem - we removed ReportViewer and now we are rendering the reports to PDF which can be downloaded afterwards. The ReportViewer itself is a very buggy, legacy and unsupported component, as it seems.

Answer 4

Hi Mek7,

Can you provide some additional information about your environment. Which version of Visual Studio was this solution developed, Windows Server version, etc? I would like to understand where you are at so that I can figure out what to investigate if you are still experiencing issues.

Thank you,

Mike

Answer 5

This document might be of assistance: https://docs.microsoft.com/en-us/azure/redis-cache/cache-aspnet-output-cache-provider

Answer 6

Hi Mek7,

Can you provide some additional information about your environment. Which version of Visual Studio was this solution developed, Windows Server version, etc? I would like to understand where you are at so that I can figure out what to investigate if you are still experiencing issues.

Thank you,

Mike

Hi,

as I wrote previously, we solved this problem by removing ReportViewer.

Answer 7

Making available to the Redis Cache User Voice link if you wish to provide constructive feedback to the product group: https://feedback.azure.com/forums/169382-cache

Note: I did not see any previous comments related to ReportViewer functionality.

Regards,

Mike

Answer 8

To anyone - especially MSFT or who might know an answer to the problem I posted above on March 6, 2018...

Getting an error in ReportViewer when I change my web.config to use Microsoft.Web.RedisSessionStateProvider with an Azure Redis Cache, instead of the default InProc SessionState.  I'm doing this so I can move my stateful web app to Azure App Services - scaled out - without AAR Affinity=ON.  You know, using Redis for SessionState like I think we should!

The error I see in ReportViewer trying to do this:

• An error occurred during local report processing.
  • An error has occurred during report processing.
    • Cannot create a data reader for dataset 'DataSet1'.

Anyone else have this problem using ReportViewer and Microsoft.Web.RedisSessionStateProvider with an Azure Redis Cache?  Any solutions?    Or suggestions for different Azure compatible RedisSessionStateProvider/etc to get the job done?

It appears if we turn AsyncRendering = false, the problem goes away, but the user interface is better with AsyncRendering = true, so I'd like to find a better solution.

Anyone from MSFT here to provide some info? We really need to solve this problem this time around.

Thanks in advance!

Bo.

Answer 9

Hello Mike,

Are you still active in this Azure Cache forum?  I would like to pursue a solution to the problem I posted last year, and am still having.  Do you have any suggestions where else I should go, or who to ask about this?

Thanks!

Bo.

Answer 10

Let me reach out to the product group. I just responded to the thread you created. 

Regards,

Mike