Azure Redis Cache Timeout

[Aleksandar]

We're using Azure Reids Cache for our report storage provider. Long running reports result in this error: 
One or more errors occurred.
Timeout performing SET (5000ms), next: SET DispatchAnywhere3Reports\16.1.22.622\Is\948f13fe22c\Rs\6b25108b8df\Ds\6faa346e5ae\R\page30, inst: 108, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: *********************, mc: 1/1/0, mgr: 7 of 10 available, clientName: WS039(SE.Redis-v2.6.48.48654), IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=44,Free=8147,Min=8,Max=8191), v: 2.6.48.48654 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Rest Reports Config:

<restReportService hostAppId="DispatchAnywhere3Reports" reportSharingTimeout="0" clientSessionTimeout="15">
<storage provider="Redis2">
<parameters>
<parameter name="configuration" value="****,password=****,ssl=True,abortConnect=False />
<parameter name="databaseNumber" value="6" />
</parameters>
</storage>
</restReportService>

Does anyone have experience using Azure Redis? Are there additional configuration settings required?

Thanks for your help!

[Todor]

Hi Aleksandar,

If I understood correctly, the problem happens only with large reports. In the particular error message, I see that the failure happens when the REST Service tries to store that generated 30th page of the report.

Did you check the Internet for similar problems? For example, the following thread Exception: Timeout performing GET (5000ms) #2066.

The article referenced in the error message explains the meanings of the parameters in the error. For example, based on 'qs=0' it seems that 'There are 0 operations currently awaiting replies from Redis server.' I don't see anything disturbing also in the other parameter values, although I am not very experienced with Redis. Can you share some other logs from the Azure Redis database? Hopefully, it will explain what causes the delay.

Does the issue reproduce if you use the Reporting REST Service with a local Redis database?

Regards,
Todor
Progress Telerik

[Aleksandar]

Hi Todor,

Thanks for the reply! I was finally able to render the report after making some modifications to the connection string. "*******.redis.cache.windows.net:6380,password=******,ssl=True,abortConnect=False,connectTimeout=15000,syncTimeout=300000"

With that said, performance on our local instance of Redis was significantly better. The report in question now takes over 5 minutes to render whereas previously it only took 20 seconds. How can we improve performance of reports using Azure Redis? I've already increased the plan to the level P1 Premium.

 

[Todor]

Hello Aleksandar,

How did you measure the rendering time? Did you attach a Trace Listener to the Reporting REST Service? Our code logs the timings for processing and rendering. If not, please do this to compare the timings before and after changing the connection string. If your REST Service is hosted in a .NET Core/5/6 application, you may check the KB Troubleshooting reporting implementation into ASP.NET Core application for details.

Generally, the rendering process shouldn't depend on the storage. The request/response timing between the viewer and the service may depend on the storage settings though, as the service uses its storage to keep the client state.

[Aleksandar]

Posting a follow-up here in case someone finds this useful. Increasing the connectTimeout and syncTimeout was not the solution and I don't recommend those settings being applied. The problem boils down to the number of Busy threads exceeding the Min threads. Stackexchange.Redis throttles new threads beyond the Min value. Full explanation here: https://techcommunity.microsoft.com/t5/azure-paas-blog/azure-redis-timeouts-client-side-issues/ba-p/2022203#:~:text=Timeouts%20in%20Azure%20Cache%20for%20Redis%20occurs%20on,client%20application%20logs%20as%20Redis%20timeout%20error%20message.

I resolved the issue following the recommendation of adjusting the threadpool settings:
ThreadPool.SetMinThreads(1024, 100);

[Todor]

Hi Aleksandar,

Thank you for digging deeper into the problem and for sharing your findings with our community.

As a token of gratitude, we have updated your Telerik points.

We will investigate further and if necessary, we will follow up in this thread.