Azure Redis Cache Timeout

2 Answers 1561 Views
Rest Service
Aleksandar
Top achievements
Rank 1
Iron
Aleksandar asked on 12 Oct 2022, 04:26 PM

We're using Azure Reids Cache for our report storage provider. Long running reports result in this error: 
One or more errors occurred.
Timeout performing SET (5000ms), next: SET DispatchAnywhere3Reports\16.1.22.622\Is\948f13fe22c\Rs\6b25108b8df\Ds\6faa346e5ae\R\page30, inst: 108, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: *********************, mc: 1/1/0, mgr: 7 of 10 available, clientName: WS039(SE.Redis-v2.6.48.48654), IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=44,Free=8147,Min=8,Max=8191), v: 2.6.48.48654 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Rest Reports Config:

<restReportService hostAppId="DispatchAnywhere3Reports" reportSharingTimeout="0" clientSessionTimeout="15">
<storage provider="Redis2">
<parameters>
<parameter name="configuration" value="****,password=****,ssl=True,abortConnect=False />
<parameter name="databaseNumber" value="6" />
</parameters>
</storage>
</restReportService>

Does anyone have experience using Azure Redis? Are there additional configuration settings required?

Thanks for your help!

2 Answers, 1 is accepted

Sort by
0
Todor
Telerik team
answered on 17 Oct 2022, 02:55 PM

Hi Aleksandar,

If I understood correctly, the problem happens only with large reports. In the particular error message, I see that the failure happens when the REST Service tries to store that generated 30th page of the report.

Did you check the Internet for similar problems? For example, the following thread Exception: Timeout performing GET (5000ms) #2066.

The article referenced in the error message explains the meanings of the parameters in the error. For example, based on 'qs=0' it seems that 'There are 0 operations currently awaiting replies from Redis server.' I don't see anything disturbing also in the other parameter values, although I am not very experienced with Redis. Can you share some other logs from the Azure Redis database? Hopefully, it will explain what causes the delay.

Does the issue reproduce if you use the Reporting REST Service with a local Redis database?

Regards,
Todor
Progress Telerik

Brand new Telerik Reporting course in Virtual Classroom - the free self-paced technical training that gets you up to speed with Telerik and Kendo UI products. Check it out at https://learn.telerik.com/.
Aleksandar
Top achievements
Rank 1
Iron
commented on 17 Oct 2022, 05:32 PM

Hi Todor,

Thanks for the reply! I was finally able to render the report after making some modifications to the connection string. "*******.redis.cache.windows.net:6380,password=******,ssl=True,abortConnect=False,connectTimeout=15000,syncTimeout=300000"

With that said, performance on our local instance of Redis was significantly better. The report in question now takes over 5 minutes to render whereas previously it only took 20 seconds. How can we improve performance of reports using Azure Redis? I've already increased the plan to the level P1 Premium.

 

Todor
Telerik team
commented on 20 Oct 2022, 08:39 AM

Hello Aleksandar,

How did you measure the rendering time? Did you attach a Trace Listener to the Reporting REST Service? Our code logs the timings for processing and rendering. If not, please do this to compare the timings before and after changing the connection string. If your REST Service is hosted in a .NET Core/5/6 application, you may check the KB Troubleshooting reporting implementation into ASP.NET Core application for details.

Generally, the rendering process shouldn't depend on the storage. The request/response timing between the viewer and the service may depend on the storage settings though, as the service uses its storage to keep the client state.

0
Aleksandar
Top achievements
Rank 1
Iron
answered on 23 Feb 2023, 03:25 PM

Posting a follow-up here in case someone finds this useful. Increasing the connectTimeout and syncTimeout was not the solution and I don't recommend those settings being applied. The problem boils down to the number of Busy threads exceeding the Min threads. Stackexchange.Redis throttles new threads beyond the Min value. Full explanation here: https://techcommunity.microsoft.com/t5/azure-paas-blog/azure-redis-timeouts-client-side-issues/ba-p/2022203#:~:text=Timeouts%20in%20Azure%20Cache%20for%20Redis%20occurs%20on,client%20application%20logs%20as%20Redis%20timeout%20error%20message.

I resolved the issue following the recommendation of adjusting the threadpool settings:
ThreadPool.SetMinThreads(1024, 100);

Todor
Telerik team
commented on 28 Feb 2023, 09:53 AM

Hi Aleksandar,

Thank you for digging deeper into the problem and for sharing your findings with our community.

As a token of gratitude, we have updated your Telerik points.

We will investigate further and if necessary, we will follow up in this thread.

Tags
Rest Service
Asked by
Aleksandar
Top achievements
Rank 1
Iron
Answers by
Todor
Telerik team
Aleksandar
Top achievements
Rank 1
Iron
Share this question
or