Large report running out of memory

[Alex]

========================================================================================================== 
1. My Question:
yes, I have an emergency issue about rendering report as pdf in windows service. Total number of rows: 60k. 
When I try 60k rows, it gives me "Running out of memory" issue. But there is no such issue show up when I try to render 20k rows.
.
==========================================================================================================
My Code:
                var reportProcessor = new Telerik.Reporting.Processing.ReportProcessor();
                var deviceInfo = new System.Collections.Hashtable();
                Telerik.Reporting.InstanceReportSource instanceReportSource = new Telerik.Reporting.InstanceReportSource();

                instanceReportSource.ReportDocument = report;  // please assume that I set 60 k rows in var of report. 

                Telerik.Reporting.Processing.RenderingResult result = reportProcessor.RenderReport(
                    "PDF", instanceReportSource, deviceInfo);

                if (!System.IO.Directory.Exists(this.reportServerInfo.TempExportPath))
                {
                    var pathInfo = System.IO.Directory.CreateDirectory(this.reportServerInfo.TempExportPath);
                }

                string filePath = System.IO.Path.Combine(this.reportServerInfo.TempExportPath, fileName);

                using (System.IO.FileStream fs = new System.IO.FileStream(filePath, System.IO.FileMode.Create))
                {
                    fs.Write(result.DocumentBytes, 0, result.DocumentBytes.Length);
                }

========================================================================================================
Error Track:
ERROR 2013-05-24 12:56:16,106 [41] System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Telerik.Reporting.Processing.TextBox.Telerik.Reporting.Processing.ILayoutElementContainer.get_Children()
   at Telerik.Reporting.Paging.TextBox.CreateContent(InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.Init(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.ElementFactory.CreateElement(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreateElement(LayoutElement source, PagingElementBase parent, InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.CreateContent(InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.Init(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.ElementFactory.CreateElement(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreateElement(LayoutElement source, PagingElementBase parent, InitializationContext initContext)
   at Telerik.Reporting.Paging.Group.CreateContent(InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.Init(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.ElementFactory.CreateElement(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreateElement(LayoutElement source, PagingElementBase parent, InitializationContext initContext)
   at Telerik.Reporting.Paging.Group.CreateContent(InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.Init(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.ElementFactory.CreateElement(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreateElement(LayoutElement source, PagingElementBase parent, InitializationContext initContext)
   at Telerik.Reporting.Paging.Group.CreateContent(InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.Init(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.ElementFactory.CreateElement(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreateElement(LayoutElement source, PagingElementBase parent, InitializationContext initContext)
   at Telerik.Reporting.Paging.Report.CreateContent(InitializationContext initContext)
   at Telerik.Reporting.Paging.PagingElementBase.Init(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.ElementFactory.CreateElement(LayoutElement source, PagingElementBase parent, IComposition composition, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreateElement(LayoutElement source, PagingElementBase parent, InitializationContext initContext)
   at Telerik.Reporting.Paging.PageCompositionBase.CreatePages()
   at Telerik.Reporting.Paging.PagerBase.Telerik.Reporting.Paging.IPager.CreatePages(IPageHandler handler, Report report)
   at Telerik.Reporting.BaseRendering.RenderingExtensionBase.Render(Report report, Hashtable renderingContext, Hashtable deviceInfo, CreateStream createStreamCallback, EvaluateHeaderFooterExpressions evalHeaderFooterCallback)
   at Telerik.Reporting.Processing.ReportProcessor.CountPages(IList`1 reports, IRenderingContext renderingContext, Hashtable deviceInfo, ExtensionInfo extensionInfo, CreateStream createStreamCallback)
   at Telerik.Reporting.Processing.ReportProcessor.RenderCore(ExtensionInfo extensionInfo, IList`1 reports, Hashtable deviceInfo, IRenderingContext renderingContext, CreateStream createStreamCallback)
   at Telerik.Reporting.Processing.ReportProcessor.ProcessAndRender(String format, ReportSource reportSource, Hashtable deviceInfo, IRenderingContext renderingContext, CreateStream createStreamCallback)
   at Telerik.Reporting.Processing.ReportProcessor.ProcessAndRenderStateless(String format, ReportSource reportSource, Hashtable deviceInfo, IRenderingContext renderingContext, CreateStream createStreamCallback)
   at Telerik.Reporting.Processing.ReportProcessor.RenderReport(String format, ReportSource reportSource, Hashtable deviceInfo)

[Alex]

Need help!!!

[IvanY]

Hi Alex,

The issue you are having is due the fact that your report is too big and the machine that tries to generate it does not have the resources required for that. Therefore you will have to either change your report so that it is smaller using report parameters and that way display only the relevant information or you will have to upgrade your machine. For more information regarding the report parameters please check this help article.

Regards,
IvanY
Telerik

[Alex]

Thank you for your reply.

Actually, can you tell me the upper bound of total number of rows from Database and total number of A4 size report pages that Telerik reporting engine can handle? 

This is very important for us to do performance testing on large reports. Then, we could improve our report based ur answer.

Thanks & regards, 

Alex

[IvanY]

Hello Alex,

Telerik Reporting does not have any internal limit for the amount of data that can be processed. The only limitation in a case with large reports are the hardware resources.

Regards,
IvanY
Telerik

[Alex]

Hi IvanY,

First of all, we cannot assume or force any customer to upgrade their hardware when they consider to buy our products. We did some testing on our IH-housing system, which configured 8-core I7, 8 GB memory plus 500GB HD, but the program still suffer from performance issue. 

Yes, the error that we mentioned still happened and  the telerik engine eat up nearly 100% CPU resource, which extremely slowed down our system. Our product have to handle big data issue. What we do is just creating a single thread and rending a single report as pdf to local disk. The total pages could be over 5000 pages.

Please let me know how to solve it and how long it should be done ideally?

Regards,

Alex

[IvanY]

Hi Alex,

Thank you for the provided input. What you should know is that handling reports (be it with large data or large number of report items) usually requires additional resources in terms of powerful CPUs, and enough memory to handle all the operations. As far as the processing time and consumed memory is concerned, the complexity of the report layout, the complexity of the data calculations (aggregates, conditional formatting),  the rendering media, the size of the generated document (in pages), and the capabilities of the underlying machine are crucial.

In general, Telerik Reporting is an embedded reporting engine, which primary goal is to analyze data and represent it in a human-readable format that is also suitable for printing. During this process, certain complex layout and paging rules are applied, so the great flexibility offered by the reporting engine comes at a price: increased processing times and memory footprint. You should agree that dumping millions of records from a database at once hardly fits in the category of report generation. Perhaps in your specific scenario another component such as our RadGrid control might be more appropriate, moreover our RadGrid control offers several export options.

As mentioned in the previous post when you have a high number of records, you should decrease their number by applying a filter with Report Parameters in order to minimize the chances of problems when you generate large reports and of course offer ability for your user to choose desired information to be shown.

Regards,
IvanY
Telerik

[Dev]

Hardware performance is not the source of this issue. After a deep analysis, I found that the pdf rendering process has some trouble in the telerik framework. 
reportProcessor.RenderReport("PDF",.... => takes 30 seconds for 600 pages to be rendered instead of 5 seconds on Devexpress. Other formats performs normally. 
Telerik teams should review that method or all subsequents calls.
Thanks.

[Dev]

To complete my previous post :

It SEEMS that RenderReport("PDF",...) works as a pdf printer :
1- Prepare the report
2- send each page on a printer pool => takes a very long time
3- launch the print

[Alex]

Hello Dev, thanks for your reply. 

I redesigned instance life scope for report rendering function. It helps to solve memory issue. But again, the slow processing issue may be still not solved. 

For telerik team, if telerik is able to provide a button to kill report rendering instance, which will help to enhance flexibility for users to generate a large report. The button shall like the one to stop executing query at MS SQL management studio.

[Stef]

Hello everyone,

We are constantly working on improving the reporting engine and the possible rendering performance. Notice that rendering in PDF and other physical media requires additional measurement and adjustment calculations while applying the paging. The call stack in the original post clearly shows that the available memory ended while initializing the paging objects. This same error would happen with each rendering extension which produces pages. You can test if exporting to Excel, where paging is not performed and rendering happens faster, suites your requirements.

The recommended solution for large amount of data is not kill the process when the machine has reached the limit of its resources. On a trial-error base check what is the amount of data rows that can be successfully processed and warn the user on data retrieval that his selection of data is too much and will not be processed. If the report document is not going to be immediately previewed, schedule a batch operation to generate it in non-working for the user time with minimal human interaction. Thus the time for generation would not be a problem.

The basic list of possible modifications that can speed up the report generation are listed in our Performance Considerations help article. Any additional improvements can be done based on further profiling of the execution of the application. Having that said, feel free to send us your report and test data for tests at our side

Regards,
Stef
Telerik

[Martin]

Hi everybody,

A while ago (April 15 / 2013) I experienced the same issue as Alex and reported it on a ticket (ID 682814 if someone from Telerik would like to take a look). One of Telerik's representatives came to me with the same argument as IvanY saying that it I had to agree that i was not readable by a person and all that stuff (I'm starting to think they have a script or something).

Ok, I have to agree, in that case a report THAT big is not easily readable, but today I'm facing a similiar issue rendering to PDF a 4 page form designed using Telerik's standalone report designer... it takes about 2 seconds each... rendering a block of 100 reports leads to a 200 seconds delay, which crashes my IIS website to timeout.

I really like Telerik products over the competitors and I wouldn't like change, but today I'm forced to ask: do Telerik have REAL plans to improve the performance of their reporting engine? Telerik must realize that It's not just me, or 1, or 2 developers but a lot of their customers that would like to use their reporting engine to generate a bunch of reports from a Windows Service, a Website or whatever, and for any reason, their current implementation does not work for us.

I hope this message just doesn't stay in this forum post, but reach someone that can do something.

[Stef]

Hi Martin,

Performance is a main point of interest for us and our constant work involves improving it. Notice that rapidly speeding the processing and rendering is a task that cannot be done for an interval of few months as it is affecting core mechanisms as the paging.

If profiling the report does not point to anything that can be improved, try setting the connections and requests timeout to greater values to overcome the issue. The best way to check the processing and rendering of your report definitions is to run the application and check the VS Output window for the processing and rendering times.

Since in the mentioned support ticket #682814, we discussed a different report with data source consisting of 1+ million records, which does not look like the current described scenario, we will appreciate it if you open a separate support ticket with a runnable example that we can test at our side. Any additional details about the purpose of the report(s), if they are needed at the time or can be executed as a batch and provided later to the user, or if filtering of data is applicable in the scenario will be helpful to give you better suggestions in your specific case.

Regards,
Stef
Telerik

[tung ngie]

Hi Alex

Can you share on how you redesigned the life scope for report rendering function?
Is there any way to break the report into parts? or ask report renderer to process certain number of pages each time?



Thanks
tungngie

[Alex]

Hi Tung,

1. Once you finish rendering report, you must deposit relative memory. 
2. breaking pages depends on how you design the report. E.G., If you have a detail report, you can process certain number of pages each time. So set up a threshold for total number of rows included in report, repeating call the rendering functions in your manageable code. I call it report splitting. But it's hard to split summary reports.  Hope that can help you solving your issues.

Alex

[tung ngie]

Hi Alex

Thank you for your reply.
What do you mean by "deposit relative memory"?
Is "finish rendering report" meant for report viewer showing the report or the reportProcessor.RenderingReport() method returns?

Regards
tungngie

[Anil]

hi Tung,

 Is there any update, related to memory consumption problem in telerik reports, i am trying to print a report, which consists almost 500 pages, it seems to be consumping almost 3 GB of memory space. even sometimes its crashes with systemoutofmemory exception.

 is there any optimization in the code level is found? or any update is there to resolve this use case.

 waiting for your response.

 

Br,

Anil Avala.

[Anil]

hi Telerik Team,  Is there any update, related to memory consumption problem in telerik reports, i am trying to print a report, which consists almost 500 pages, it seems to be consumping almost 3 GB of memory space. even sometimes its crashes with systemoutofmemory exception. is there any optimization in the code level is found? or any update is there to resolve this use case.

 waiting for your response. 

 

Br,Anil Avala.

[Nasko]

Hello Anil,

We haven't changed the mechanism the Reporting engine uses to store objects in memory while processing and rendering a report.
At the moment you can modify the report, so it uses less memory or increase the memory of the machine where the report is processed.
The basic list of possible modifications that can speed up the report generation and lower memory consumption are listed in our Performance Considerations help article.

Regards,
Nasko
Telerik

 

 

[tung ngie]

Hi Anil

No, I still face the issue. What I have done in the end, to pop up a message box to warn the user of possible out of memory when I detected the number of items are more than certain number that it will cause the report to be generated with several pages...

The user can then choose whether to continue with it or abort the operation...

Regards
tungngie

[Dale]

We are running into a CPU usage issue when trying to export multiple reports to one PDF file. This a C# web application using telerik reporting.  The original process would create a new report and use InstanceReportSource to load the report to the reportviewer, or to export using reportprocessor.RenderReport. This works great for one report, however the user now wants to be able to view and export hundreds or even thousands of reports into one PDF. When looping though the original process the server CPU usage spikes to near 99%, slowing down the entire network. We were running into out of memory errors, but even after adjusting that though my web.config the CPU spiking is too great.  Is there a better way to create and view multiple reports and export them into one PDF file without using all our server resources? We are using Report version 9.2.15.1126. Thank you.

[Hinata]

Hi Dale 

 

Exporting thousands of reports will most likely take all of the CPU resources. You can limit the CPU usage for the application within the IIS configuration. More info here: CPU Settings for an Application Pool.