Telerik Reporting Service Loses DB Connectivity Intermittently

[Dan]

Hi everyone. 

 

We have an extremely frustrating problem we are attempting to solve.

We have a .NET 7 Web Application using a Blazor Server hosting model for reporting. The application is containerized and runs as part of a service via AWS Fargate. We have a series of legacy reports that we have converted to '.trdp' format, several of these reports use a named connection string to interact with RDS (AWS SQL DB).

 

We are using the 'Blazor' reporting solution that is merely a wrapper over the HTML5 reporting. The assembly versions are as follows:

Telerik.Drawing.Skia, 17.2.23.1114 
Telerik.Reporting, 17.2.23.1114 
Telerik.Reporting.OpenXmlRendering, 17.2.23.1114 
Telerik.Reporting.Services.AspNetCore, 17.2.23.1114
Telerik.ReportViewer.Blazor, 17.2.23.1114

As part of our build process - we inject a secrets file into the container definition with the credentials for accessing the db. This is modified during launch to include the correct environment db and db credentials:

if (ConnectionUtility.IsRunningInFargate())
{
    // Create new copy of string in memory so we can modify it
    var connectionString = connectionStrings[Constants.ReportingSqlConnectionKey];
    var databaseUriForEnvironment = ConnectionUtility.GetDatabaseUriForEnvironment();
    var databaseForEnvironment = ConnectionUtility.GetDatabaseForEnvironment();

    // Replace placeholders with values
    var protoConnectionString =
        connectionString!
            .Replace("{0}", databaseUriForEnvironment)
            .Replace("{1}", databaseForEnvironment);

    // Important! - This replaces the 'in-memory' connection string that reports use to interact with the primary db
    connectionStrings[Constants.ReportingSqlConnectionKey] = protoConnectionString.TransformSecret(Constants.SettingClass.ConnectionString);
}

The connection string key isn't important here, but it is used to provide the connection key for each report.; 

The Telerik services are injected:

builder.Services.TryAddSingleton<IReportServiceConfiguration>(sp => new ReportServiceConfiguration
{
    ReportingEngineConfiguration = sp.GetService<IConfiguration>(),
    HostAppId = "X.Reporting",
    Storage = new FileStorage(),
    ExceptionsVerbosity = "detailed",
    ReportSourceResolver =
        new CustomReportSourceResolverWithFallBack(
            new TypeReportSourceResolver()
                .AddFallbackResolver(
                    new UriReportSourceResolver(
                        Path.Combine(
                            GetReportsDir(sp))))),
});

 

The resolver is less important, but for the sake of completion:

using Telerik.Reporting;
using Telerik.Reporting.Services;

namespace X.Reporting.Services
{
    public class CustomReportSourceResolverWithFallBack : IReportSourceResolver
    {
        private readonly IReportSourceResolver? _parentResolver;

        public CustomReportSourceResolverWithFallBack(IReportSourceResolver? parentResolver)
        {
            _parentResolver = parentResolver;
        }

        public ReportSource Resolve(string report, OperationOrigin operationOrigin, IDictionary<string, object> currentParameterValues)
        {
            var reportDocument = ResolveCustomReportSource(report, operationOrigin, currentParameterValues);

            if (null == reportDocument && null != _parentResolver)
            {
                reportDocument = _parentResolver.Resolve(report, operationOrigin, currentParameterValues);
            }

            return reportDocument;
        }


        private ReportSource ResolveCustomReportSource(string reportId, OperationOrigin operationOrigin, IDictionary<string, object> currentParameterValues)
        {
            var reportBook = new ReportBook();
            var splitString = reportId.Split(',').ToList();

            foreach (var report in splitString)
            {
                var uriReportSource = new UriReportSource
                {
                    Uri = $"Reports/{report}.trdp"
                };

                reportBook.ReportSources.Add(uriReportSource);

                if (operationOrigin == OperationOrigin.ResolveReportParameters)
                {
                    reportBook.ReportSources.Add(GetReportParameters(report, currentParameterValues));
                }
            }

            return new InstanceReportSource { ReportDocument = reportBook };
        }

        private ReportSource GetReportParameters(string reportId, IDictionary<string, object> currentParameterValues)
        {
            UriReportSource report = new UriReportSource();

            foreach (var parameterValue in currentParameterValues)
            {
                if (parameterValue.Value is not null and not (string)"standardReport")
                {
                    var par = new Parameter
                    {
                        Name = parameterValue.Key,
                        Value = parameterValue.Value.ToString()
                    };

                    report.Parameters.Add(par);
                    report.Uri = $"Reports/{reportId}.trdp";
                }
            }

            return report;
        }
    }
}

 

90 % of the time, we can render reports without issue. However, there are times where any report that uses a SQL connection seems to be unable to reach our primary database. EF Core (which interacts with the same DB prior to and after rendering reports) seems to never have this issue - even when Telerik does. Both share the exact same connection string. Rendering a report with an included SQL query returns something like the following: 

On occasion, this issue seems to resolve itself after ~5 minutes. Most of time, we need to kill the container instance and restart it a number of times before reports render without issue. I'm confounded why this happens - it 'feels' like the underlying SQL connection is reused.

Any help is appreciated - this has been a critical issue to our clients, we can't keep using this reporting solution without a fix.

[Todor]

Hello Dan,

Thank you for the detailed description of the issue, the code snippets, and the image.

The code seems fine to me and the issue looks related to resolving the connection string. Generally, our code takes it from the IConfiguration when the Reporting REST Service is configured. Importantly, this happens only once, the first time the Reporting REST Service needs configuration details. That said, if you change the connection string without restarting the application, this won't take effect for the Reporting part of your application.

If this is the case, since you already use a Custom Report Source Resolver, you may change the connection strings dynamically in the Resolve method - check Edit the Report Data Source Components at Runtime.

If you don't change the connection string while the Reporting REST Service is running, I cannot explain the problem without additional details. You may attach a Trace Listener to the REST Service to check for the stack trace of the error. I hope it will give more information about the issue.

Regards,
Todor
Progress Telerik

[Dan]

@Todor 

We don't perform any additional manipulation of the connection string after initial configuration - in theory, because it is resolved from IConfiguration during startup, it should be ok. 

I will add tracing and see if that provides us with more insight. Thanks for your recommendation. If we can reproduce the issue, I'll reply back with the stack trace / log.