We're dealing with a strange error that appears to be a race condition while the kendo menu is checking permissions to "trim" the menu items. The error message we've logged is: System.IndexOutOfRangeExceptionIndex was outside the bounds of the array (see stack trace below).
This seems to be a race condition as it happens infrequently, and apparently only at times of heavy use. We also can't reproduce it in any of our test or development environments; we see it logged only in production.
I've downloaded the kendo source bundle, but it (rather surprisingly) doesn't contain any C# code! So we can't actually get any insight into what might be causing this error to be thrown by the MVC menu wrapper.
My guess is that the kendo menu is parallelizing the requests to check action authorization, and in rare cases this causes a race condition on the collection access (again, see stack trace) which is not thread-safe.
Can you provide any guidance on this issue?
Thanks!
Stack Trace:
System.Web.HttpException (0x80004005): Error executing child request for handler 'System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper'. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
at System.Collections.ArrayList.Add(Object value)
at Kendo.Mvc.Infrastructure.Implementation.ControllerAuthorization.IsAccessibleToUser(RequestContext requestContext, String controllerName, String actionName, RouteValueDictionary routeValues)
at Kendo.Mvc.Infrastructure.Implementation.NavigationItemAuthorization.IsAccessibleToUser(RequestContext requestContext, INavigatable navigationItem)
at Kendo.Mvc.UI.NavigationItemContainerExtensions.WriteItem[TComponent,TItem](TItem item, TComponent component, IHtmlNode parentTag, INavigationComponentHtmlBuilder`1 builder)
at Kendo.Mvc.UI.NavigationItemContainerExtensions.<>c__DisplayClass5`2.<WriteItem>b__1(TItem child)
at Kendo.Mvc.Extensions.EnumerableExtensions.Each[T](IEnumerable`1 instance, Action`1 action)
at Kendo.Mvc.UI.NavigationItemContainerExtensions.WriteItem[TComponent,TItem](TItem item, TComponent component, IHtmlNode parentTag, INavigationComponentHtmlBuilder`1 builder)
at Kendo.Mvc.UI.Menu.<>c__DisplayClass1.<WriteHtml>b__0(MenuItem item)
at Kendo.Mvc.Extensions.EnumerableExtensions.Each[T](IEnumerable`1 instance, Action`1 action)
at Kendo.Mvc.UI.Menu.WriteHtml(HtmlTextWriter writer)
at Kendo.Mvc.UI.WidgetBase.ToHtmlString()
at Kendo.Mvc.UI.Fluent.WidgetBuilderBase`2.ToHtmlString()
at System.Web.WebPages.WebPageBase.Write(Object value)
at ASP._Page_Views_Menu_Application_cshtml.Execute() in d:\WWWRoot\SASB\Views\Menu\Application.cshtml:line 14
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.<>c__DisplayClass4.<Wrap>b__3()
at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func)
at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride)
at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride)
at System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage)
at System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm)
at System.Web.Mvc.Html.ChildActionExtensions.ActionHelper(HtmlHelper htmlHelper, String actionName, String controllerName, RouteValueDictionary routeValues, TextWriter textWriter)
at System.Web.Mvc.Html.ChildActionExtensions.Action(HtmlHelper htmlHelper, String actionName, String controllerName, RouteValueDictionary routeValues)
at ASP._Page_Views_Shared__ApplicationLayout_cshtml.Execute() in d:\WWWRoot\SASB\Views\Shared\_ApplicationLayout.cshtml:line 9
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
at System.Web.WebPages.WebPageBase.<>c__DisplayClass3.<RenderPageCore>b__2(TextWriter writer)
at System.Web.WebPages.WebPageBase.Write(HelperResult result)
at System.Web.WebPages.WebPageBase.RenderSurrounding(String partialViewName, Action`1 body)
at System.Web.WebPages.WebPageBase.PopContext()
at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)
8 Answers, 1 is accepted
I have answered to your question in the support ticket, that you have opened on the same topic. I would suggest you to continue our communication in one of the two threads (the one you find more appropriate).
As other developers may be interested in the discussion, I am attaching my answer below:
The Kendo.Mvc.dll source can be downloaded from the respective page on your Telerik profile. It is available in the two archives on the bottom of the page:
- telerik.ui.for.aspnetmvc.2017.3.913.commercial-source.zip; or
- telerik.ui.for.aspnetmvc.2017.3.913.commercial-source.7z;
The source-code project is located in the src\Kendo.Mvc folder.
Concerning the issue described, as you correctly noticed, it can be cause by the non-thread safe Security trimming that the Kendo MVC Menu performs. Here is a Feedback portal item suggesting Thread safe implementation for this functionality of the Menu, which you could vote for.
Concerning the details of the error and the specific cause, note that we have not observed such an issue before. In order to be able to troubleshoot it, we will need to be able to locally reproduce it.
Regards,
Veselin Tsvetanov
Progress Telerik
Rank 1
Thanks for the link to the source, though it doesn't seem to match our stack trace exactly. I'm guessing there were some minor changes in the most recent version.
I can't wrap my head around how this is a multi-threading issue, as there doesn't seem to be anything spawning threads in your code, and I can't understand how two requests would end up sharing the same object instance, but maybe Kendo is using a singleton somewhere or reusing an object from a DI container or something? I can't tell.
I've voted on the fix request, as you've suggested (that's a slightly different issue, which we've also seen in production on our busy application).
I don't have a working example to reproduce the issue (it's non-deterministic), but I can give you a few pointers from what we're seeing:
- We've only seen this issue in production, on our busiest application. All the smaller applications seem to be unaffected so far.
- The menus that we render in the affected application are fairly large, with a lot of conditional logic to display different things based on state, or hide things from users who can't access them, etc.
- The exception only happens rarely, but seems to be at busier times (makes sense, if it's a race condition). We had two users get the same exception on different requests with the same timestamp, but haven't seen it any other times.
My advice would be to create a website that has a fairly complex menu, then hammer it with traffic from a load testing tool. You should be able to reproduce the issue, though it will be sporadic.
I am afraid, that we were not able to reproduce and isolate the cause for the issue after a careful review of the Security trimming implementation.
Your observation about the multi-threading are absolute right. We do not depend on any parallelization to determine whether a link is authorized to be rendered or not. Indeed, we do instantiate our own instances of authorization attribute classes, but they will just execute synchronously.
For your information the internal details could be found in the ControllerAuthorization class of the Kendo.Mvc.Infrastructure.Implementation namespace. You will notice, that the IsAccessibleToUser() method implementation iterates over the retrieved authorization attributes and sets the state for each Controller / Action name pair.
The best we can offer is to tweak the security trimming code and see whether the issue still persists. You can even try to disable the security trimming logic and spin your own, which could be monitored better and modified easier. Any findings that could shed more light on this issue will help us immensely to narrow the cause down and introduce an enhancement on our side.
Regards,
Veselin Tsvetanov
Progress Telerik
Rank 2
We are seeing this error occur rarely in our production environment as well. I was able to reproduce it in a test environment by load-testing with 50 simulated users all simultaneously hitting the same page a handful of times. This particular page has around 16 tabs on it. Even then, it only happens about once every few hundred requests.
I was able to hit the exception with WinDbg attached but am not advanced enough to troubleshoot the root cause. Probably not worth too much effort as rarely as it seems to occur.
Thank you for your input on the discussed.
I would like to ask you for a bit more details about the test environment in question. May I ask you to share with us the test project that you have used? Which tool do you use to simulate the heavy traffic? What are the steps required to reproduce the issue in question? Do you simply load the same page by multiple users numerous times?
Regards,
Veselin Tsvetanov
Progress Telerik
Rank 2
Veselin,
It was a staging environment, containing our production code, not test code.
I tried to create a test project to simulate, and it gets a similar race condition, but an Object reference null error instead on AuthorizeCore. Likely this should highlight the fundamental issue, which I think is in a shared caching layer that is not thread-safe.
I have attached sample project with a 20-tab strip and Authorize Attribute to simulate the issue. The dll reference will have to be fixed, I removed because it took over the size limit. The following C# code (in LINQPad) can simulate load to repro the issue. Note that this requires a fairly high-end performance environment and probably Release build before the race condition will reproduce reliably.
System.Net.ServicePointManager.DefaultConnectionLimit = 32;
Enumerable.Range(0, 200).AsParallel().ForAll(async c =>
{
var client = new WebClient();
var result = await client.DownloadStringTaskAsync("http://localhost:50021/Home");
});
Regards,
Kurt
Thank you for the sample prepared and for the additional information provided. We will carefully review and test the scenario in question. I will come back to you, as soon as I know something more on that matter.
Regards,
Veselin Tsvetanov
Progress Telerik
Thanks to your sample we were able to reproduce the issue locally. Here you could find a GitHub item on that bug. When our priority queue allows that, we will further investigate the case in question.
As a small token of gratitude for helping us in isolating the issue in question, I have updated your Telerik points. I have also updated Cole's points as he initially reported that problem.
Regards,
Veselin Tsvetanov
Progress Telerik