Possible race condition in menu security trimming

4 posts, 0 answers
  1. Cole
    Cole avatar
    2 posts
    Member since:
    Nov 2015

    Posted 13 Sep 2017 Link to this post

    We're dealing with a strange error that appears to be a race condition while the kendo menu is checking permissions to "trim" the menu items. The error message we've logged is: System.IndexOutOfRangeExceptionIndex was outside the bounds of the array (see stack trace below).

    This seems to be a race condition as it happens infrequently, and apparently only at times of heavy use. We also can't reproduce it in any of our test or development environments; we see it logged only in production.

    I've downloaded the kendo source bundle, but it (rather surprisingly) doesn't contain any C# code! So we can't actually get any insight into what might be causing this error to be thrown by the MVC menu wrapper.

    My guess is that the kendo menu is parallelizing the requests to check action authorization, and in rare cases this causes a race condition on the collection access (again, see stack trace) which is not thread-safe.

    Can you provide any guidance on this issue?

    Thanks!

     

    Stack Trace:

    System.Web.HttpException (0x80004005): Error executing child request for handler 'System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper'. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.

       at System.Collections.ArrayList.Add(Object value)
       at Kendo.Mvc.Infrastructure.Implementation.ControllerAuthorization.IsAccessibleToUser(RequestContext requestContext, String controllerName, String actionName, RouteValueDictionary routeValues)
       at Kendo.Mvc.Infrastructure.Implementation.NavigationItemAuthorization.IsAccessibleToUser(RequestContext requestContext, INavigatable navigationItem)
       at Kendo.Mvc.UI.NavigationItemContainerExtensions.WriteItem[TComponent,TItem](TItem item, TComponent component, IHtmlNode parentTag, INavigationComponentHtmlBuilder`1 builder)
       at Kendo.Mvc.UI.NavigationItemContainerExtensions.<>c__DisplayClass5`2.<WriteItem>b__1(TItem child)
       at Kendo.Mvc.Extensions.EnumerableExtensions.Each[T](IEnumerable`1 instance, Action`1 action)
       at Kendo.Mvc.UI.NavigationItemContainerExtensions.WriteItem[TComponent,TItem](TItem item, TComponent component, IHtmlNode parentTag, INavigationComponentHtmlBuilder`1 builder)
       at Kendo.Mvc.UI.Menu.<>c__DisplayClass1.<WriteHtml>b__0(MenuItem item)
       at Kendo.Mvc.Extensions.EnumerableExtensions.Each[T](IEnumerable`1 instance, Action`1 action)
       at Kendo.Mvc.UI.Menu.WriteHtml(HtmlTextWriter writer)
       at Kendo.Mvc.UI.WidgetBase.ToHtmlString()
       at Kendo.Mvc.UI.Fluent.WidgetBuilderBase`2.ToHtmlString()
       at System.Web.WebPages.WebPageBase.Write(Object value)
       at ASP._Page_Views_Menu_Application_cshtml.Execute() in d:\WWWRoot\SASB\Views\Menu\Application.cshtml:line 14
       at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
       at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
       at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
       at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
       at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)
       at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
       at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState)
       at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
       at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
       at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
       at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
       at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)
       at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
       at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
       at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.<>c__DisplayClass4.<Wrap>b__3()
       at System.Web.Mvc.HttpHandlerUtil.ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func)
       at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride)
       at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride)
       at System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage)
       at System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm)
       at System.Web.Mvc.Html.ChildActionExtensions.ActionHelper(HtmlHelper htmlHelper, String actionName, String controllerName, RouteValueDictionary routeValues, TextWriter textWriter)
       at System.Web.Mvc.Html.ChildActionExtensions.Action(HtmlHelper htmlHelper, String actionName, String controllerName, RouteValueDictionary routeValues)
       at ASP._Page_Views_Shared__ApplicationLayout_cshtml.Execute() in d:\WWWRoot\SASB\Views\Shared\_ApplicationLayout.cshtml:line 9
       at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
       at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
       at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
       at System.Web.WebPages.WebPageBase.<>c__DisplayClass3.<RenderPageCore>b__2(TextWriter writer)
       at System.Web.WebPages.WebPageBase.Write(HelperResult result)
       at System.Web.WebPages.WebPageBase.RenderSurrounding(String partialViewName, Action`1 body)
       at System.Web.WebPages.WebPageBase.PopContext()
       at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
       at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
       at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)

  2. Veselin Tsvetanov
    Admin
    Veselin Tsvetanov avatar
    747 posts

    Posted 14 Sep 2017 Link to this post

    Hello Cole,

    I have answered to your question in the support ticket, that you have opened on the same topic. I would suggest you to continue our communication in one of the two threads (the one you find more appropriate).

    As other developers may be interested in the discussion, I am attaching my answer below:

    The Kendo.Mvc.dll source can be downloaded from the respective page on your Telerik profile. It is available in the two archives on the bottom of the page:
    - telerik.ui.for.aspnetmvc.2017.3.913.commercial-source.zip; or
    - telerik.ui.for.aspnetmvc.2017.3.913.commercial-source.7z;

    The source-code project is located in the src\Kendo.Mvc folder.

    Concerning the issue described, as you correctly noticed, it can be cause by the non-thread safe Security trimming that the Kendo MVC Menu performs. Here is a Feedback portal item suggesting Thread safe implementation for this functionality of the Menu, which you could vote for.

    Concerning the details of the error and the specific cause, note that we have not observed such an issue before. In order to be able to troubleshoot it, we will need to be able to locally reproduce it.

    Regards,
    Veselin Tsvetanov
    Progress Telerik
    Try our brand new, jQuery-free Angular 2 components built from ground-up which deliver the business app essential building blocks - a grid component, data visualization (charts) and form elements.
  3. Cole
    Cole avatar
    2 posts
    Member since:
    Nov 2015

    Posted 14 Sep 2017 in reply to Veselin Tsvetanov Link to this post

    Thanks for the link to the source, though it doesn't seem to match our stack trace exactly. I'm guessing there were some minor changes in the most recent version.

    I can't wrap my head around how this is a multi-threading issue, as there doesn't seem to be anything spawning threads in your code, and I can't understand how two requests would end up sharing the same object instance, but maybe Kendo is using a singleton somewhere or reusing an object from a DI container or something? I can't tell.

    I've voted on the fix request, as you've suggested (that's a slightly different issue, which we've also seen in production on our busy application).

    I don't have a working example to reproduce the issue (it's non-deterministic), but I can give you a few pointers from what we're seeing:

    - We've only seen this issue in production, on our busiest application. All the smaller applications seem to be unaffected so far.

    - The menus that we render in the affected application are fairly large, with a lot of conditional logic to display different things based on state, or hide things from users who can't access them, etc.

    - The exception only happens rarely, but seems to be at busier times (makes sense, if it's a race condition). We had two users get the same exception on different requests with the same timestamp, but haven't seen it any other times.

    My advice would be to create a website that has a fairly complex menu, then hammer it with traffic from a load testing tool. You should be able to reproduce the issue, though it will be sporadic.

  4. Veselin Tsvetanov
    Admin
    Veselin Tsvetanov avatar
    747 posts

    Posted 18 Sep 2017 Link to this post

    Hi Cole,

    I am afraid, that we were not able to reproduce and isolate the cause for the issue after a careful review of the Security trimming implementation.
     
    Your observation about the multi-threading are absolute right. We do not depend on any parallelization to determine whether a link is authorized to be rendered or not. Indeed, we do instantiate our own instances of authorization attribute classes, but they will just execute synchronously. 

    For your information the internal details could be found in the ControllerAuthorization class of the Kendo.Mvc.Infrastructure.Implementation namespace. You will notice, that the IsAccessibleToUser() method implementation iterates over the retrieved authorization attributes and sets the state for each Controller / Action name pair.
     
    The best we can offer is to tweak the security trimming code and see whether the issue still persists. You can even try to disable the security trimming logic and spin your own, which could be monitored better and modified easier. Any findings that could shed more light on this issue will help us immensely to narrow the cause down and introduce an enhancement on our side.

    Regards,
    Veselin Tsvetanov
    Progress Telerik
    Try our brand new, jQuery-free Angular 2 components built from ground-up which deliver the business app essential building blocks - a grid component, data visualization (charts) and form elements.
Back to Top