The RadTreeView used in RadFileExplorer is configured to using Client-Side ExpandMode for its nodes to increase the overall performance of the control. In this configuration, the HTML for the nodes that will be displayed is returned from the server with the JSON string and this is causing the reported error to occur when trying to load folder with more than 10000 sub-folders.
At present, this behavior cannot be modified and unfortunately, I am unable to provide suitable workaround for this case.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed