The described issue is a browser limitation. The browser cannot support such level of nesting. If you get the rendered html of the RadTreeView and put it on a sample page and run it, you will notice the same behavior.
All the best,
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed