Generally it is best if the XHTML 1.0 Transitional doctype is used for both the main and the content pages. If you are overriding some of the CSS of the RadControls it is possible that this is the reason for this layout issue, yet I cannot say anything more accurate at this point. In case using quirks mode works for you you can keep using it like that, but keep an eye out for other issues this may produce, as this rendering mode is full of bugs since it actually resembles extremely old browsers like IE6 and even IE 5.5. What I can advise is that you enable the correct doctype and try to find the CSS rule from your stylesheets that causes the issue, but whether you do this is decision only you can make.
All the best,
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed