The RadWindow control is displayed in a Popup control to make sure it is over the rest of the application, but it seems that the Popup control work very well with 3D content inside it. I searched the MS forums and found this forдаum
thread which suggests a work-around for this limitation. You could try it and check if it helps.
the Telerik team