The RadWindow control is basically an IFRAME and it behaves just like one. This being said, if your implementation logic is working fine with an IFRAME or a standard browser's popup (window.open()), the same approach will work with RadWindow as well.
If you decide to implement your logic with RadWindow, I believe that the following documentation topic will be of help:
Create an AJAX-enabled Sharepoint WebPart that uses RadControls
Incorporating RadControls for ASP.NET Ajax in MOSS WebSites
the Telerik team