I have just replied your support ticket on the matter and I am pasting here my findings in case they can help someone else with similar issues:
The problem stems from the AjaxControlToolkit assembly. Its mere presence in the Bin causes the site to output modified versions of the MS AJAX scripts that have issues with event handling. Once I removed the dll things started working as expected with me. Our controls rely on the MS AJAX Framework in order to function properly and having that modified can cause our controls to break.
Another thing I would generally advise against is adding event handlers like you are doing. The event handler function must be in the same context as the object that uses it. In your case the RadWindow instance is created in the main page because of your frame buster function (OpenRadWindow() in the master page). The function objects, however, exist in the content pages of each RadWindow which are essentially iframes and therefore - a separate context. This means that the event handlers must also be in the main page. They receive the RadWindow that fired them as a first argument so you can use this approach to call a function in the given content page.
Yet another thing I noticed is the doctype - I advise on using the XTML 1.0 Transtitional or a newer doctype (e.g. the HTML5) to ensure standards mode is properly invoked in the browser.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed