Would you try to isolate the problem and send us a project that exactly shows the problem, because it's quite hard to reproduce your issues without any information. Transparency of the header can be controlled by editing the control template of RadWindow in blend and introducing your own definition of it. We still don't have any support for nested RadWindows and such type of modality that your application requires. Excuse us for the late response and feel free to contact us for any further assistance.
the Telerik team
Check out Telerik Trainer
, the state of the art learning tool for Telerik products.