I am not aware of such a problem and when I tried the setup (attached is my sample) things worked fine with me in IE, Chrome and Firefox (all the major browsers I tested) - the browser popup remained foremost. This was the behavior from the button, from the first level menu items and from the second level menu items as well.
What I can suggest is adding a timeout
around the call to window.open() OR around GetRadWindow().close() and see if this helps. Even 0ms could be enough to let the browser recalculate the active window and lead to the behavior you desire.
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed