This behavior would indicate that it has AutoPostBack set to true and it performs a postback when enter is pressed within it. Another option is that there is a DefaultButton set which is triggered when there isn't validation present. Please check which of these is true and in the code-behind of which of the two conrols you are showing the RadWIndow so that you can correct this behavior (e.g. with a check for validity).
Please also make sure you are opening the RadWindow from the server correclty
and not via setting the VisibleOnPageLoad property to true, as it is prone to showing a RadWindow after a postback (which is its concept and functionality, actually).
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed