You should understand that RadEditor is primarily designed to be an HTML Editor and not for editing XML. RadEditor can really be thought of as an advanced <asp:TextBox> and that it's exported content is a standard string.
Since your XSL content is not HTML, once it gets loaded in the iframe content area, the rich text engine of the browser and RadEditor's filters will modify it, trying to validate it using HTML rules. RadEditor's filters, by default, will try to convert it to XHTML.
You can disable RadEditor's built-in content filters and the editor will leave the browser to change the content:
Then, it is up to you to handle the exported content and write it in the required way in the desired location.
Hope this helps!