Could you please clarify which version of the control you are using? Note that the approach you are using is for the 5.x version of RadEditor for MOSS (the ASP.NET AJAX version).
If you are using a prior version please review the MyCustomDialog.html
in the following help article - there you can review how to achieve the desired behavior:
I hope this helps.
the Telerik team