I am not aware of such a interoperability problem between RadEditor and the AjaxToolkit ModalPopupExtender. The provided information is not enough to find what is causing the problem.
Please provide a sample working project that demonstrates the error along with reliable steps so that we can debug the problem and find a solution for it?
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed