I am not aware of such a problem with RadEditor in a modal popup extender. Please, test the attached project editorinmodalpopupextender.zip
to my post in the following forum thread Link Manager Dialog with FireFox
and let me know if you are able to reproduce the same problem.
If the issue is specific to your project only, isolate it and send it for examination via a support ticket.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed