Currently I can only make only assumptions on why this problem occurs. This behavior might originates due to custom stylization, customized layout etc.
Nevertheless, in the latest release of RadControls (2013 Q3) the RadEditor's dialogs are greatly improved, especially their layout. I suggest trying to upgrade the version and see if that resolves the problem.
would help me find a possible solution for this case.
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed