Thank you for bringing our attention to this issue. I have logged it to our database and we will do our best to handle it for one of the upcoming releases.
To apply a custom CSS to the RadEditor's dialogs you need to set the editors DialogsCssFile property to point to a file that contains the custom CSS. More information regarding this property is available in the following help article:
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed