In order to help you further I will need some additional information. Provide me with the web.config file of your SharePoint site and the configuration files of RadEditor for SharePoint, which are located in the /Program Files/Common Files/Microsoft Shared/web server extensions/wpresources/RadEditorSharePoint/6.x.x.0__1f131a624888eeed/Resources/ folder.
In general editor's dialogs should work with https without problems.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed