By default, all RadEditor dialogs are embedded as resource fils in the
Telerik.Web.UI.dll file, which makes their customization harder. In order to
provide an easy way to modify the editor's dialogs, the RadControls installation
now comes with an extra folder named EditorDialogs
contains the dialogs' ascx files. You can configure RadEditor to use the
external dialog files by copying the EditorDialogs
the root of the web application and setting the
property to point to it.
You can find additional information in http://www.telerik.com/help/aspnet-ajax/customizingimagedialog.html
Also you should know that in order to modify the ExternalDialoguesPath property you need to modify the respective ConfigFile.xml located in C:\Program Files\Common Files\Microsoft Shared\web server extensions\wpresources\RadEditorSharePoint\220.127.116.11__1f131a624888eeed\Resources folder (18.104.22.168__1f131a624888eeed depends on the control version).
In addition you should know that there is an option of using the Image and Document managers outside RadEditor. You can find more information in the following KB article http://www.telerik.com/support/kb/aspnet-ajax/editor/using-the-image-and-document-managers-outside-radeditor.aspx
the Telerik team