RadEditor dialogs utilize RadFileExplorer to provide file browsing functionality and you can assign custom handles to it by modifying the editor's built-in dialog FileBrowser.ascx.
You can find detailed information on how to modify built-in dialogs in the following live demo:
Another approach to achieve the required result is to subclass the dialog's content provider and override its MoveFile()/MoveDirectory() methods to implement the custom logic to the rename functionality. You can find example on how to implement custom provider to the dialog in the following help article:
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed