From the screen shots, which you provided I can tell that you are using the OOB link manager dialog, which is using the sharepoint's asset picker dialog. Unfortunately RadEditor can not control this dialog. What I suggest you to try is is using the Telerik provided LinkManager dialog. In order to do this change MOSSLinkManager
in the ToolsFile.xml and ListToolsFile.xml, which are located in the /Program Files/Common Files/Microsoft Shared/web server extensions/wpresources/RadEditorSharePoint/5.x.x.0__1f131a624888eeed/Resources/ folder.
the Telerik team
Check out Telerik Trainer
, the state of the art learning tool for Telerik products.