Currently, it is not possible to open a custom RadEditor dialog
from a built-in editor dialog.
Nevertheless, you can try to achieve your scenario by adding a button to the link manager that will fire the browser's showModalDialog
method and will open a custom browser dialog. This method also allows you to
return value to the parent window.
You can find information about the showModalDialog method in the following articles:
the Telerik team