This is a DirectX
What you try is to configure RadEditor dialogs to use the browsers dialogs as it is shown in this KB article: Using browser modal dialog instead of RadWindow
You can see how to attach to the events of RadEditor for MOSS in this KB article: http://www.telerik.com/support/kb/aspnet-ajax/sharepoint/how-to-execute-custom-code-called-on-radeditor-s-events.aspx You should edit the following file when working with RadEditor for SharePoint 2010 C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\wpresources\RadEditorSharePoint\188.8.131.52__1f131a624888eeed\Resources\SPEditorTools.js
All the best,
the Telerik team
Register for the Q2 2011 What's New Webinar Week. Mark your calendar for the week starting July 18th and book your seat for a walk through of all the exciting stuff we will ship with the new release!