For your convenience I prepared and attached an example how to modify the FileBrowser dialog of the regular RadEditor for ASP.NET AJAX control.
The example is based on the following articles:
Displaying single upload control in the FileBrowser Upload manager
Adding custom columns in the RadFileExplorer's grid
Using custom FileBrowserContentProvider
The first KB article will show you how to get a reference to the RadFileExplorer control in the FileBrowser dialog from the server. Therefore you will be able to implement a FileSystemContentProvider and implement the solution in the second and third articles. All these steps are implemented in the attached project. You can see how it works in this video: http://screencast.com/t/AuUl2VpYVJ
In order to integrate this solution in SharePoint you should do the following:
1) See how to register the external dialogs of RadEditor for SharePoint: ExternalDialogsPath property
2) To be able to reference the UserControl.ascx in the FileBrowser.ascx dialog, it should be build as an assembly which is registered in the GAC (Global Assembly Cache). You should create a project which contains the UserControl.ascx control and build it. After that add the produced assembly to your GAC.
3) The final step is to add a SafeControl for this assembly in the web.config file of the SharePoint application (inetpub\wwwroot\wss\VirtualDirectories\80\web.config).
the Telerik team