DotNetNuke offers their own providers for Telerik's RadEditor and they have written their file browser content provider for the file browser dialogs as well. I can tell you that It is not directly supported by Telerik. So, you will probably need to contact them for assistance.
Take a look at the DotNetNuke & RadEditor issue
forum thread. It covers a similar issue and should give you some good resources to resolve the issue.
Hope this helps!