I already answered your support ticket on the subject. For convenience I will paste my answer here as well.
The most possible reason for this error to be displayed is a server error that occurs during an AJAX call. In order to attach the RadFileExplorer to a database you need to implement custom filebrowser content provider. An example of such content provider is available in the following code library:
Connect RadFileExplorer or RadEditor to a SQL databse
If you have implemented your custom provider we will need to examine it and debug it locally to provide more-to-the-point-answer. Could you please provide a sample runnable project reproducing the problem?
the Telerik team
Registration for Q1 2011 What’s New Webinar Week is now open. Mark your calendar for the week starting March 21st and book your seat
for a walk through all the exciting stuff we ship with the new release!