This problem is not directly related to RadFileExplorer. When an attempt to open file occurs in the explorer the file is downloaded by the IIS without any interventions by the control. Could you please verify that you have the correct permissions set to the files and the IIS is configured correctly?
More detailed information regarding Widnows Authentication is available in the following article:
the Telerik team