The easiest way to reset the RadControls folder is to retract the editor solution, delete the leftover files from C:\Program Files\Common Files\Microsoft Shared\web server extensions\wpresources\RadEditorSharePoint\ and finally deploy the editor solution again. Do not forget to save any modifications you made before you retract the solution.
Have you made any modifications to the IIS configuration? In order to work properly, the editor dialogs need to be in the same web application as the editor itself. That is why we put the dialogs in the _wpresources folder, which is configured as a virtual directory without its own application under the SharePoint web app.
the Telerik team