Unfortunately, I will have to disappoint you. RadFileExplorer does not support implementation of custom upload dialog because the controls used in the dialog are children of RadFileExplorer control and the upload functionality depend on them.
Could you please provide more detailed information on what exactly you are trying to achieve so we can think of a possible solution?
In addition, you can modify the existing Upload dialog with additional fields. Please take a look at the following forum thread for more information on the subject:
Custom Upload File Fields
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed