You can see how to add a custom item to the context menu of RadFileExplorer in the following help article: Adding a custom button to the toolbar and an item to the tree context menu
. Once clicked the item should execute the clearSelection() method of Telerik.Web.UI.FileExplorer.FileList control.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed