In such scenarios we recommend using OnClientFileUploadFailed event, check the error message you are getting and do your custom functionality after that.
I have tested the scenario described with DisablePlugins with .xlsx file opened in the same time in Excel by could not achieve any error. Here is a video of my test and I am attaching a sample web page. Please review them and let me know what else should be made in order to observe the same error and be more helpful?
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed