In general clicking on the cancel button cancels the request to the server with the notable exception of uploading files in IE6/7/8/9. In those versions of IE it is not possible to cancel the upload once sent to the server.
On a side note what do you do in your custom handler so that it hangs the server? Perhaps there is a way to optimize that functionality so that it doesn't cause freezes.
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed