When RadAsyncUpload is using Silverlight/Flash module the filtering functionality is available. On other hand when using the IFrame/FileApi module it's not - browser limitations. If the user select a file with different extensions that the allowed ones, this file won't be uploaded. It will appear as invalid and a red dot will be shown in front of it.
All the best,
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed