I am still not able to reproduce your issue. It is really interesting that after you added the RadUpload control on the page everything works fine. A file goes into invalid collection if it's size is bigger than the MaxFileSize property or it has different file type than allowed.
It is not possible to specify excluded file types. To achieve that you need to implement your custom server side logic.
the Telerik team