I have tried to use this standard code against a public available modal popup and I was not able to reproduce your issue. The test executes in Firefox and Chrome as expected. One thing you can do is to increase the timeout of the connect to the popup window step and give it a try.
The other way we can proceed is if you record a sample test against your application containing a few steps and provide it along with access to your application so we can review this issue further.
Please note that now we are communicating in a forum thread which is public available and if you are willing to provide any sensitive information you should open a support ticket where all the information remains private and confidential.
All the best,
the Telerik team