I found that two steps are causing the odd "out-of-browser" clicks:
261. Click 'ContentPlaceHolder1UcEditPQBBtnExportImage'
262. Handle 'Download' dialog.
When I disabled these steps, the subsequent clicks worked as expected and the test continued successfully. I am unclear why the download handling caused this issue, but I have passed it on to our developers to gain further insight.
On a side note, a test of this size (450 steps) is extremely difficult to work with and to maintain. It took me a considerable amount of time just to trim it down to a workable size where I could reproduce the issue quickly and reliably. As part of best practices, we highly recommend modularizing your project into smaller tests, and executing those tests as part of a test list. Testing an entire section of your application in one test is inherently fragile and difficult to maintain, as I'm sure you already realize.
We also recommend using Execution Delays and the "Continue on Failure" feature sparingly. Each action has a built-in "Wait on elements timeout", so an extra delay is often unnecessary (find the global setting under Quick Execution
Also, I never saw errors on the "Connect to pop-up window" steps. I'm on version 2011.1.729 and using IE9.
the Telerik team