I was able to reproduce the issue. I get the same error when clicking on the Open button from the download dialog. See the recorded video: http://screencast.com/t/3AclxtSI. The .chm file opens fine if I save it first on the hard drive.
If the problem continues, check the security settings of the .chm file by right-click on the chm, select properties and click Unblock button from the General Tab as described in this article.
Also when installing/updating Test Studio you are getting the latest API Reference at the moment(the one on site might be an older version). The easiest way to find it is from the Start menu, Telerik->Testing Framework 2011.1->Documentation->Testing Framework API Reference(see screenshot).
Hope this helps!
Greetings,
Plamen
the Telerik team
Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items