I am confused by your statement "a popup opens in the Iframe".
Typically we see HTML or Modal popup windows when recording (which can contain frames) that are launched from the parent page, or a frame containing separate content on the parent page. All of these scenarios are handled by Test Studio and should not require additional configuration unless the popup URLs or frame IDs are dynamic.
If you are indeed trying to identify elements through a frame, you must ensure the frame properties are properly set. For example, if the frame has a dynamic ID, you should strip the dynamic portion and insert a tilde (~) before the static portion in the ID field to indicate a partial match.
Of course, this assessment is mostly speculation without more clarification from you. Ideally we'd like to see the issue first-hand directly in your app. If it is publicly accessible, please provide us the means to test and reproduce the issue. If not, then the Step Failure Details and a Jing video (a free screen recorder) are great starts.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items