The scenario that you have - to switch between different tabs in the browser, is not supported out of the box in our product. This is why I created a public feature request on your behalf and our engineering team will investigate it further. Then, we can plan it for figure releases, so please vote and follow the feature request for more details.
I did some tests on my end and found a solution for you with a coded step. The idea is to remove the new window (the pop-up) from the collection of active browsers, so that Test Studio can change its focus to the previous tab. I also added some keyboard action that visually switch between the tabs in the browser and ensure that desktop click and simulated user actions will work as expected.
Please explore the sample test against one of our documentation pages and analyze the code and my comments there. I remain available to discuss this further and help you complete this test scenario.
P.S. As a small token of gratitude for bringing this missing feature to our knowledge, I have updated the Telerik points in your account. Thank you once again.