maybe you can try testing your PDF pages with regular navigation steps instead of treating them as pop-ups?
For instance the following step:
"Navigate to 'http://www.opm.gov/hcaaf_resource_center/assets/Lead_tool3.pdf'"
will work in Internet Explorer. Once you've logged on the page you can write a Verification steps checking whether an "embed" tag with the desired properties is present on the page. You can add the embed tag to your project by opening the Dom Explorer for the page and locating the embed tag there. You can't access the embed tag from the page otherwise - the PDF file takes up the entire page and WebUI Test Studio has no way of interacting with PDF.
Also one more thing: WebUI Test Studio will regard a PDF page as immediately ready to use - it won't wait for the PDF to load because PDF files are meant to be loaded asynchronously. If you want to wait for the PDF to load you'll need to delay execution yourself (i.e. Delay Execution step).
Once you're done with the PDF page you can simply navigate back from it.
Let me know whether this solution works for you!
All the best,
the Telerik team