Placing RadPdfViewer in a ScrollViewer does have that side effect. This is so because the ScrollViewer handles the Key events and they do not reach the PdfViewer. You can verify that by placing a Grid or some other control in a ScrollViewer and handling its KeyDown event.
Overall, it is not recommended to use RadPdfViewer inside ScrollViewer (or any container that will measure it in Infinite size) for a variety of reasons. This not only causes performance issues, because RadPdfViewer cannot use virtualization when showing a larger document, but also disables functionality such as go to page, find text, etc.
the Telerik team