When a PDF document contains big images it is completely normal the memory consumption to be large. Note that RadPdfViewer requires an array of pixels in order to do some preprocessing of an image. This means that it cannot use the encoded data, but needs to decode it in order to add some corrections. Also, it will need a ready ImageSource instance in order to draw it when necessary. Basically, one large image consumes about 60mb which is why memory consumption is high.
Moreover, Silverlight has its own issues when it comes to big images as well (Out of Memory exceptions are thrown sometimes, when the images are large).
We will continue trying to reduce the memory consumption but at this point there is not much we can do.