This is a browser specific behavior and we cannot control it. When a image is bigger than the size of an IFRAME (RadWindow is based on an IFRAME), then scrollbars will be showed in IE. The FF browsers, however, automatically scale the content (you can see this in the title of the RadWindow). In FF when an image is scaled and when the mouse is over the image a "+" sign appears and if you click the mouse's left button, then the image will be stretched to its full size. Please note that this behavior occurs when an standard IFRAME is used as well (shown in the attached project) and it is not related to RadControls.
A solution of this behavior can be to set the size of the Preview dialog to be greater than the size of the images. This approach is shown in the attached attached demo.
I hope this helps.
the Telerik team