I have tested the ScreenTip.Icon property with an image 48x48 and the Icon inside the screen tip was displayed with correct dimensions. Could you check the size of the icons you are applying.
I have attached a sample project containing image 48x48 applied to the ScreenTip.Icon of a RadRibbonGroup, RadRibbonTab and a RadRibbonButton. I also attached a screenshot showing how the ScreenTip looks on our side with Q1 2013 binaries.
the Telerik team