Thank you for pointing out this problem.
Incorrect caret positioning and mouse selection problems when RadRichTextBox is used in RadWindow and zoomed browser is resolved for Silverlight 5 version of our assemblies. Unfortunately, the only solution for you could be to upgrade your project to Silverlight 5. Otherwise, as a workaround you could use the zoom functionality of RadRichTextBox to zoom in/out its content instead of zooming the browser.
I hope this information helps. Please, do not hesitate to contact us if you need further assistance.
the Telerik team