There is no way of restricting the TargetFrame to NewWindow only. You have two options.
You can handle the HyperlinkClicked event of RadRichTextBox and "manually" open all Hyperlinks in another window, regardless of their target.
A better approach, however, would be to implement your custom insert hyperlink dialog by implementing IInsertHyperlinkDialog and use it in your RadRichTextBox. If you would like, you can open a support ticket and we can send you the code of InsertHyperlinkDialog for reference.
I hope that answers your question. Looking forward to hearing from you about the solution you choose.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items