All dialogs in RadRichTextBox's predefined UI can be easily replaced with a custom one. Please find attached a sample project demonstrating how this can be achieved. The sample creates a custom InsertSymbolDialog which has the implementation of the default one.
I hope this helps!
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.