I found out an old code library submission on adding a custom control to the editor's toolbar. Probably the logic used in the project should help you get started: Adding customer controls to RadEditor toolbars
Hope this helps atleast to some extent :)
-Princy.