Take a look at the following forum thread. There you can find the solution how to achieve the required functionality: Enable ToolBar Button in preview mode.
Note that 90-95% of RadEditor's commands are written for Design mode only (editable iframe) and they will not function properly or at all in HTML mode( which is a textarea).
All the best,
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed