The "magic" happens in the RadRichTextBoxRibbonUI.RichTextCommand attached property. You can read more about it in this blogpost (a little outdated, but the main idea remains), this help page, and this forum thread (describing how to create your custom UI for the commands RadRichTextBox exposes).
Essentially most of the commands exposes event that is raised whenever state changes (e.g. the selection is over bold text); you can subscribe to this event and update the state of the control in the event handler.
Don't hesitate to contact us if you have other questions.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items