Just looked into this and think I have a solution for you. The first thing to note is that RadToolBarItem is just a generic item, so to speak, which is the base for any of the RadToolBar button types. Since they all don't support being checked, the base type doesn't support it.
So, in a case like yours I'm gonna guess you might be using a RadToolBarButton, in which case you'd just do something like this:
That'll give you access to the Checked property and you'll be nearly set. I say nearly because it appears that the Checked functionality isn't turned on for the individual item unless you have CheckedOnClick="true" in the markup, in which case it'll reflect the Checked style. Otherwise you can set Checked all day, but the functionality won't be there unless you've told the control in advance that it can be checked (CheckedOnClick="true").