It seems that the problem you are experiencing is caused in IE in quirks mode. In brief - elements that are originally inline, but are set to display: block and floated do not respect the width that is applied to them, but expand to 100% as is the case with RadEditor's buttons. Internally, this problem is already fixed and with the release of SP1 next week will not cause trouble anymore. In the meantime, you may use the following quick fix until SP1 is released:
_display: inline !important;
_width: 32px !important;
the Telerik team
Check out Telerik Trainer
, the state of the art learning tool for Telerik products.