The RadCollapsiblePanel will try to fit as many as possible items on Height direction and if the buttons are in Medium state, their Height is just enough for the CollapsiblePanel to fit 3 of them. So in order to fit just two buttons, you'll have to make them bigger, which I'm not sure if it is ok for you. You can also accomplish this by changing the default Style of the RadRibbonButton, find the "textRow1" element in the style and modify its size or Margins for example. (the Margin might be better solution for you since, it's not going to make the button bigger.)
Please let us know if this is working for you.
the Telerik team