Thank you for writing.
Currently, there is an issue which prevents the shape to be applied to the border of the button. Here you can add your vote for this issue and also subscribe for its status updates: http://www.telerik.com/support/pits.aspx#/public/winforms/12581
For the time being, you can hide the button border and apply the desired shape to the button:
RoundRectShape shape =
radButton1.ButtonElement.Shape = shape;
I hope this helps.
All the best,
the Telerik team