Please note that the RadToolTip's corners are rounded by default in some of the skins and this works in all browsers, as it is done via background images and CSS.
If that is not enough for your needs the quickest way is to remove the default borders, as they are rendered with table elements and use the content wrapper and display its border. You can find a simple page that demonstrates this approach attached.
Please note that this method will only work under newer browsers (like IE9, Firefox 4, Opera 11) and under older browsers it will display regular square borders. It is also by far not the best way to modify the appearance of the tooltip.
If you need to support all browsers you will have to create a custom skin and fully customize the appearance to match you needs. More information on the RadToolTip's CSS classes is available in the online documentation (http://www.telerik.com/help/aspnet-ajax/tooltip-appearance-css-selectors.html) and can be observed in tools like firebug for fine-tuning.
the Telerik team