The predefined dialogs can only take HTML elements and not complex server controls, because their rendering is different and they have more complex requirements regarding the page lifecycle. The built-in buttons are designed to provide a consistent look with the rest of the controls and offer localization as explained in this help article
and shown in this online demo
, so I believe you needn't use RadButtons for the predefined dialogs. Nevertheless, if you need to use them you can create a RadWIndow configured look like an alert (size, modality, behaviors, etc.) and in its content template you can place the controls you need. You would only need to attach custom event handlerse for the buttons that get a proper reference to the control in order to close it with the respective argument.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed