Latest AJAX Control Toolkits could not work with RadScriptManager and there is no available workaround for this. The AJAX Control Toolkit requires that you use ToolkitScriptManager instead in order to work correctly.
Attached you could find a sample page with "ModalPopupExtender" with RadGrid in it that works without any issues.
Moreover, you could use our RadWindow
instead of the ModalPopupExtender, thus will make the use of RadScriptManager possible.
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed